$i \in [n]$, thus

$$ \max_{j \in [n]} \langle z_i(t), z_j(t) - z_i(t) \rangle \ge \max_{j \in [m]} \langle z_i(t), v_j - z_i(t) \rangle \ge \frac{c'}{2}. $$

Since $c'$ is independent of $\delta$, we deduce the claim in this case (notice that it suffices to prove the claim for sufficiently small $\delta$).

**Second case.** Secondly, we prove the claim when $z_i(t) \in S_{\delta_0} \setminus S_{\delta}$. The proof mainly relies on the following result:

**Lemma 8.4.** For any $w \in S$, there exists $\beta > 0$ such that if $\delta \in K \cap B(w, \delta_0)$, then

$$ \max_{j \in [m]} \langle x, v_j - x \rangle \ge \beta \|x - w\|. \quad (8.19) $$

We postpone the proof of Lemma 8.4 and show how to conclude the proof of Claim 2. Fix $\delta > 0$. We set

$$ \eta := \frac{\beta \delta}{6R} $$

where

$$ R := \max_{j \in [n]} \|z_j\|_{L^{\infty}(\mathbb{R})}. $$

Since $\text{conv}(\{z_j(t)\}_{j \in [n]})$ converges to $K$ as $t \to +\infty$, there exists $T(\delta) > 0$ such that for any $t \ge T(\delta)$, if $z_i(t) \in B(w, \delta_0) \setminus B(w, \delta)$ for some $w \in S$, then

$$ \|z_i(t) - x\| \le \eta $$

for some $x \in K \cap (B(w, \delta_0) \setminus B(w, \delta))$. Therefore, using Lemma 8.4,

$$ \begin{aligned} \max_{j \in [m]} \langle z_i(t), v_j - z_i(t) \rangle &\ge \max_{j \in [m]} \langle x, v_j - x \rangle - 3R\eta \\ &\ge \beta\delta - 3R\eta \\ &= \frac{\beta}{2}\delta. \end{aligned} $$

To summarize, we have found that for any $\delta > 0$ there exists $T(\delta) > 0$ such that if $t \ge T(\delta)$ and $z_i(t) \in S_{\delta_0} \setminus S_{\delta}$, then

$$ \max_{j \in [m]} \langle z_i(t), v_j - z_i(t) \rangle \ge \frac{\beta}{2} \delta. \quad (8.20) $$

Combining (8.20) with

$$ \max_{j \in [n]} \langle z_i(t), z_j(t) - z_i(t) \rangle \ge \max_{j \in [m]} \langle z_i(t), v_j - z_i(t) \rangle $$

concludes the proof of Claim 2 in this second case. $\square$

**Proof of Lemma 8.4.** Let us first address the case where $w = 0$. Writing any $x \in K \setminus \{0\}$ as a convex combination of the vertices: $x = \sum_{j=1}^{m} \alpha_j v_j$, we find

$$ 0 = \left\langle x, \sum_{j=1}^{m} \alpha_j (v_j - x) \right\rangle = \sum_{j=1}^{m} \alpha_j \langle x, v_j - x \rangle. \quad (8.21) $$

<footnote n="6">⁶Here, $B(y, r)$ denotes the closed ball with center $y \in \mathbb{R}^d$ and radius $r > 0$.</footnote>
$i \in [n]$, thus

$$ \max_{j \in [n]} \langle z_i(t), z_j(t) - z_i(t) \rangle \ge \max_{j \in [m]} \langle z_i(t), v_j - z_i(t) \rangle \ge \frac{c'}{2}. $$

Since $c'$ is independent of $\delta$, we deduce the claim in this case (notice that it suffices to prove the claim for sufficiently small $\delta$).

**Second case.** Secondly, we prove the claim when $z_i(t) \in S_{\delta_0} \setminus S_{\delta}$. The proof mainly relies on the following result:

**Lemma 8.4.** For any $w \in S$, there exists $\beta > 0$ such that if $\delta \in K \cap B(w, \delta_0)$, then

$$ \max_{j \in [m]} \langle x, v_j - x \rangle \ge \beta \|x - w\|. \quad (8.19) $$

We postpone the proof of Lemma 8.4 and show how to conclude the proof of Claim 2. Fix $\delta > 0$. We set

$$ \eta := \frac{\beta \delta}{6R} $$

where

$$ R := \max_{j \in [n]} \|z_j\|_{L^{\infty}(\mathbb{R})}. $$

Since $\text{conv}(\{z_j(t)\}_{j \in [n]})$ converges to $K$ as $t \to +\infty$, there exists $T(\delta) > 0$ such that for any $t \ge T(\delta)$, if $z_i(t) \in B(w, \delta_0) \setminus B(w, \delta)$ for some $w \in S$, then

$$ \|z_i(t) - x\| \le \eta $$

for some $x \in K \cap (B(w, \delta_0) \setminus B(w, \delta))$. Therefore, using Lemma 8.4,

$$ \begin{aligned} \max_{j \in [m]} \langle z_i(t), v_j - z_i(t) \rangle &\ge \max_{j \in [m]} \langle x, v_j - x \rangle - 3R\eta \\ &\ge \beta\delta - 3R\eta \\ &= \frac{\beta}{2}\delta. \end{aligned} $$

To summarize, we have found that for any $\delta > 0$ there exists $T(\delta) > 0$ such that if $t \ge T(\delta)$ and $z_i(t) \in S_{\delta_0} \setminus S_{\delta}$, then

$$ \max_{j \in [m]} \langle z_i(t), v_j - z_i(t) \rangle \ge \frac{\beta}{2} \delta. \quad (8.20) $$

Combining (8.20) with

$$ \max_{j \in [n]} \langle z_i(t), z_j(t) - z_i(t) \rangle \ge \max_{j \in [m]} \langle z_i(t), v_j - z_i(t) \rangle $$

concludes the proof of Claim 2 in this second case. $\square$

**Proof of Lemma 8.4.** Let us first address the case where $w = 0$. Writing any $x \in K \setminus \{0\}$ as a convex combination of the vertices: $x = \sum_{j=1}^{m} \alpha_j v_j$, we find

$$ 0 = \left\langle x, \sum_{j=1}^{m} \alpha_j (v_j - x) \right\rangle = \sum_{j=1}^{m} \alpha_j \langle x, v_j - x \rangle. \quad (8.21) $$

<footnote n="6">⁶Here, $B(y, r)$ denotes the closed ball with center $y \in \mathbb{R}^d$ and radius $r > 0$.</footnote>
We can exclude having $\langle x, v_j - x \rangle = 0$ for all $j \in [m]$, as this would necessarily
imply that $\|x\|^2 = 2 \sum_{j=1}^m \alpha_j \langle x, v_j - x \rangle = 0$. We deduce from (8.21) that

$$
\max_{j \in [m]} \langle x, v_j - x \rangle > 0
$$

for any $x \in \mathcal{K} \setminus \{0\}$. Hence, it is sufficient to prove (8.19) for $\|x\|$ small enough. We
notice that for any $x \in \mathcal{K} \setminus \{0\}$ written as above,

$$
\|x\|^2 = \sum_{j=1}^{m} \alpha_j \langle v_j, x \rangle.
$$

Hence $x \mapsto \max_{j \in [m]} \langle v_j, x \rangle$ is positive for $x \in \mathcal{K} \setminus \{0\}$. Since this function is continuous and homogeneous in $x$, we deduce the existence of $\beta > 0$ such that

$$
\max_{j \in [m]} \langle v_j, x \rangle \ge 2\beta \|x\|
$$

for any $x \in \mathcal{K}$. For $x \in \mathcal{K}$ with $\|x\|$ sufficiently small, we obtain (8.19).
We now assume that $w \in S \setminus \{0\}$. We set

$$
I_w := \{j \in [n] : \|w\|^2 = \langle w, v_j \rangle\}
$$

and

$$
\mathcal{A} := \operatorname{span} (\{v_j - w : j \in \mathcal{I}_w\}),
$$

which is orthogonal to $w$. We also introduce

$$
\mathcal{R} := (\mathbb{R}w \oplus \mathcal{A})^{\perp},
$$

and we denote by $\pi_{\mathcal{R}}$ the orthogonal projection on $\mathcal{R}$. We claim that there exists
some $\rho > 0$ such that for any $j \in [m]$, we have

$$
\langle w - v_j, w \rangle \ge \rho \| \pi_{\mathcal{R}} v_j \|.
$$

This follows from the observation that $[m]$ is finite, and that $\|\pi_{\mathcal{R}} v_j\| > 0$ implies
$\langle w - v_j, w \rangle > 0$. Therefore, for any $x \in \mathcal{K}$, writing $x$ as a convex combination of
the vertices, namely $x = \sum_{j=1}^m \alpha_j v_j$, we find that

$$
\rho \|\pi_{\mathcal{R}} x\| \le \sum_{j=1}^{m} \alpha_j \|\pi_{\mathcal{R}} v_j\| \le \sum_{j=1}^{m} \alpha_j \langle w - v_j, w \rangle = \langle w - x, w \rangle. \quad (8.22)
$$

Fix $x \in \mathcal{K} \cap B(w, \delta_0)$. We write $x = w + \delta'u$ with $0 \le \delta' \le \delta_0$ and $\|u\| = 1$. Then
we have the orthogonal decomposition

$$
u = bw + a + r \tag{8.23}
$$

where $a \in \mathcal{A}$, $r \in \mathcal{R}$ and $b \in \mathbb{R}$. Since $a$ is a convex combination of the form

$$
a = \sum_{j \in I_w} \beta_j (v_j - w),
$$

we have

$$
\|a\|^2 = \sum_{j \in I_w} \beta_j \langle v_j - w, a \rangle,
$$

whence

$$
\max_{j \in I_w} \langle a, v_j - w \rangle \ge \|a\|^2.
$$
We can exclude having $\langle x, v_j - x \rangle = 0$ for all $j \in [m]$, as this would necessarily
imply that $\|x\|^2 = 2 \sum_{j=1}^m \alpha_j \langle x, v_j - x \rangle = 0$. We deduce from (8.21) that

$$
\max_{j \in [m]} \langle x, v_j - x \rangle > 0
$$

for any $x \in \mathcal{K} \setminus \{0\}$. Hence, it is sufficient to prove (8.19) for $\|x\|$ small enough. We
notice that for any $x \in \mathcal{K} \setminus \{0\}$ written as above,

$$
\|x\|^2 = \sum_{j=1}^{m} \alpha_j \langle v_j, x \rangle.
$$

Hence $x \mapsto \max_{j \in [m]} \langle v_j, x \rangle$ is positive for $x \in \mathcal{K} \setminus \{0\}$. Since this function is continuous and homogeneous in $x$, we deduce the existence of $\beta > 0$ such that

$$
\max_{j \in [m]} \langle v_j, x \rangle \ge 2\beta \|x\|
$$

for any $x \in \mathcal{K}$. For $x \in \mathcal{K}$ with $\|x\|$ sufficiently small, we obtain (8.19).
We now assume that $w \in S \setminus \{0\}$. We set

$$
I_w := \{j \in [n] : \|w\|^2 = \langle w, v_j \rangle\}
$$

and

$$
\mathcal{A} := \operatorname{span} (\{v_j - w : j \in \mathcal{I}_w\}),
$$

which is orthogonal to $w$. We also introduce

$$
\mathcal{R} := (\mathbb{R}w \oplus \mathcal{A})^{\perp},
$$

and we denote by $\pi_{\mathcal{R}}$ the orthogonal projection on $\mathcal{R}$. We claim that there exists
some $\rho > 0$ such that for any $j \in [m]$, we have

$$
\langle w - v_j, w \rangle \ge \rho \|\pi_{\mathcal{R}} v_j\|.
$$

This follows from the observation that $[m]$ is finite, and that $\|\pi_{\mathcal{R}} v_j\| > 0$ implies
$\langle w - v_j, w \rangle > 0$. Therefore, for any $x \in \mathcal{K}$, writing $x$ as a convex combination of
the vertices, namely $x = \sum_{j=1}^m \alpha_j v_j$, we find that

$$
\rho \|\pi_{\mathcal{R}} x\| \le \sum_{j=1}^{m} \alpha_j \|\pi_{\mathcal{R}} v_j\| \le \sum_{j=1}^{m} \alpha_j \langle w - v_j, w \rangle = \langle w - x, w \rangle. \quad (8.22)
$$

Fix $x \in \mathcal{K} \cap B(w, \delta_0)$. We write $x = w + \delta'u$ with $0 \le \delta' \le \delta_0$ and $\|u\| = 1$. Then
we have the orthogonal decomposition

$$
u = bw + a + r \tag{8.23}
$$

where $a \in \mathcal{A}$, $r \in \mathcal{R}$ and $b \in \mathbb{R}$. Since $a$ is a convex combination of the form

$$
a = \sum_{j \in \mathcal{I}_w} \beta_j (v_j - w),
$$

we have

$$
\|a\|^2 = \sum_{j \in \mathcal{I}_w} \beta_j \langle v_j - w, a \rangle,
$$

whence

$$
\max_{j \in \mathcal{I}_w} \langle a, v_j - w \rangle \ge \|a\|^2.
$$
We deduce that

$$
\begin{align}
\max_{j \in \mathcal{I}_w} \langle x, v_j - x \rangle &= \max_{j \in \mathcal{I}_w} \langle w + \delta' u, (v_j - w) - \delta' u \rangle \nonumber \\
&= -\delta' b \|w\|^2 - \delta'^2 + \delta' \max_{j \in \mathcal{I}_w} \langle a, v_j - w \rangle \nonumber \\
&\geq -\delta' b \|w\|^2 - \delta'^2 + \delta' \|a\|^2. \tag{8.24}
\end{align}
$$

Notice that $b \le 0$ by combining (8.22) and (8.23). Since $\|u\| = 1$ and using (8.22)
we have

$$
1 = b^2 + \|a\|^2 + \|r\|^2 \le \|a\|^2 + \kappa b^2 \le \kappa (\|a\|^2 + b^2)
$$

where $\kappa := 1 + \rho^{-2} \|w\|^4$. We deduce that either $\|a\|^2 \ge (2\kappa)^{-1}$ or $-b = |b| \ge (2\kappa)^{-1/2}$.
Plugging this knowledge in (8.24) and using the fact that $\|w\| > 0$, we finally deduce
the existence of an $\alpha > 0$ (independent of $\delta > 0$ and $x \in \mathcal{K} \cap B(w, \delta_0)$) such that

$$
\max_{j \in [m]} \langle x, v_j - x \rangle \ge \alpha \delta' - \delta'^2 = \alpha \|x - w\| - \|x - w\|^2.
$$

This proves (8.19) when $\|x - w\| \le \alpha/2$.

It thus remains to show that (8.19) holds for all $x \in \mathcal{K} \cap (B(w, \delta_0)\setminus B(w, \frac{\alpha}{2}))$. To this end, we notice that $x \mapsto \max_{j \in [m]} \langle x, v_j - x \rangle$ is continuous in the connected set $\mathcal{K} \cap (B(w, \delta_0)\setminus B(w, \frac{\alpha}{2}))$, non-negative according to (8.5), and it is nowhere 0 (by definition of $\mathcal{S}$). Therefore, it is strictly positive, and denote by $\alpha' > 0$ some lower bound. Then for $x \in \mathcal{K} \cap (B(w, \delta_0)\setminus B(w, \frac{\alpha}{2}))$, we have

$$
\max_{j \in [m]} \langle x, v_j - x \rangle \ge \alpha' \ge \frac{\alpha'}{\delta_0} \|x - w\|.
$$

This concludes the proof of Lemma 8.4.

<figure><img src="image_18.png" /></figure>

8.2. **A cluster at the origin.** We complete this section by addressing the case $V = -I_d$, for which the convergence of the solutions of (1.1) is the simplest, since a unique cluster forms at the origin. We also suppose that $Q^\top K = I_d$: in other words, we consider the dynamics

$$
\dot{x}_i(t) = - \sum_{j=1}^{n} \left( \frac{e^{\langle x_i(t), x_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle x_i(t), x_k(t) \rangle}} \right) x_j(t), \quad t \in [0, +\infty), \quad (8.25)
$$

with a prescribed initial condition {$x_i(0)$}$_{i \in [n]} \subset \mathbb{R}^d$.

**Theorem 8.5 (Convergence toward the origin).** Suppose $V = -I_d$ and $Q^\top K = I_d$.
Then, for any initial sequence of tokens $\{x_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, and for any $i \in [n]$, we
have $\|x_i(t)\| \to 0$ as $t \to +\infty$.

**Remark 8.6.** In the setting of Theorem 8.5, the self-attention matrix $P(t)$ defined in (1.2) converges, as $t \to +\infty$, to the $n \times n$ matrix with all entries equal to $1/n$.

8.2.1. *Proof of Theorem 8.5.* We begin by showing that for any $i \in [n]$, the solution to (8.25) is uniformly bounded for all $t > 0$. In the sequel, we fix an initial configuration $\{x_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$.

**Lemma 8.7.** The trajectories of (8.25) are uniformly bounded in time—namely, there exists $R > 0$ (depending solely on $n$ and the initial configuration) such that the solution $x_i(\cdot)$ to (8.25) satisfies $\|x_i(t)\| \le R$ for any $i \in [n]$ and $t \ge 0$.
We deduce that

$$
\begin{align}
\max_{j \in \mathcal{I}_w} \langle x, v_j - x \rangle &= \max_{j \in \mathcal{I}_w} \langle w + \delta' u, (v_j - w) - \delta' u \rangle \nonumber \\
&= -\delta' b \|w\|^2 - \delta'^2 + \delta' \max_{j \in \mathcal{I}_w} \langle a, v_j - w \rangle \nonumber \\
&\geq -\delta' b \|w\|^2 - \delta'^2 + \delta' \|a\|^2. \tag{8.24}
\end{align}
$$

Notice that $b \le 0$ by combining (8.22) and (8.23). Since $\|u\| = 1$ and using (8.22)
we have

$$
1 = b^2 + \|a\|^2 + \|r\|^2 \le \|a\|^2 + \kappa b^2 \le \kappa(\|a\|^2 + b^2)
$$

where $\kappa := 1 + \rho^{-2} \|w\|^4$. We deduce that either $\|a\|^2 \ge (2\kappa)^{-1}$ or $-b = |b| \ge (2\kappa)^{-1/2}$.
Plugging this knowledge in (8.24) and using the fact that $\|w\| > 0$, we finally deduce
the existence of an $\alpha > 0$ (independent of $\delta > 0$ and $x \in \mathcal{K} \cap B(w, \delta_0)$) such that

$$
\max_{j \in [m]} \langle x, v_j - x \rangle \ge \alpha \delta' - \delta'^2 = \alpha \|x - w\| - \|x - w\|^2.
$$

This proves (8.19) when $\|x - w\| \le \alpha/2$.

It thus remains to show that (8.19) holds for all $x \in \mathcal{K} \cap (B(w, \delta_0) \setminus B(w, \frac{\alpha}{2}))$. To this end, we notice that $x \mapsto \max_{j \in [m]} \langle x, v_j - x \rangle$ is continuous in the connected set $\mathcal{K} \cap (B(w, \delta_0) \setminus B(w, \frac{\alpha}{2}))$, non-negative according to (8.5), and it is nowhere 0 (by definition of $\mathcal{S}$). Therefore, it is strictly positive, and denote by $\alpha' > 0$ some lower bound. Then for $x \in \mathcal{K} \cap (B(w, \delta_0) \setminus B(w, \frac{\alpha}{2}))$, we have

$$
\max_{j \in [m]} \langle x, v_j - x \rangle \ge \alpha' \ge \frac{\alpha'}{\delta_0} \|x - w\|.
$$

This concludes the proof of Lemma 8.4.

<figure><img src="image_19.png" /></figure>

8.2. **A cluster at the origin.** We complete this section by addressing the case $V = -I_d$, for which the convergence of the solutions of (1.1) is the simplest, since a unique cluster forms at the origin. We also suppose that $Q^\top K = I_d$: in other words, we consider the dynamics

$$
\dot{x}_i(t) = - \sum_{j=1}^{n} \left( \frac{e^{\langle x_i(t), x_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle x_i(t), x_k(t) \rangle}} \right) x_j(t), \quad t \in [0, +\infty), \quad (8.25)
$$

with a prescribed initial condition {$x_i(0)$}$_{i \in [n]} \subset \mathbb{R}^d$.

**Theorem 8.5 (Convergence toward the origin).** Suppose $V = -I_d$ and $Q^\top K = I_d$.
Then, for any initial sequence of tokens $\{x_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, and for any $i \in [n]$, we
have $\|x_i(t)\| \to 0$ as $t \to +\infty$.

**Remark 8.6.** In the setting of Theorem 8.5, the self-attention matrix $P(t)$ defined in (1.2) converges, as $t \to +\infty$, to the $n \times n$ matrix with all entries equal to $1/n$.

8.2.1. *Proof of Theorem 8.5.* We begin by showing that for any $i \in [n]$, the solution to (8.25) is uniformly bounded for all $t > 0$. In the sequel, we fix an initial configuration $\{x_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$.

**Lemma 8.7.** The trajectories of (8.25) are uniformly bounded in time—namely, there exists $R > 0$ (depending solely on $n$ and the initial configuration) such that the solution $x_i(\cdot)$ to (8.25) satisfies $\|x_i(t)\| \le R$ for any $i \in [n]$ and $t \ge 0$.
*Proof of Lemma 8.7.* We fix $i \in [n]$. For $t \ge 0$, we denote by $D_i(t)$ the set of points $x_k(t)$ such that $\langle x_i(t), x_k(t) \rangle \ge 0$. We also set

$$S_i(t) := \sum_{k \in D_i(t)} e^{\langle x_i(t), x_k(t) \rangle} \langle x_i(t), x_k(t) \rangle,$$

and

$$R_i(t) := \sum_{k=1}^{n} e^{\langle x_i(t), x_k(t) \rangle}.$$

Since $1 + x \le e^x$ whence $e^{-x}x \le 1$, we deduce that

$$\frac{1}{2} \frac{d}{dt} \|x_i(t)\|^2 = - \frac{\sum_{k=1}^{n} e^{\langle x_i(t), x_k(t) \rangle} \langle x_i(t), x_k(t) \rangle}{R_i(t)} \le \frac{-S_i(t) + n}{R_i(t)}.$$

Now since $1 - x \le e^{-x}$ whence $e^x \le 1 + e^x x$, we find that $R_i(t) \le n + S_i(t)$. Consequently, if we assume that $\|x_i(t)\|^2 \ge 2n$ then $S_i(t) \ge 2n$, and therefore

$$\frac{1}{2} \frac{d}{dt} \|x_i(t)\|^2 \le \frac{-S_i(t) + n}{n + S_i(t)} \le -1.$$

This shows that $\|x_i(t)\| \le \max\{\|x_i(0)\|, \sqrt{2n}\}$ for any $t \ge 0$, which concludes the proof. □

By virtue of Lemma 7.1, we are able to characterize the stationary configurations for the dynamics (8.25)—namely, the set of points $(\bar{x}_1, \dots, \bar{x}_n) \in (\mathbb{R}^d)^n$ satisfying

$$\sum_{j=1}^{n} \left( \frac{e^{\langle \bar{x}_i, \bar{x}_j \rangle}}{\sum_{k=1}^{n} e^{\langle \bar{x}_i, \bar{x}_k \rangle}} \right) \bar{x}_j = 0$$

for all $i \in [n]$.

**Lemma 8.8.** The only stationary configuration for the dynamics (8.25) is $\bar{x}_1 = \dots = \bar{x}_n = 0$.

*Proof.* Assume that $(\bar{x}_1, \dots, \bar{x}_n) \in (\mathbb{R}^d)^n$ is a stationary configuration for the dynamics (8.25). We consider $f : \mathbb{R}^d \to \mathbb{R}$ defined as

$$f : x \mapsto \log \left( \sum_{j=1}^{n} e^{\langle x, \bar{x}_j \rangle} \right).$$

Per Lemma 7.1, $f$ is convex, whence

$$f(x) \ge f(\bar{x}_i) + \langle \nabla f(\bar{x}_i), x - \bar{x}_i \rangle$$

for $x \in \mathbb{R}^d$ and $i \in [n]$. Since $\nabla f(\bar{x}_i) = 0$ for any $i \in [n]$, we gather that $f(x) \ge f(\bar{x}_i)$, whence $\bar{x}_i$ is a global minimizer of $f$ for any $i \in [n]$. By convexity, $f$ is constant on $\text{conv}(\{\bar{x}_i\}_{i \in [n]})$. Since $f$ is analytic on the affine space $E$ spanned by the points $\bar{x}_i$, $i \in [n]$, it is then constant on $E$ as well. Now assume that not all of the points $\bar{x}_i$ are equal, and pick an index $i_0 \in [n]$ such that $\bar{x}_{i_0}$ is not equal to the projection of the origin onto $E$. Then there exists some $j_0 \in [n]$ such that $\langle \bar{x}_{i_0} - \bar{x}_{j_0}, \bar{x}_{i_0} \rangle \ne 0$. For any $s \in \mathbb{R}$, we set $P_s := \bar{x}_{j_0} + s(\bar{x}_{i_0} - \bar{x}_{j_0}) \in E$, and we notice that $f(P_s) \ge \langle P_s, \bar{x}_{i_0} \rangle$, where the lower bound tends to $+\infty$ either when $s \to +\infty$ or when $s \to -\infty$. This contradicts the fact that $f$ is constant on $E$. We conclude that the $\bar{x}_i$ are all equal for $i \in [n]$. The only value they can then take is necessarily 0. □
*Proof of Lemma 8.7.* We fix $i \in [n]$. For $t \ge 0$, we denote by $D_i(t)$ the set of points $x_k(t)$ such that $\langle x_i(t), x_k(t) \rangle \ge 0$. We also set

$$S_i(t) := \sum_{k \in D_i(t)} e^{\langle x_i(t), x_k(t) \rangle} \langle x_i(t), x_k(t) \rangle,$$

and

$$R_i(t) := \sum_{k=1}^{n} e^{\langle x_i(t), x_k(t) \rangle}.$$

Since $1 + x \le e^x$ whence $e^{-x}x \le 1$, we deduce that

$$\frac{1}{2} \frac{d}{dt} \|x_i(t)\|^2 = - \frac{\sum_{k=1}^{n} e^{\langle x_i(t), x_k(t) \rangle} \langle x_i(t), x_k(t) \rangle}{R_i(t)} \le \frac{-S_i(t) + n}{R_i(t)}.$$

Now since $1 - x \le e^{-x}$ whence $e^x \le 1 + e^x x$, we find that $R_i(t) \le n + S_i(t)$. Consequently, if we assume that $\|x_i(t)\|^2 \ge 2n$ then $S_i(t) \ge 2n$, and therefore

$$\frac{1}{2} \frac{d}{dt} \|x_i(t)\|^2 \le \frac{-S_i(t) + n}{n + S_i(t)} \le -1.$$

This shows that $\|x_i(t)\| \le \max\{\|x_i(0)\|, \sqrt{2n}\}$ for any $t \ge 0$, which concludes the proof. $\square$

By virtue of Lemma 7.1, we are able to characterize the stationary configurations for the dynamics (8.25)—namely, the set of points $(\bar{x}_1, \dots, \bar{x}_n) \in (\mathbb{R}^d)^n$ satisfying

$$\sum_{j=1}^{n} \left( \frac{e^{\langle \bar{x}_i, \bar{x}_j \rangle}}{\sum_{k=1}^{n} e^{\langle \bar{x}_i, \bar{x}_k \rangle}} \right) \bar{x}_j = 0$$

for all $i \in [n]$.

**Lemma 8.8.** The only stationary configuration for the dynamics (8.25) is $\bar{x}_1 = \dots = \bar{x}_n = 0$.

*Proof.* Assume that $(\bar{x}_1, \dots, \bar{x}_n) \in (\mathbb{R}^d)^n$ is a stationary configuration for the dynamics (8.25). We consider $f : \mathbb{R}^d \to \mathbb{R}$ defined as

$$f : x \mapsto \log \left( \sum_{j=1}^{n} e^{\langle x, \bar{x}_j \rangle} \right).$$

Per Lemma 7.1, $f$ is convex, whence

$$f(x) \ge f(\bar{x}_i) + \langle \nabla f(\bar{x}_i), x - \bar{x}_i \rangle$$

for $x \in \mathbb{R}^d$ and $i \in [n]$. Since $\nabla f(\bar{x}_i) = 0$ for any $i \in [n]$, we gather that $f(x) \ge f(\bar{x}_i)$, whence $\bar{x}_i$ is a global minimizer of $f$ for any $i \in [n]$. By convexity, $f$ is constant on $\text{conv}(\{\bar{x}_i\}_{i \in [n]})$. Since $f$ is analytic on the affine space $E$ spanned by the points $\bar{x}_i$, $i \in [n]$, it is then constant on $E$ as well. Now assume that not all of the points $\bar{x}_i$ are equal, and pick an index $i_0 \in [n]$ such that $\bar{x}_{i_0}$ is not equal to the projection of the origin onto $E$. Then there exists some $j_0 \in [n]$ such that $\langle \bar{x}_{i_0} - \bar{x}_{j_0}, \bar{x}_{i_0} \rangle \ne 0$. For any $s \in \mathbb{R}$, we set $P_s := \bar{x}_{j_0} + s(\bar{x}_{i_0} - \bar{x}_{j_0}) \in E$, and we notice that $f(P_s) \ge \langle P_s, \bar{x}_{i_0} \rangle$, where the lower bound tends to $+\infty$ either when $s \to +\infty$ or when $s \to -\infty$. This contradicts the fact that $f$ is constant on $E$. We conclude that the $\bar{x}_i$ are all equal for $i \in [n]$. The only value they can then take is necessarily 0. $\square$
**Lemma 8.9.** The trajectories of (8.25) satisfy $\int_{0}^{+\infty} \|\dot{x}_i(t)\|^2 dt < +\infty$ for any $i \in [n]$.

*Proof.* The function

$$ \mathcal{L}: t \mapsto \sum_{i=1}^{n} \sum_{j=1}^{n} e^{\langle x_i(t), x_j(t) \rangle} $$

is non-increasing, as demonstrated by the following simple computation:

$$ \begin{aligned} \frac{d\mathcal{L}(t)}{dt} &= 2 \sum_{i=1}^{n} \sum_{j=1}^{n} e^{\langle x_i(t), x_j(t) \rangle} \langle \dot{x}_i(t), x_j(t) \rangle = 2 \sum_{i=1}^{n} \left\langle \dot{x}_i(t), \sum_{j=1}^{n} e^{\langle x_i(t), x_j(t) \rangle} x_j(t) \right\rangle \\ &= -2 \sum_{i=1}^{n} \sum_{j=1}^{n} e^{\langle x_i(t), x_j(t) \rangle} \|\dot{x}_i(t)\|^2. \end{aligned} $$

Being non-negative, $\mathcal{L}(t)$ thus converges as $t \to +\infty$. Since $\langle x_i(t), x_j(t) \rangle \ge R$ for some (possibly negative) $R \in \mathbb{R}$ by virtue of Lemma 8.7, we deduce that

$$ \int_{0}^{+\infty} \|\dot{x}_i(t)\|^2 dt \le e^{-R} \int_{0}^{+\infty} \sum_{i=1}^{n} \sum_{j=1}^{n} e^{\langle x_i(t), x_j(t) \rangle} \|\dot{x}_i(t)\|^2 dt = e^{-R}(\mathcal{L}(0) - \lim_{t \to +\infty} \mathcal{L}(t)), $$

which concludes the proof. □

We are now able to conclude the proof of Theorem 8.5.

*Proof of Theorem 8.5.* We set $\mathbf{X}(t) := (x_1(t), \dots, x_n(t)) \in (\mathbb{R}^d)^n$. If $\mathbf{X}(t)$ does not converge to 0, the compactness provided by Lemma 8.7 implies that there is a sequence $\{t_k\}_{k=1}^{+\infty}$ with $t_k \to +\infty$, and $\mathbf{X}^* = (x_1^*(t), \dots, x_n^*(t)) \in (\mathbb{R}^d)^n \setminus \{0\}$, such that $\mathbf{X}(t_k) \to \mathbf{X}^*$ as $k \to +\infty$. To conclude the proof, it suffices to show that $\mathbf{X}^*$ is a stationary configuration of the dynamics: this directly leads to a contradiction per Lemma 8.8. Therefore, assume that $\mathbf{X}^*$ is not a stationary configuration of the dynamics. We denote by $\mathbf{X}^*(t) = (x_1^*(t), \dots, x_n^*(t))$ the solution of (8.25) with initial condition $\mathbf{X}^*$. Then, there exists $i \in [n]$ such that $\dot{x}_i^*(0) \ne 0$. We set $\varepsilon = \|\dot{x}_i^*(0)\|$. We select $T_0 > 0$ (possibly small) such that $\|\dot{x}_i^*(t)\| \ge \varepsilon/2$ for $t \in [0, T_0]$. It follows from (6.9) (which is verified according to Corollary 6.6) that for any $\delta > 0$ there exists $k_0 \in \mathbb{N}$ such that $\|\mathbf{X}(t_k + t) - \mathbf{X}^*(t)\| \le \delta$ for any $t \in [0, T_0]$ and any $k \ge k_0$. By (6.5) (which is verified according to Corollary 6.6), we obtain that $\|\dot{x}_i(t_k + t) - \dot{x}_i^0(t)\| \le C\delta$ for $t \in [0, T_0]$ and any $k \ge k_0$. Choosing $\delta > 0$ sufficiently small, we obtain that $\|\dot{x}_i(t_k + t)\| \ge \varepsilon/4$ for $t \in [0, T_0]$ and any $k \ge k_0$. This contradicts Lemma 8.9. □

## 9. PROOF OF THEOREM 4.2

To ensure clarity, we present the proof of Theorem 4.2 under the assumption that $V$ is diagonalizable. However, this assumption is not necessary. In Remark 9.5, we explain how the proof can be modified to accommodate for non-diagonalizable $V$.

Let us therefore assume that $V$ is diagonalizable. Let $(\varphi_1, \dots, \varphi_d)$ be an orthonormal basis of eigenvectors associated to eigenvalues $(\lambda_1, \dots, \lambda_d)$, ordered in a decreasing manner with respect to their modulus: $|\lambda_1| \ge \dots \ge |\lambda_d|$. (Starting from this point and throughout, we use the symbol $\lambda$ exclusively to denote the eigenvalues of $V$.) Except for $\lambda_1 \in \mathbb{R}$, all the other eigenvalues (and eigenvectors) may be complex. We denote by $(\varphi_1^*, \dots, \varphi_d^*)$ the dual basis of $(\varphi_1, \dots, \varphi_d)$.
**Lemma 8.9.** The trajectories of (8.25) satisfy $\int_0^{+\infty} \|\dot{x}_i(t)\|^2 dt < +\infty$ for any $i \in [n]$.

*Proof.* The function

$$ \mathcal{L}: t \mapsto \sum_{i=1}^{n} \sum_{j=1}^{n} e^{\langle x_i(t), x_j(t) \rangle} $$

is non-increasing, as demonstrated by the following simple computation:

$$ \begin{aligned} \frac{d\mathcal{L}(t)}{dt} &= 2 \sum_{i=1}^{n} \sum_{j=1}^{n} e^{\langle x_i(t), x_j(t) \rangle} \langle \dot{x}_i(t), x_j(t) \rangle = 2 \sum_{i=1}^{n} \left\langle \dot{x}_i(t), \sum_{j=1}^{n} e^{\langle x_i(t), x_j(t) \rangle} x_j(t) \right\rangle \\ &= -2 \sum_{i=1}^{n} \sum_{j=1}^{n} e^{\langle x_i(t), x_j(t) \rangle} \|\dot{x}_i(t)\|^2. \end{aligned} $$

Being non-negative, $\mathcal{L}(t)$ thus converges as $t \to +\infty$. Since $\langle x_i(t), x_j(t) \rangle \ge R$ for some (possibly negative) $R \in \mathbb{R}$ by virtue of Lemma 8.7, we deduce that

$$ \int_0^{+\infty} \|\dot{x}_i(t)\|^2 dt \le e^{-R} \int_0^{+\infty} \sum_{i=1}^n \sum_{j=1}^n e^{\langle x_i(t), x_j(t) \rangle} \|\dot{x}_i(t)\|^2 dt = e^{-R} \mathcal{L}(0) - \lim_{t \to +\infty} \mathcal{L}(t), $$

which concludes the proof. □

We are now able to conclude the proof of Theorem 8.5.

*Proof of Theorem 8.5.* We set $\mathbf{X}(t) := (x_1(t), \dots, x_n(t)) \in (\mathbb{R}^d)^n$. If $\mathbf{X}(t)$ does not converge to 0, the compactness provided by Lemma 8.7 implies that there is a sequence $\{t_k\}_{k=1}^{+\infty}$ with $t_k \to +\infty$, and $\mathbf{X}^* = (x_1^*(t), \dots, x_n^*(t)) \in (\mathbb{R}^d)^n \setminus \{0\}$, such that $\mathbf{X}(t_k) \to \mathbf{X}^*$ as $k \to +\infty$. To conclude the proof, it suffices to show that $\mathbf{X}^*$ is a stationary configuration of the dynamics: this directly leads to a contradiction per Lemma 8.8. Therefore, assume that $\mathbf{X}^*$ is not a stationary configuration of the dynamics. We denote by $\mathbf{X}^*(t) = (x_1^*(t), \dots, x_n^*(t))$ the solution of (8.25) with initial condition $\mathbf{X}^*$. Then, there exists $i \in [n]$ such that $\dot{x}_i^*(0) \ne 0$. We set $\varepsilon = \|\dot{x}_i^*(0)\|$. We select $T_0 > 0$ (possibly small) such that $\|\dot{x}_i^*(t)\| \ge \varepsilon/2$ for $t \in [0, T_0]$. It follows from (6.9) (which is verified according to Corollary 6.6) that for any $\delta > 0$ there exists $k_0 \in \mathbb{N}$ such that $\|\mathbf{X}(t_k + t) - \mathbf{X}^*(t)\| \le \delta$ for any $t \in [0, T_0]$ and any $k \ge k_0$. By (6.5) (which is verified according to Corollary 6.6), we obtain that $\|\dot{x}_i(t_k + t) - \dot{x}_i^0(t)\| \le C\delta$ for $t \in [0, T_0]$ and any $k \ge k_0$. Choosing $\delta > 0$ sufficiently small, we obtain that $\|\dot{x}_i(t_k + t)\| \ge \varepsilon/4$ for $t \in [0, T_0]$ and any $k \ge k_0$. This contradicts Lemma 8.9. □

## 9. PROOF OF THEOREM 4.2

To ensure clarity, we present the proof of Theorem 4.2 under the assumption that $V$ is diagonalizable. However, this assumption is not necessary. In Remark 9.5, we explain how the proof can be modified to accommodate for non-diagonalizable $V$.

Let us therefore assume that $V$ is diagonalizable. Let $(\varphi_1, \dots, \varphi_d)$ be an orthonormal basis of eigenvectors associated to eigenvalues $(\lambda_1, \dots, \lambda_d)$, ordered in a decreasing manner with respect to their modulus: $|\lambda_1| \ge \dots \ge |\lambda_d|$. (Starting from this point and throughout, we use the symbol $\lambda$ exclusively to denote the eigenvalues of $V$.) Except for $\lambda_1 \in \mathbb{R}$, all the other eigenvalues (and eigenvectors) may be complex. We denote by $(\varphi_1^*, \dots, \varphi_d^*)$ the dual basis of $(\varphi_1, \dots, \varphi_d)$.
**9.1. Some monotonicity properties and bounds.** To start, we present some general facts that are prove useful in all subsequent sub-cases.

**Lemma 9.1.** Suppose $k \in [d]$ is such that $\lambda_k \ge 0$. Then $t \mapsto \max_{j \in [n]} \varphi_k^*(z_j(t))$ is a non-increasing and bounded function, and $t \mapsto \min_{j \in [n]} \varphi_k^*(z_j(t))$ is a non-decreasing and bounded function. In particular, $t \mapsto \varphi_k^*(z_i(t))$ is uniformly bounded as a function on $[0, +\infty)$ for any $i \in [n]$.

*Proof.* For any $k \in [d]$ and any $t \ge 0$, set

$$ \alpha_k(t) = \min_{j \in [n]} \varphi_k^*(z_j(t)), \quad \beta_k(t) = \max_{j \in [n]} \varphi_k^*(z_j(t)). $$

Let $i \in [n]$ be an index such that $\alpha_k(t) = \varphi_k^*(z_i(t))$. Then we have

$$ \begin{aligned} \frac{d}{dt} \varphi_k^*(z_i(t)) &= \sum_{j=1}^{n} P_{ij}(t) \varphi_k^*(V(z_j(t) - z_i(t))) \\ &= \lambda_k \sum_{j=1}^{n} P_{ij}(t) (\varphi_k^*(z_j(t)) - \varphi_k^*(z_i(t))) \ge 0 \end{aligned} $$

where the last inequality stems from the fact that $\lambda_k \ge 0$ and the choice of index $i$. This proves that $\alpha_k(\cdot)$ is non-decreasing, as desired. Arguing similarly, one finds that $\beta_k(\cdot)$ is non-increasing. As a consequence, $\alpha_k(0) \le \alpha_k(t) \le \beta_k(t) \le \beta_k(0)$ for any $t \ge 0$, which shows that $\alpha_k(\cdot)$ and $\beta_k(\cdot)$ are bounded. $\square$

**Corollary 9.2.** If $V$ only has real non-negative eigenvalues, then $z_i(\cdot) \in L^\infty([0, +\infty))$.

**Lemma 9.3.** Fix $k \in [d]$ and $i \in [n]$. Then there exists a constant $C > 0$ such that

$$ |\varphi_k^*(e^{tV}z_i(t))| \le C e^{\lambda_k |t|} $$

holds for all $t \ge 0$.

*Proof.* We naturally make use of the equation for $x_i(t) := e^{tV}z_i(t)$. Fix $t \ge 0$. We have

$$ \begin{aligned} \frac{d}{dt} |\varphi_k^*(x_i(t))|^2 &= 2 \cdot \operatorname{Re} \left( \overline{\varphi_k^*(x_i(t))} \frac{d}{dt} \varphi_k^*(x_i(t)) \right) \\ &= 2 \cdot \operatorname{Re} \left( \sum_{j=1}^n P_{ij}(t) \varphi_k^*(V x_j(t)) \overline{\varphi_k^*(x_i(t))} \right) \\ &= 2 \cdot \operatorname{Re} \left( \sum_{j=1}^n P_{ij}(t) \lambda_k \varphi_k^*(x_j(t)) \overline{\varphi_k^*(x_i(t))} \right) \\ &\le 2 |\lambda_k| \max_{j \in [n]} |\varphi_k^*(x_j(t))|^2. \end{aligned} $$

Choosing $i \in [n]$ running over the set of indices such that $|\varphi_k^*(x_i(t))|$ is maximal, we obtain

$$ \frac{d}{dt} \max_{j \in [n]} |\varphi_k^*(x_j(t))|^2 \le 2 |\lambda_k| \max_{j \in [n]} |\varphi_k^*(x_j(t))|^2. $$

We conclude the proof by applying Grönwall's lemma. $\square$

<figure><img src="image_19.png" /></figure>
**9.1. Some monotonicity properties and bounds.** To start, we present some general facts that are prove useful in all subsequent sub-cases.

**Lemma 9.1.** Suppose $k \in [d]$ is such that $\lambda_k \ge 0$. Then $t \mapsto \max_{j \in [n]} \varphi_k^*(z_j(t))$ is a non-increasing and bounded function, and $t \mapsto \min_{j \in [n]} \varphi_k^*(z_j(t))$ is a non-decreasing and bounded function. In particular, $t \mapsto \varphi_k^*(z_i(t))$ is uniformly bounded as a function on $[0, +\infty)$ for any $i \in [n]$.

*Proof.* For any $k \in [d]$ and any $t \ge 0$, set

$$ \alpha_k(t) = \min_{j \in [n]} \varphi_k^*(z_j(t)), \quad \beta_k(t) = \max_{j \in [n]} \varphi_k^*(z_j(t)). $$

Let $i \in [n]$ be an index such that $\alpha_k(t) = \varphi_k^*(z_i(t))$. Then we have

$$ \begin{aligned} \frac{d}{dt} \varphi_k^*(z_i(t)) &= \sum_{j=1}^{n} P_{ij}(t) \varphi_k^*(V(z_j(t) - z_i(t))) \\ &= \lambda_k \sum_{j=1}^{n} P_{ij}(t) (\varphi_k^*(z_j(t)) - \varphi_k^*(z_i(t))) \ge 0 \end{aligned} $$

where the last inequality stems from the fact that $\lambda_k \ge 0$ and the choice of index $i$. This proves that $\alpha_k(\cdot)$ is non-decreasing, as desired. Arguing similarly, one finds that $\beta_k(\cdot)$ is non-increasing. As a consequence, $\alpha_k(0) \le \alpha_k(t) \le \beta_k(t) \le \beta_k(0)$ for any $t \ge 0$, which shows that $\alpha_k(\cdot)$ and $\beta_k(\cdot)$ are bounded. $\square$

**Corollary 9.2.** If $V$ only has real non-negative eigenvalues, then $z_i(\cdot) \in L^\infty([0, +\infty))$.

**Lemma 9.3.** Fix $k \in [d]$ and $i \in [n]$. Then there exists a constant $C > 0$ such that

$$ |\varphi_k^*(e^{tV}z_i(t))| \le C e^{\lambda_k |t|} $$

holds for all $t \ge 0$.

*Proof.* We naturally make use of the equation for $x_i(t) := e^{tV}z_i(t)$. Fix $t \ge 0$. We have

$$ \begin{aligned} \frac{d}{dt} |\varphi_k^*(x_i(t))|^2 &= 2 \cdot \operatorname{Re} \left( \overline{\varphi_k^*(x_i(t))} \frac{d}{dt} \varphi_k^*(x_i(t)) \right) \\ &= 2 \cdot \operatorname{Re} \left( \sum_{j=1}^n P_{ij}(t) \varphi_k^*(V x_j(t)) \overline{\varphi_k^*(x_i(t))} \right) \\ &= 2 \cdot \operatorname{Re} \left( \sum_{j=1}^n P_{ij}(t) \lambda_k \varphi_k^*(x_j(t)) \overline{\varphi_k^*(x_i(t))} \right) \\ &\le 2 |\lambda_k| \max_{j \in [n]} |\varphi_k^*(x_j(t))|^2. \end{aligned} $$

Choosing $i \in [n]$ running over the set of indices such that $|\varphi_k^*(x_i(t))|$ is maximal, we obtain

$$ \frac{d}{dt} \max_{j \in [n]} |\varphi_k^*(x_j(t))|^2 \le 2 |\lambda_k| \max_{j \in [n]} |\varphi_k^*(x_j(t))|^2. $$

We conclude the proof by applying Grönwall's lemma. $\square$

<figure><img src="image_20.png" /></figure>
9.2. **Proof of Theorem 4.2.** We now prove Theorem 4.2. We again recall that $\lambda_1$ is simple and positive, and the eigenvalues of $V$ are ordered in decreasing order of modulus: $\lambda_1 > |\lambda_2| \ge \dots \ge |\lambda_d|$.

*Proof of Theorem 4.2.* We look to prove that for any $i \in [n]$, the component of $z_i(t)$ along the principal eigenvector $\varphi_1$, i.e. $\varphi_1^*(z_i(t))$, converges as $t \to +\infty$. We also show that there exists a set of at most 3 real numbers (depending on the initial datum $(z_1(0), \dots, z_n(0))$) such that for any $i \in [n]$ the limit of $\varphi_1^*(z_i(t))$ belongs to this set. Theorem 4.2 directly follows from these facts.

Let $i \in [n]$ be fixed. Recall from Lemma 9.1 that $\varphi_1^*(z_i(t))$ is uniformly bounded for any $t \in [0, +\infty)$. We set

$$a := \lim_{t \to +\infty} \min_{j \in [n]} \varphi_1^*(z_j(t)), \quad b := \lim_{t \to +\infty} \max_{j \in [n]} \varphi_1^*(z_j(t)). \quad (9.1)$$

(Note that by Lemma 9.1, $a \ge \min_{j \in [n]} \varphi_1^*(z_j(0))$ and $b \le \max_{j \in [n]} \varphi_1^*(z_j(0))$.) For $c \in \{0, a, b\}$, we define the candidate limiting hyperplanes for $z_i(t)$:

$$H_c := \{x \in \mathbb{R}^d : \varphi_1^*(x) = c\}.$$

We show that $z_i(t)$ converges either to $H_0$, to $H_a$ or to $H_b$. If $a = b = 0$, then according to (9.1) all particles converge to $H_0$ and there is nothing left to prove. We now distinguish two scenarios:

(i) either for any $\varepsilon > 0$, $|\varphi_1^*(z_i(t))| \le \varepsilon$ for $t$ large enough—in which case, we deduce that $z_i(t)$ converges toward $H_0$ as $t \to +\infty$—,

(ii) or $|\varphi_1^*(z_i(t_k))| > \varepsilon_0$ for some $\varepsilon_0 > 0$ and for some sequence of positive times $\{t_k\}_{k=1}^{+\infty}$ with $t_k \to +\infty$.

Since case (i) is straightforward, let us handle case (ii). Without loss of generality, we can extract a subsequence of times (which we do not relabel, for simplicity of notation) along which

$$\varphi_1^*(z_i(t_k)) > \varepsilon_0. \quad (9.2)$$

Let $\varepsilon \in (0, \varepsilon_0]$ be fixed and to be chosen later. We set

$$w_j(t) := \langle Q e^{tV} z_i(t), K e^{tV} z_j(t) \rangle,$$

so that

$$\frac{1}{\lambda_1} \frac{d}{dt} \varphi_1^*(z_i(t)) = \sum_{j=1}^{n} \frac{e^{w_j(t)}}{\sum_{k=1}^{n} e^{w_k(t)}} (\varphi_1^*(z_j(t)) - \varphi_1^*(z_i(t))). \quad (9.3)$$

We look to obtain a lower bound for the right-hand side in the above identity. Let us use the shorthand

$$c_{k\ell} := \langle Q\varphi_k, K\varphi_\ell \rangle$$

for $k, \ell \in [d]$. By assumption, $c_{11} > 0$. We have $\varphi_k^*(e^{tV}z_i(t)) = e^{t\lambda_k}\varphi_k^*(z_i(t))$ and the following spectral expansion holds:

$$e^{tV} z_i(t) = \sum_{k=1}^{d} e^{t\lambda_k} \varphi_k^*(z_i(t)) \varphi_k.$$
9.2. **Proof of Theorem 4.2.** We now prove Theorem 4.2. We again recall that $\lambda_1$ is simple and positive, and the eigenvalues of $V$ are ordered in decreasing order of modulus: $\lambda_1 > |\lambda_2| \ge \dots \ge |\lambda_d|$.

*Proof of Theorem 4.2.* We look to prove that for any $i \in [n]$, the component of $z_i(t)$ along the principal eigenvector $\varphi_1$, i.e. $\varphi_1^*(z_i(t))$, converges as $t \to +\infty$. We also show that there exists a set of at most 3 real numbers (depending on the initial datum $(z_1(0), \dots, z_n(0))$) such that for any $i \in [n]$ the limit of $\varphi_1^*(z_i(t))$ belongs to this set. Theorem 4.2 directly follows from these facts.

Let $i \in [n]$ be fixed. Recall from Lemma 9.1 that $\varphi_1^*(z_i(t))$ is uniformly bounded for any $t \in [0, +\infty)$. We set

$$a := \lim_{t \to +\infty} \min_{j \in [n]} \varphi_1^*(z_j(t)), \quad b := \lim_{t \to +\infty} \max_{j \in [n]} \varphi_1^*(z_j(t)). \quad (9.1)$$

(Note that by Lemma 9.1, $a \ge \min_{j \in [n]} \varphi_1^*(z_j(0))$ and $b \le \max_{j \in [n]} \varphi_1^*(z_j(0))$.) For $c \in \{0, a, b\}$, we define the candidate limiting hyperplanes for $z_i(t)$:

$$H_c := \{x \in \mathbb{R}^d : \varphi_1^*(x) = c\}.$$

We show that $z_i(t)$ converges either to $H_0$, to $H_a$ or to $H_b$. If $a = b = 0$, then according to (9.1) all particles converge to $H_0$ and there is nothing left to prove. We now distinguish two scenarios:

(i) either for any $\varepsilon > 0$, $|\varphi_1^*(z_i(t))| \le \varepsilon$ for $t$ large enough—in which case, we deduce that $z_i(t)$ converges toward $H_0$ as $t \to +\infty$—,

(ii) or $|\varphi_1^*(z_i(t_k))| > \varepsilon_0$ for some $\varepsilon_0 > 0$ and for some sequence of positive times $\{t_k\}_{k=1}^{+\infty}$ with $t_k \to +\infty$.

Since case (i) is straightforward, let us handle case (ii). Without loss of generality, we can extract a subsequence of times (which we do not relabel, for simplicity of notation) along which

$$\varphi_1^*(z_i(t_k)) > \varepsilon_0. \quad (9.2)$$

Let $\varepsilon \in (0, \varepsilon_0]$ be fixed and to be chosen later. We set

$$w_j(t) := \langle Q e^{tV} z_i(t), K e^{tV} z_j(t) \rangle,$$

so that

$$\frac{1}{\lambda_1} \frac{d}{dt} \varphi_1^*(z_i(t)) = \sum_{j=1}^{n} \frac{e^{w_j(t)}}{\sum_{k=1}^{n} e^{w_k(t)}} (\varphi_1^*(z_j(t)) - \varphi_1^*(z_i(t))). \quad (9.3)$$

We look to obtain a lower bound for the right-hand side in the above identity. Let us use the shorthand

$$c_{k\ell} := \langle Q\varphi_k, K\varphi_\ell \rangle$$

for $k, \ell \in [d]$. By assumption, $c_{11} > 0$. We have $\varphi_k^*(e^{tV}z_i(t)) = e^{t\lambda_k}\varphi_k^*(z_i(t))$ and the following spectral expansion holds:

$$e^{tV} z_i(t) = \sum_{k=1}^{d} e^{t\lambda_k} \varphi_k^*(z_i(t)) \varphi_k.$$
Using this fact, as well as Lemma 9.3, we gather that

$$
\begin{align*}
|w_j(t) - c_{11} e^{2\lambda_1 t} \varphi_1^*(z_i(t)) \varphi_1^*(z_j(t))| &= \left| \sum_{(k,\ell) \neq (1,1)} c_{k\ell} \varphi_k^*(e^{tV} z_i(t)) \varphi_\ell^*(e^{tV} z_j(t)) \right| \\
&\le \sum_{(k,\ell) \neq (1,1)} |c_{k\ell}| |\varphi_k^*(e^{tV} z_i(t))| |\varphi_\ell^*(e^{tV} z_j(t))| \\
&\le C^2 \|Q^\top K\|_{\text{op}} \sum_{(k,\ell) \neq (1,1)} e^{(|\lambda_k| + |\lambda_\ell|)t} \\
&\le \underbrace{C^2 \|Q^\top K\|_{\text{op}} (d-1)^2}_{=:C'} e^{(\lambda_1 + |\lambda_2|)t} \tag{9.4}
\end{align*}
$$

holds for all $t \ge 0$ and $j \in [n]$. Now since $\lambda_1 > 0$, Lemma 9.1 implies that for any $t \ge 0$ there exists an index $i_0(t) \in [n]$ such that

$$
\varphi_1^*(z_{i_0}(t)) \ge b. \tag{9.5}
$$

With $j_0(t) \in \operatorname{argmax}_{j \in [n]} w_j(t)$, using (9.4) and (9.5) we see that

$$
w_{j_0}(t) \ge w_{i_0}(t) \ge c_{11} \varphi_1^*(z_i(t)) b e^{2\lambda_1 t} - C' e^{(\lambda_1 + |\lambda_2|)t}. \quad (9.6)
$$

Now for any $t$ within the sequence $\{t_k\}_{k=1}^{+\infty}$, combining the first inequality in (9.6)
with the fact that $c_{11} > 0$, (9.2) and (9.4), we deduce that

$$
\varphi_1^*(z_{j_0}(t)) - \varphi_1^*(z_{i_0}(t)) \geq - \frac{2C'}{c_{11}\varepsilon} e^{-(\lambda_1 - |\lambda_2|)t}. \quad (9.7)
$$

As $\lambda_1 > |\lambda_2|$, for $t$ large enough, we find that we can lower bound the above expression by $-\frac{\varepsilon}{4}$. We now define the set of indices

$$
N(t) := \{j \in [n] : \varphi_1^*(z_i(t)) - \varphi_1^*(z_j(t)) \ge 0\}.
$$

Take $t$ within the sequence $\{t_k\}_{k=1}^{+\infty}$ such that $\varphi_1^*(z_i(t)) \le b - \varepsilon$ and large enough so
that (9.7) is lower bounded by $-\frac{\varepsilon}{4}$ (if such a $t$ does not exist, we immediately con-
clude that $\varphi_1^*(z_i(t)) \to b$ as $t \to +\infty$). Using (9.5) and the subsequent derivations,
we deduce that

$$
\varphi_1^*(z_{j_0}(t)) - \varphi_1^*(z_i(t)) \ge \frac{3\varepsilon}{4},
$$

and since $\varphi_1^*(z_j(t)) - \varphi_1^*(z_i(t)) \ge 0$ for $j \notin N(t)$, we expand in (9.3) to get

$$
\begin{equation}
\begin{split}
\frac{1}{\lambda_1} \frac{d}{dt} \varphi_1^*(z_i(t)) \geq & \frac{e^{w_{j_0}(t)}}{\sum_{k=1}^n e^{w_k(t)}} \frac{3\varepsilon}{4} + \sum_{j \in N(t)} \frac{e^{w_j(t)}}{\sum_{k=1}^n e^{w_k(t)}} (\varphi_1^*(z_j(t)) - \varphi_1^*(z_i(t))).
\end{split}
\tag{9.8}
\end{equation}
$$

On another hand, for $j \in N(t)$, we may use (9.4) to find

$$
w_j(t) \le c_{11} \varphi_1^*(z_i(t))^2 e^{2\lambda_1 t} + C' e^{(\lambda_1 + |\lambda_2|)t}. \quad (9.9)
$$

We set

$$
C_0 := \max_{j \in [n]} \varphi_1^*(z_j(0)) - \min_{j \in [n]} \varphi_1^*(z_j(0)).
$$
Using this fact, as well as Lemma 9.3, we gather that

$$
\begin{align*}
|w_j(t) - c_{11}e^{2\lambda_1 t} \varphi_1^*(z_i(t)) \varphi_1^*(z_j(t))| &= \left| \sum_{(k,\ell) \neq (1,1)} c_{k\ell} \varphi_k^*(e^{tV}z_i(t)) \varphi_\ell^*(e^{tV}z_j(t)) \right| \\
&\le \sum_{(k,\ell) \neq (1,1)} |c_{k\ell}| |\varphi_k^*(e^{tV}z_i(t))| |\varphi_\ell^*(e^{tV}z_j(t))| \\
&\le C^2 \|Q^\top K\|_{\text{op}} \sum_{(k,\ell) \neq (1,1)} e^{(|\lambda_k|+|\lambda_\ell|)t} \\
&\le \underbrace{C^2 \|Q^\top K\|_{\text{op}} (d-1)^2}_{=:C'} e^{(\lambda_1+|\lambda_2|)t} \tag{9.4}
\end{align*}
$$

holds for all $t \ge 0$ and $j \in [n]$. Now since $\lambda_1 > 0$, Lemma 9.1 implies that for any $t \ge 0$ there exists an index $i_0(t) \in [n]$ such that

$$
\varphi_1^*(z_{i_0}(t)) \ge b. \tag{9.5}
$$

With $j_0(t) \in \argmax_{j \in [n]} w_j(t)$, using (9.4) and (9.5) we see that

$$
w_{j_0}(t) \ge w_{i_0}(t) \ge c_{11} \varphi_1^*(z_i(t)) b e^{2\lambda_1 t} - C' e^{(\lambda_1 + |\lambda_2|)t}. \quad (9.6)
$$

Now for any $t$ within the sequence $\{t_k\}_{k=1}^{+\infty}$, combining the first inequality in (9.6)
with the fact that $c_{11} > 0$, (9.2) and (9.4), we deduce that

$$
\varphi_1^*(z_{j_0}(t)) - \varphi_1^*(z_{i_0}(t)) \geq - \frac{2C'}{c_{11}\varepsilon} e^{-(\lambda_1 - |\lambda_2|)t}. \quad (9.7)
$$

As $\lambda_1 > |\lambda_2|$, for $t$ large enough, we find that we can lower bound the above expression by $-\frac{\varepsilon}{4}$. We now define the set of indices

$$
N(t) := \{j \in [n] : \varphi_1^*(z_i(t)) - \varphi_1^*(z_j(t)) \ge 0\}.
$$

Take $t$ within the sequence $\{t_k\}_{k=1}^{+\infty}$ such that $\varphi_1^*(z_i(t)) \le b - \varepsilon$ and large enough so
that (9.7) is lower bounded by $-\frac{\varepsilon}{4}$ (if such a $t$ does not exist, we immediately con-
clude that $\varphi_1^*(z_i(t)) \to b$ as $t \to +\infty$). Using (9.5) and the subsequent derivations,
we deduce that

$$
\varphi_1^*(z_{j_0}(t))(t)) - \varphi_1^*(z_i(t))(t)) \ge \frac{3\varepsilon}{4},
$$

and since $\varphi_1^*(z_j(t)) - \varphi_1^*(z_i(t)) \ge 0$ for $j \notin N(t)$, we expand in (9.3) to get

$$
\begin{equation}
\begin{split}
\frac{1}{\lambda_1} \frac{d}{dt} \varphi_1^*(z_i(t)) \geq & \frac{e^{w_{j_0}(t)}}{\sum_{k=1}^n e^{w_k(t)}} \frac{3\varepsilon}{4} + \sum_{j \in N(t)} \frac{e^{w_j(t)}}{\sum_{k=1}^n e^{w_k(t)}} (\varphi_1^*(z_j(t)) - \varphi_1^*(z_i(t))).
\end{split}
\tag{9.8}
\end{equation}
$$

On another hand, for $j \in N(t)$, we may use (9.4) to find

$$
w_j(t) \le c_{11} \varphi_1^*(z_i(t))^2 e^{2\lambda_1 t} + C' e^{(\lambda_1 + |\lambda_2|)t}. \quad (9.9)
$$

We set

$$
C_0 := \max_{j \in [n]} \varphi_1^*(z_j(0)) - \min_{j \in [n]} \varphi_1^*(z_j(0)).
$$
Using the monotonicity properties from Lemma 9.1, as well as (9.9) in (9.8), we obtain

$$ \frac{1}{\lambda_1} \frac{d}{dt} \varphi_1^*(z_i(t)) \geq \frac{3\varepsilon}{4n} - C_0 n \frac{\exp\left(c_{11} \varphi_1^*(z_i(t))^2 e^{2\lambda_1 t} + C' e^{(\lambda_1 + |\lambda_2|)t}\right)}{\exp\left(c_{11} \varphi_1^*(z_i(t)) b e^{2\lambda_1 t} - C' e^{(\lambda_1 + |\lambda_2|)t}\right)}. $$

Given our choice of $t$, we have $\varphi_1^*(z_i(t))^2 - b\varphi_1^*(z_i(t)) \leq -\varepsilon(b - \varepsilon)$, so, we conclude from the inequality just above that

$$ \frac{1}{\lambda_1} \frac{d}{dt} \varphi_1^*(z_i(t)) \geq \frac{3\varepsilon}{4n} - C_0 n \exp \left( -c_{11}\varepsilon(b-\varepsilon)e^{2\lambda_1 t} + 2C' e^{(\lambda_1 + |\lambda_2|)t} \right). \quad (9.10) $$

Since $\lambda_1 > |\lambda_2|$, it follows from (9.10) that there exists $T > 0$ such that for any $t$ within the sequence $\{t_k\}_{k=1}^{+\infty}$ for which $t \geq T$ and $\varphi_1^*(z_i(t)) \in [\varepsilon, b - \varepsilon]$, there holds

$$ \frac{d}{dt} \varphi_1^*(z_i(t)) \geq \frac{\lambda_1 \varepsilon}{2n}. $$

This shows the existence of a larger time horizon $T' > T$ such that $\varphi_1^*(z_i(t)) \geq b - \varepsilon$ whenever $t \geq T'$. And since $\varepsilon$ can be taken arbitrarily small, we deduce that $\varphi_1^*(z_i(t))$ converges toward $b$, namely that $z_i(t)$ converges toward $H_b$, as $t \to +\infty$.

Arguing in the same way as above, and assuming without loss of generality that $a < 0$, we may find that all indices $i \in [n]$ for which $\varphi_1^*(z_i(t_k)) \leq -\varepsilon_0$ for some $\varepsilon_0 > 0$ and some sequence $t_k \to +\infty$, the particle $z_i(t)$ converges toward $H_a$ as $t \to +\infty$. This concludes the proof. $\square$

### 9.3. Remarks.

**Remark 9.4.** Theorem 4.2 establishes the convergence of $\varphi_1^*(z_i(t))$ for any $i \in [n]$ as $t \to +\infty$, but does not preclude the fact that $\|z_i(t)\|$ may diverge toward $+\infty$ (along the hyperplane) as $t \to +\infty$. This is indeed expected (and observed numerically—see Fig. 6) when $V$ has some negative eigenvalues. We also note that when all the eigenvalues of $V$ are non-negative, Corollary 9.2 shows that all the $z_i(t)$ remain bounded.

**Remark 9.5** (The case where $V$ is not diagonalizable). If $V$ is not assumed to be diagonalizable, Lemma 9.3 (or, at least the proof thereof) requires some modifications. Let $\delta := \lambda_1 - |\lambda_2| > 0$. Let $\varepsilon > 0$ be fixed and to be chosen later. We decompose $V$ in Jordan blocks, and we consider

$$ \mathbb{C}^d = \bigoplus_{k=1}^{m} \mathcal{F}_k, \quad (9.11) $$

where $\mathcal{F}_k$ is the span of the Jordan chain corresponding to the $k$-th Jordan block. By a slight abuse of notation (solely for the purpose of this remark), we denote by $\lambda_k$ the eigenvalue associated to the $k$-th Jordan block. We recall that we can choose a basis $(\varphi_{k,1}, \dots, \varphi_{k,j_k})$ of each $\mathcal{F}_k$ in a way that $V|_{\mathcal{F}_k}$ reads in this basis as$^7$

$$ \begin{bmatrix} \lambda_k & \varepsilon \\ \vdots & \vdots \\ \vdots & \varepsilon \\ \lambda_k \end{bmatrix}. \quad (9.12) $$

<footnote>$^7$Recall that Jordan blocks are commonly written with a +1 in the superdiagonal. This can be replaced by any non-zero complex scalar as done here—see [HJ12, Chapter 3, Corollary 3.1.21].</footnote>
Using the monotonicity properties from Lemma 9.1, as well as (9.9) in (9.8), we obtain

$$ \frac{1}{\lambda_1} \frac{d}{dt} \varphi_1^*(z_i(t)) \geq \frac{3\varepsilon}{4n} - C_0 n \frac{\exp\left(c_{11} \varphi_1^*(z_i(t))^2 e^{2\lambda_1 t} + C' e^{(\lambda_1 + |\lambda_2|)t}\right)}{\exp\left(c_{11} \varphi_1^*(z_i(t)) b e^{2\lambda_1 t} - C' e^{(\lambda_1 + |\lambda_2|)t}\right)}. $$

Given our choice of $t$, we have $\varphi_1^*(z_i(t))^2 - b\varphi_1^*(z_i(t)) \leq -\varepsilon(b - \varepsilon)$, so, we conclude from the inequality just above that

$$ \frac{1}{\lambda_1} \frac{d}{dt} \varphi_1^*(z_i(t)) \geq \frac{3\varepsilon}{4n} - C_0 n \exp \left( -c_{11}\varepsilon(b-\varepsilon)e^{2\lambda_1 t} + 2C' e^{(\lambda_1 + |\lambda_2|)t} \right). \quad (9.10) $$

Since $\lambda_1 > |\lambda_2|$, it follows from (9.10) that there exists $T > 0$ such that for any $t$ within the sequence $\{t_k\}_{k=1}^{+\infty}$ for which $t \geq T$ and $\varphi_1^*(z_i(t)) \in [\varepsilon, b - \varepsilon]$, there holds

$$ \frac{d}{dt} \varphi_1^*(z_i(t)) \geq \frac{\lambda_1 \varepsilon}{2n}. $$

This shows the existence of a larger time horizon $T' > T$ such that $\varphi_1^*(z_i(t)) \geq b - \varepsilon$ whenever $t \geq T'$. And since $\varepsilon$ can be taken arbitrarily small, we deduce that $\varphi_1^*(z_i(t))$ converges toward $b$, namely that $z_i(t)$ converges toward $H_b$, as $t \to +\infty$.

Arguing in the same way as above, and assuming without loss of generality that $a < 0$, we may find that all indices $i \in [n]$ for which $\varphi_1^*(z_i(t_k)) \leq -\varepsilon_0$ for some $\varepsilon_0 > 0$ and some sequence $t_k \to +\infty$, the particle $z_i(t)$ converges toward $H_a$ as $t \to +\infty$. This concludes the proof. $\square$

### 9.3. Remarks.

**Remark 9.4.** Theorem 4.2 establishes the convergence of $\varphi_1^*(z_i(t))$ for any $i \in [n]$ as $t \to +\infty$, but does not preclude the fact that $\|z_i(t)\|$ may diverge toward $+\infty$ (along the hyperplane) as $t \to +\infty$. This is indeed expected (and observed numerically—see Fig. 6) when $V$ has some negative eigenvalues. We also note that when all the eigenvalues of $V$ are non-negative, Corollary 9.2 shows that all the $z_i(t)$ remain bounded.

**Remark 9.5** (The case where $V$ is not diagonalizable). If $V$ is not assumed to be diagonalizable, Lemma 9.3 (or, at least the proof thereof) requires some modifications. Let $\delta := \lambda_1 - |\lambda_2| > 0$. Let $\varepsilon > 0$ be fixed and to be chosen later. We decompose $V$ in Jordan blocks, and we consider

$$ \mathbb{C}^d = \bigoplus_{k=1}^{m} \mathcal{F}_k, \qquad (9.11) $$

where $\mathcal{F}_k$ is the span of the Jordan chain corresponding to the $k$-th Jordan block. By a slight abuse of notation (solely for the purpose of this remark), we denote by $\lambda_k$ the eigenvalue associated to the $k$-th Jordan block. We recall that we can choose a basis $(\varphi_{k,1}, \dots, \varphi_{k,j_k})$ of each $\mathcal{F}_k$ in a way that $V|_{\mathcal{F}_k}$ reads in this basis as$^7$

$$ \begin{bmatrix} \lambda_k & \varepsilon \\ \vdots & \vdots \\ \vdots & \varepsilon \\ \lambda_k \end{bmatrix}. \qquad (9.12) $$

<footnote>$^7$Recall that Jordan blocks are commonly written with a +1 in the superdiagonal. This can be replaced by any non-zero complex scalar as done here—see [HJ12, Chapter 3, Corollary 3.1.21].</footnote>
We observe that if $\epsilon$ is chosen sufficiently small (depending only on $\delta$), Lemma 9.3 may be replaced by the following estimate in each $\mathcal{F}_k$:

$$ \exists C > 0, \forall t \ge 0, \forall i \in [n], \quad \|\pi_{\mathcal{F}_k}(e^{tV}z_i(t))\| \le C e^{(|\lambda_k|+\delta)t}. \quad (9.13) $$

Here, $\pi_{\mathcal{F}_k}$ denotes the orthogonal projection onto $\mathcal{F}_k$. To prove estimate (9.13), we follow the proof of Lemma 9.3, with $\frac{d}{dt}\|\pi_{\mathcal{F}_k}(x_i(t))\|^2$ playing the role of $\frac{d}{dt}|\varphi_k^*(x_i(t))|^2$. The key observation is that combining (9.11) and (9.12) we obtain

$$ \|\pi_{\mathcal{F}_k}(Vx_i(t))\| \le (|\lambda_k| + \delta)\|\pi_{\mathcal{F}_k}(x_i(t))\|, $$

provided $\epsilon$ is chosen sufficiently small. Then (9.13) follows as in Lemma 9.3.

With (9.11) at hand, the proof of Theorem 4.2 carries through, under the implicit modification that $Ce^{(\lambda_1+|\lambda_2|+\delta)t}$ replaces (9.4) (and subsequent estimates are modified in the same way).

## 10. PROOF OF THEOREM 5.2

In this section, we establish the proof for Theorem 5.2. Since the proof is essentially a combination of the proofs of Theorems 4.2 and 8.1, we may occasionally skip certain details and refer to the proofs of these two results. As done throughout this work, we set

$$ A := (Q^\top K)^{\frac{1}{2}}. $$

We denote by $\pi_{\mathcal{F}}: \mathbb{R}^d \to \mathcal{F}$ the projection onto $\mathcal{F}$ parallel to $\mathcal{G}$, and by $\pi_{\mathcal{G}}: \mathbb{R}^d \to \mathcal{G}$ the projection onto $\mathcal{G}$ parallel to $\mathcal{F}$. The set $\pi_{\mathcal{F}}(\text{conv}(\{z_i(t)\}_{i \in [n]}))$ is a convex subset of $\mathcal{F}$ which is non-increasing with respect to $t$ (the proof of this fact is identical to that of Proposition 8.2). It therefore converges toward some convex polytope $\mathcal{K}$ as $t \to +\infty$.

Fix $i \in [n]$. We have

$$ \begin{aligned} \pi_{\mathcal{F}}(\dot{z}_i(t)) &= \sum_{j=1}^{n} \left( \frac{e^{\langle Ae^{tV}z_i(t), Ae^{tV}z_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle Ae^{tV}z_i(t), Ae^{tV}z_k(t) \rangle}} \right) \pi_{\mathcal{F}}(V(z_j(t) - z_i(t))) \\ &= \sum_{j=1}^{n} \left( \frac{e^{\langle Ae^{tV}z_i(t), Ae^{tV}(z_j(t)-z_i(t)) \rangle}}{\sum_{k=1}^{n} e^{\langle Ae^{tV}z_i(t), Ae^{tV}(z_k(t)-z_i(t)) \rangle}} \right) \pi_{\mathcal{F}}(V(z_j(t) - z_i(t))). \end{aligned} $$

From this point on, we follow the proof of Theorem 8.1, and we solely highlight the changes compared to the original proof. Roughly speaking, this new proof amounts to adding projections $\pi_{\mathcal{F}}$ at several places. We denote by $\mathcal{S} \subset \mathcal{F}$ the set of points $w \in \mathcal{K}$ such that

$$ \|\pi_{\mathcal{F}}(Aw)\|^2 = \max_{j \in [m]} \langle \pi_{\mathcal{F}}(Aw), \pi_{\mathcal{F}}(Av_j) \rangle. $$

The fact that $\mathcal{S} \subset \partial\mathcal{K}$ and that $\mathcal{S}$ has finite cardinality is proved precisely as Claim 1 (in the proof of Theorem 8.1), simply by replacing all occurrences of $A$ by $\pi_{\mathcal{F}}(A\cdot)$. Once again, $\mathcal{S}_{\delta}$ denotes the set of all points in $\mathcal{K}$ at distance $\le \delta$ to some point of $\mathcal{S}$.

Step 2 in the proof of Theorem 8.1 (i.e., (8.7)) is replaced by the following statement:

**Step 2:** There exists a constant $\gamma = \gamma(\mathcal{K}) > 0$ (depending only on the geometry of $\mathcal{K}$) such that for any $\delta \in (0, \delta_0]$, there exists $T = T(\delta) > 0$ such that if $t \ge T$ and
We observe that if $\epsilon$ is chosen sufficiently small (depending only on $\delta$), Lemma 9.3 may be replaced by the following estimate in each $\mathcal{F}_k$:

$$ \exists C > 0, \forall t \ge 0, \forall i \in [n], \quad \|\pi_{\mathcal{F}_k}(e^{tV}z_i(t))\| \le C e^{(|\lambda_k|+\delta)t}. \quad (9.13) $$

Here, $\pi_{\mathcal{F}_k}$ denotes the orthogonal projection onto $\mathcal{F}_k$. To prove estimate (9.13), we follow the proof of Lemma 9.3, with $\frac{d}{dt}\|\pi_{\mathcal{F}_k}(x_i(t))\|^2$ playing the role of $\frac{d}{dt}|\varphi_k^*(x_i(t))|^2$. The key observation is that combining (9.11) and (9.12) we obtain

$$ \|\pi_{\mathcal{F}_k}(Vx_i(t))\| \le (|\lambda_k| + \delta)\|\pi_{\mathcal{F}_k}(x_i(t))\|, $$

provided $\epsilon$ is chosen sufficiently small. Then (9.13) follows as in Lemma 9.3.

With (9.11) at hand, the proof of Theorem 4.2 carries through, under the impactless modification that $Ce^{(\lambda_1+|\lambda_2|+\delta)t}$ replaces (9.4) (and subsequent estimates are modified in the same way).

## 10. PROOF OF THEOREM 5.2

In this section, we establish the proof for Theorem 5.2. Since the proof is essentially a combination of the proofs of Theorems 4.2 and 8.1, we may occasionally skip certain details and refer to the proofs of these two results. As done throughout this work, we set

$$ A := (Q^\top K)^{\frac{1}{2}}. $$

We denote by $\pi_{\mathcal{F}}: \mathbb{R}^d \to \mathcal{F}$ the projection onto $\mathcal{F}$ parallel to $\mathcal{G}$, and by $\pi_{\mathcal{G}}: \mathbb{R}^d \to \mathcal{G}$ the projection onto $\mathcal{G}$ parallel to $\mathcal{F}$. The set $\pi_{\mathcal{F}}(\text{conv}(\{z_i(t)\}_{i \in [n]}))$ is a convex subset of $\mathcal{F}$ which is non-increasing with respect to $t$ (the proof of this fact is identical to that of Proposition 8.2). It therefore converges toward some convex polytope $\mathcal{K}$ as $t \to +\infty$.

Fix $i \in [n]$. We have

$$ \begin{aligned} \pi_{\mathcal{F}}(\dot{z}_i(t)) &= \sum_{j=1}^{n} \left( \frac{e^{\langle Ae^{tV}z_i(t), Ae^{tV}z_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle Ae^{tV}z_i(t), Ae^{tV}z_k(t) \rangle}} \right) \pi_{\mathcal{F}}(V(z_j(t) - z_i(t))) \\ &= \sum_{j=1}^{n} \left( \frac{e^{\langle Ae^{tV}z_i(t), Ae^{tV}(z_j(t)-z_i(t)) \rangle}}{\sum_{k=1}^{n} e^{\langle Ae^{tV}z_i(t), Ae^{tV}(z_k(t)-z_i(t)) \rangle}} \right) \pi_{\mathcal{F}}(V(z_j(t) - z_i(t))). \end{aligned} $$

From this point on, we follow the proof of Theorem 8.1, and we solely highlight the changes compared to the original proof. Roughly speaking, this new proof amounts to adding projections $\pi_{\mathcal{F}}$ at several places. We denote by $\mathcal{S} \subset \mathcal{F}$ the set of points $w \in \mathcal{K}$ such that

$$ \|\pi_{\mathcal{F}}(Aw)\|^2 = \max_{j \in [m]} \langle \pi_{\mathcal{F}}(Aw), \pi_{\mathcal{F}}(Av_j) \rangle. $$

The fact that $\mathcal{S} \subset \partial\mathcal{K}$ and that $\mathcal{S}$ has finite cardinality is proved precisely as Claim 1 (in the proof of Theorem 8.1), simply by replacing all occurrences of $A$ by $\pi_{\mathcal{F}}(A\cdot)$. Once again, $\mathcal{S}_{\delta}$ denotes the set of all points in $\mathcal{K}$ at distance $\le \delta$ to some point of $\mathcal{S}$.

Step 2 in the proof of Theorem 8.1 (i.e., (8.7)) is replaced by the following statement:

**Step 2:** There exists a constant $\gamma = \gamma(\mathcal{K}) > 0$ (depending only on the geometry of $\mathcal{K}$) such that for any $\delta \in (0, \delta_0]$, there exists $T = T(\delta) > 0$ such that if $t \ge T$ and
$\pi_{\mathcal{F}}(z_i(t)) \notin S_{\delta}$, then

$$ \frac{d}{dt} \|\pi_{\mathcal{F}}(Az_i(t))\|^2 \geq \gamma\delta. $$

We now proceed in proving this statement.

*Proof of Step 2*. We set

$$ a_j(t) := \langle \pi_{\mathcal{F}}(Az_i(t)), \pi_{\mathcal{F}}(A(z_j(t) - z_i(t))) \rangle $$

and

$$ r_j(t) := \langle Ae^{tV}z_i(t), Ae^{tV}(z_j(t) - z_i(t)) \rangle - a_j(t)e^{2\lambda_1 t}. $$

We find

$$ \begin{aligned} \frac{1}{2} \frac{d}{dt} \|\pi_{\mathcal{F}}(Az_i(t))\|^2 &= \langle \pi_{\mathcal{F}}(Az_i(t)), \pi_{\mathcal{F}}(Az_i(t)) \rangle \\ &= \sum_{j=1}^{n} \left( \frac{e^{\langle Ae^{tV}z_i(t), Ae^{tV}z_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle Ae^{tV}z_i(t), Ae^{tV}z_k(t) \rangle}} \right) \langle \pi_{\mathcal{F}}(A(z_j(t) - z_i(t))), \pi_{\mathcal{F}}(Az_i(t)) \rangle \\ &= \sum_{j=1}^{n} \left( \frac{e^{\langle Ae^{tV}z_i(t), Ae^{tV}(z_j(t)-z_i(t)) \rangle}}{\sum_{k=1}^{n} e^{\langle Ae^{tV}z_i(t), Ae^{tV}(z_k(t)-z_i(t)) \rangle}} \right) \langle \pi_{\mathcal{F}}(A(z_j(t) - z_i(t))), \pi_{\mathcal{F}}(Az_i(t)) \rangle \\ &= \sum_{j=1}^{n} \underbrace{\left( \frac{e^{a_j(t)e^{2\lambda_1 t} + r_j(t)}}{\sum_{k=1}^{n} e^{a_k(t)e^{2\lambda_1 t} + r_k(t)}} \right)}_{=: b_j(t)} a_j(t). \end{aligned} \quad (10.1) $$

We now make use of the following adaptation of Claim 2.

**Claim 3.** There exists some constant $\gamma' = \gamma'(K) > 0$ depending only on the geometry of K such that the following holds. Fix $\delta \in (0, \delta_0]$. There exists $T = T(\delta) > 0$ such that if $t \ge T$ and $z_i(t) \notin S_{\delta} \times \mathcal{G}$, then there exists $j \in [n]$ such that $a_j(t) \ge \gamma'\delta$.

Compared to Step 2 in the proof of Theorem 8.1, we now have to estimate the coefficients $r_j(t)$. To this end, setting $y_j(t) := Ae^{tV}z_j(t)$ for $j \in [n]$, we notice that $r_j(t) = P_1(t) + P_2(t) + P_3(t)$ where

$$ P_1(t) = \langle \pi_{\mathcal{F}}(y_i(t)), \pi_{\mathcal{G}}(y_j(t) - y_i(t)) \rangle, $$

$$ P_2(t) = \langle \pi_{\mathcal{G}}(y_i(t)), \pi_{\mathcal{F}}(y_j(t) - y_i(t)) \rangle, $$

$$ P_3(t) = \langle \pi_{\mathcal{G}}(y_i(t)), \pi_{\mathcal{G}}(y_j(t) - y_i(t)) \rangle. $$

By virtue of Lemma 9.3 we have $|\pi_{\mathcal{F}}(y_j(t))| \le Ce^{\lambda_1 t}$ and $|\pi_{\mathcal{G}}(y_j(t))| \le Ce^{t|\lambda_2|}$ for any $t \ge 0$ (or $Ce^{t|\lambda_2|+\epsilon}$ if $V_{|\mathcal{G}}$ is not diagonalizable—see Remark 9.5), hence

$$ |r_j(t)| \le Ce^{t(\lambda_1 + |\lambda_2|)}. \quad (10.2) $$

Since $\pi_{\mathcal{F}}(z_j(t))$ is uniformly bounded in $t \in [0, +\infty)$ for any $j \in [n]$ due to Corollary 8.3, we get $a_j(\cdot) \in L^{\infty}(0, +\infty)$. So, we may set

$$ \kappa := \max_{j \in [n]} \sup_{t \ge 0} |a_j(t)|. $$

Let $t \ge 0$. We define

$$ B(t) := \{j \in [n] : a_j(t)e^{2\lambda_1 t} + r_j(t) \ge 0\}. $$

Let $j_0(t) \in \operatorname*{argmax}_{j \in [n]} (a_j(t)e^{2\lambda_1 t} + r_j(t))$. Note that $j_0(t) \in B(t)$ since

$$ a_{j_0}(t)e^{2\lambda_1 t} + r_{j_0}(t) \ge a_i(t)e^{2\lambda_1 t} + r_i(t) = 0. $$
$\pi_{\mathcal{F}}(z_i(t)) \notin S_{\delta}$, then

$$ \frac{d}{dt} \|\pi_{\mathcal{F}}(Az_i(t))\|^2 \geq \gamma\delta. $$

We now proceed in proving this statement.

*Proof of Step 2*. We set

$$ a_j(t) := \langle \pi_{\mathcal{F}}(Az_i(t)), \pi_{\mathcal{F}}(A(z_j(t) - z_i(t))) \rangle $$

and

$$ r_j(t) := \langle Ae^{tV}z_i(t), Ae^{tV}(z_j(t) - z_i(t)) \rangle - a_j(t)e^{2\lambda_1 t}. $$

We find

$$ \begin{aligned} \frac{1}{2} \frac{d}{dt} \|\pi_{\mathcal{F}}(Az_i(t))\|^2 &= \langle \pi_{\mathcal{F}}(Az_i(t)), \pi_{\mathcal{F}}(Az_i(t)) \rangle \\ &= \sum_{j=1}^{n} \left( \frac{e^{\langle Ae^{tV}z_i(t), Ae^{tV}z_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle Ae^{tV}z_i(t), Ae^{tV}z_k(t) \rangle}} \right) \langle \pi_{\mathcal{F}}(A(z_j(t) - z_i(t))), \pi_{\mathcal{F}}(Az_i(t)) \rangle \\ &= \sum_{j=1}^{n} \left( \frac{e^{\langle Ae^{tV}z_i(t), Ae^{tV}(z_j(t)-z_i(t)) \rangle}}{\sum_{k=1}^{n} e^{\langle Ae^{tV}z_i(t), Ae^{tV}(z_k(t)-z_i(t)) \rangle}} \right) \langle \pi_{\mathcal{F}}(A(z_j(t) - z_i(t))), \pi_{\mathcal{F}}(Az_i(t)) \rangle \\ &= \sum_{j=1}^{n} \underbrace{\left( \frac{e^{a_j(t)e^{2\lambda_1 t} + r_j(t)}}{\sum_{k=1}^{n} e^{a_k(t)e^{2\lambda_1 t} + r_k(t)}} \right)}_{=: b_j(t)} a_j(t). \end{aligned} \quad (10.1) $$

We now make use of the following adaptation of Claim 2.

**Claim 3.** There exists some constant $\gamma' = \gamma'(K) > 0$ depending only on the geometry of K such that the following holds. Fix $\delta \in (0, \delta_0]$. There exists $T = T(\delta) > 0$ such that if $t \ge T$ and $z_i(t) \notin S_{\delta} \times \mathcal{G}$, then there exists $j \in [n]$ such that $a_j(t) \ge \gamma'\delta$.

Compared to Step 2 in the proof of Theorem 8.1, we now have to estimate the coefficients $r_j(t)$. To this end, setting $y_j(t) := Ae^{tV}z_j(t)$ for $j \in [n]$, we notice that $r_j(t) = P_1(t) + P_2(t) + P_3(t)$ where

$$ P_1(t) = \langle \pi_{\mathcal{F}}(y_i(t)), \pi_{\mathcal{G}}(y_j(t) - y_i(t)) \rangle, $$

$$ P_2(t) = \langle \pi_{\mathcal{G}}(y_i(t)), \pi_{\mathcal{F}}(y_j(t) - y_i(t)) \rangle, $$

$$ P_3(t) = \langle \pi_{\mathcal{G}}(y_i(t)), \pi_{\mathcal{G}}(y_j(t) - y_i(t)) \rangle. $$

By virtue of Lemma 9.3 we have $|\pi_{\mathcal{F}}(y_j(t))| \le Ce^{\lambda_1 t}$ and $|\pi_{\mathcal{G}}(y_j(t))| \le Ce^{t|\lambda_2|}$ for any $t \ge 0$ (or $Ce^{t|\lambda_2|+\varepsilon}$ if $V_{|\mathcal{G}}$ is not diagonalizable—see Remark 9.5), hence

$$ |r_j(t)| \le Ce^{t(\lambda_1 + |\lambda_2|)}. \quad (10.2) $$

Since $\pi_{\mathcal{F}}(z_j(t))$ is uniformly bounded in $t \in [0, +\infty)$ for any $j \in [n]$ due to Corollary 8.3, we get $a_j(\cdot) \in L^{\infty}(0, +\infty)$. So, we may set

$$ \kappa := \max_{j \in [n]} \sup_{t \ge 0} |a_j(t)|. $$

Let $t \ge 0$. We define

$$ B(t) := \{j \in [n] : a_j(t)e^{2\lambda_1 t} + r_j(t) \ge 0\}. $$

Let $j_0(t) \in \underset{j \in [n]}{\operatorname{argmax}}(a_j(t)e^{2\lambda_1 t} + r_j(t))$. Note that $j_0(t) \in B(t)$ since

$$ a_{j_0}(t)e^{2\lambda_1 t} + r_{j_0}(t) \ge a_i(t)e^{2\lambda_1 t} + r_i(t) = 0. $$
We notice the following three properties:

• For $j = j_0(t)$, we have $b_{j_0(t)}(t) \ge \frac{a_{j_0(t)}(t)}{n}$ (recall the definition of $b_j$ in (10.1));

• for any $j \in B(t) \setminus \{j_0\}$, we have $b_j(t) \ge 0$;

• for any $j \notin B(t)$, we have

$$b_j(t) \ge -\kappa \exp \left( -a_{j_0}(t)e^{2\lambda_1 t} + Ce^{(\lambda_1 + |\lambda_2|)t} \right).$$

Indeed, using the fact that $j \in B(t)$ and (10.2), we find

$$\begin{aligned} \frac{\exp(a_j(t)e^{2\lambda_1 t} + r_j(t))}{\sum_{k=1}^{n} \exp(a_k(t)e^{2\lambda_1 t} + r_k(t))} &\le \frac{1}{\sum_{k=1}^{n} \exp(a_k(t)e^{2\lambda_1 t} + r_k(t))} \\ &\le \frac{1}{\exp(a_{j_0}(t)e^{2\lambda_1 t} + r_{j_0}(t))} \\ &\le \exp\left(-a_{j_0}(t)e^{2\lambda_1 t} + Ce^{(\lambda_1 + |\lambda_2|)t}\right). \end{aligned}$$

Making use of these properties in (10.1) yields the desired lower bound—indeed, if $t$ is sufficiently large and $z_i(t) \notin S_\delta \times \mathcal{G}$, we have $\{j \in [n]: a_j(t) \ge \gamma'\delta\} \ne \emptyset$ according to Claim 3, and so we deduce that

$$\frac{1}{2} \frac{d}{dt} \|Az_i(t)\|^2 \ge \frac{\gamma'\delta}{n} - \kappa n e^{-\gamma'\delta e^{2\lambda_1 t} + Ce^{(\lambda_1 + |\lambda_2|)t}}.$$

Taking $t$ possibly larger (and depending on $\delta$), we obtain the result of Step 2'. $\square$

Steps 3 and 4 in the proof of Theorem 8.1 are essentially unchanged—we replace all the occurrences of $\|A \cdot\|$ by $\|\pi_{\mathcal{F}}(A \cdot)\|$ (for instance in (8.13) and (8.14)). Although $\|Az_i(t)\|$ may not be uniformly bounded in $t$, it is important to note that $\|\pi_{\mathcal{F}}(Az_i(t))\|$ is uniformly bounded. Similarly, while $\dot{z}_i(t) \notin L^\infty([0, +\infty))$, we do have $\|\frac{d}{dt}\pi_{\mathcal{F}}(z_i(\cdot))\|_{L^\infty([0, +\infty))} < +\infty$. The sets $S_\delta$, $\mathcal{C}_k$ and $\mathcal{C}_k^r$ are replaced by $S_\delta \times \mathcal{G}$, $\mathcal{C}_k \times \mathcal{G}$ and $\mathcal{C}_k^r \times \mathcal{G}$ respectively. The conclusion is that $\|\pi_{\mathcal{F}}(Az_i(t))\|^2$ has to increase by at least

$$\frac{\gamma \delta^{\frac{1}{2}} (\delta^{\frac{1}{4}} - \delta)}{\|\dot{z}_i\|_{L^\infty([0, +\infty))}} \ge \frac{\delta^{\frac{3}{4}}}{2 \|\dot{z}_i\|_{L^\infty([0, +\infty))}} > 4R \|A\|_{\text{op}} \delta$$

during a travel from $\mathcal{C}_k \times \mathcal{G}$ to the complement of $\mathcal{C}_k^{\frac{1}{4}} \times \mathcal{G}$. As in the proof of Theorem 8.1 this implies that for any $i \in [n]$ there exists $s \in S$ such that $z_i(t)$ remains at distance at most $\delta$ away from $\{s\} \times \mathcal{G}$. This being true for any $\delta > 0$, we obtain the desired result.

## 11. NUMERICAL EXPERIMENTS

11.1. **Setup.** Unless indicated otherwise, all figures presented in this paper were generated by discretizing the underlying dynamics (either (1.1) or (3.1)) using a fourth order Runge-Kutta scheme with a step size of 0.1. All points in the initial sequence were drawn independently from the uniform distribution over the hypercube $[-5, 5]^d$. Random matrices (e.g., $Q, K, V$) have entries drawn independently from the uniform distribution on $[-1, 1]$. Codes and animated plots of all examples may be found online at
We notice the following three properties:

• For $j = j_0(t)$, we have $b_{j_0(t)}(t) \ge \frac{a_{j_0(t)}(t)}{n}$ (recall the definition of $b_j$ in (10.1));

• for any $j \in B(t) \setminus \{j_0\}$, we have $b_j(t) \ge 0$;

• for any $j \notin B(t)$, we have

$$b_j(t) \ge -\kappa \exp \left( -a_{j_0}(t)e^{2\lambda_1 t} + Ce^{(\lambda_1 + |\lambda_2|)t} \right).$$

Indeed, using the fact that $j \in B(t)$ and (10.2), we find

$$\begin{aligned}
\frac{\exp(a_j(t)e^{2\lambda_1 t} + r_j(t))}{\sum_{k=1}^{n} \exp(a_k(t)e^{2\lambda_1 t} + r_k(t))} &\le \frac{1}{\sum_{k=1}^{n} \exp(a_k(t)e^{2\lambda_1 t} + r_k(t))} \\
&\le \frac{1}{\exp(a_{j_0}(t)e^{2\lambda_1 t} + r_{j_0}(t))} \\
&\le \exp\left(-a_{j_0}(t)e^{2\lambda_1 t} + Ce^{(\lambda_1 + |\lambda_2|)t}\right).
\end{aligned}$$

Making use of these properties in (10.1) yields the desired lower bound—indeed, if $t$ is sufficiently large and $z_i(t) \notin S_\delta \times \mathcal{G}$, we have $\{j \in [n]: a_j(t) \ge \gamma'\delta\} \ne \emptyset$ according to Claim 3, and so we deduce that

$$\frac{1}{2} \frac{d}{dt} \|Az_i(t)\|^2 \ge \frac{\gamma'\delta}{n} - \kappa n e^{-\gamma'\delta e^{2\lambda_1 t} + Ce^{(\lambda_1 + |\lambda_2|)t}}.$$

Taking $t$ possibly larger (and depending on $\delta$), we obtain the result of Step 2'. $\square$

Steps 3 and 4 in the proof of Theorem 8.1 are essentially unchanged—we replace all the occurrences of $\|A \cdot\|$ by $\|\pi_{\mathcal{F}}(A \cdot)\|$ (for instance in (8.13) and (8.14)). Although $\|Az_i(t)\|$ may not be uniformly bounded in $t$, it is important to note that $\|\pi_{\mathcal{F}}(Az_i(t))\|$ is uniformly bounded. Similarly, while $\dot{z}_i(t) \notin L^\infty([0, +\infty))$, we do have $\|\frac{d}{dt}\pi_{\mathcal{F}}(z_i(\cdot))\|_{L^\infty([0, +\infty))} < +\infty$. The sets $S_\delta$, $\mathcal{C}_k$ and $\mathcal{C}_k^r$ are replaced by $S_\delta \times \mathcal{G}$, $\mathcal{C}_k \times \mathcal{G}$ and $\mathcal{C}_k^r \times \mathcal{G}$ respectively. The conclusion is that $\|\pi_{\mathcal{F}}(Az_i(t))\|^2$ has to increase by at least

$$\frac{\gamma \delta^{\frac{1}{2}} (\delta^{\frac{1}{4}} - \delta)}{\|\dot{z}_i\|_{L^\infty([0, +\infty))}} \ge \frac{\delta^{\frac{3}{4}}}{2 \|\dot{z}_i\|_{L^\infty([0, +\infty))}} > 4R \|A\|_{\text{op}} \delta$$

during a travel from $\mathcal{C}_k \times \mathcal{G}$ to the complement of $\mathcal{C}_k^{\frac{1}{4}} \times \mathcal{G}$. As in the proof of Theorem 8.1 this implies that for any $i \in [n]$ there exists $s \in S$ such that $z_i(t)$ remains at distance at most $\delta$ away from $\{s\} \times \mathcal{G}$. This being true for any $\delta > 0$, we obtain the desired result.

## 11. NUMERICAL EXPERIMENTS

11.1. **Setup.** Unless indicated otherwise, all figures presented in this paper were generated by discretizing the underlying dynamics (either (1.1) or (3.1)) using a fourth order Runge-Kutta scheme with a step size of 0.1. All points in the initial sequence were drawn independently from the uniform distribution over the hypercube $[-5, 5]^d$. Random matrices (e.g., $Q, K, V$) have entries drawn independently from the uniform distribution on $[-1, 1]$. Codes and animated plots of all examples may be found online at
https://github.com/borjanG/2023-transformers.

We now present some experiments which motivate some conjectures and claims made in what precedes.

## 11.2. Eigenvalues of ALBERT's value matrices.
In Figure 10 we illustrate the eigenvalues of the value matrices $V_h$ for a couple of heads $h$ in a pre-trained ALBERT model. We focus on ALBERT-xlarge-v2 available online at

https://huggingface.co/Albert-xlarge-v2.

This version uses 16 heads, with sequences of length $n = 256$ and tokens of dimension $d = 128$. While not all value matrices $V_h$ per head $h \in [16]$ satisfy the assumptions made in Section 4, we illustrate the eigenvalues of a couple of them which do.

<figure><img src="image_20.png" /><figcaption>**Figure 10.** The eigenvalues of $V_5$ and $V_{14}$ in the pre-trained ALBERT satisfy the eigenvalue assumption made in Definition 4.1. Furthermore, the second assumption made in Definition 4.1 is satisfied by $(Q_5, K_5)$ and $(Q_{14}, K_{14})$ (the inner products evaluated along the eigenvector of norm 1 equal 1.3060 and 0.6719 respectively). In other words, the triples $(Q_h, K_h, V_h)$ corresponding to heads $h = 5$ and $h = 14$ in ALBERT satisfy all the assumptions made in the statement of Theorem 4.2.</figcaption></figure>

## 11.3. Experiments related to Theorem 2.1.
We begin with the setup of Theorem 2.1, which we recall was proven to hold in the case $d = 1$. Herein we present a couple of examples (Figures 11 and 12) which elucidate the role that $d$ and $n$ appear to play in this fact.

Notably, as seen in Fig. 4, we believe that the conclusion of Theorem 2.1 could plausibly be extended to any $d > 1$, assuming $V > 0$.

## 11.4. Illustrating Theorem 4.2 in $\mathbb{R}^3$. 
To precisely illustrate the appearance of at most three hyperplanes in the setting of Theorem 4.2, we gave an example in $\mathbb{R}^2$. We expand on this and provide a couple of toy examples in $\mathbb{R}^3$ for the purpose of visualization (we recall that these are toy models, as Transformers in practice are high-dimensional), and namely focus in both examples on the case where the two latter eigenvalues are complex. In Fig. 14, we see the effect of having eigenvalues with a negative real part, and the complementary case is illustrated in Fig. 13.
https://github.com/borjanG/2023-transformers.

We now present some experiments which motivate some conjectures and claims made in what precedes.

## 11.2. Eigenvalues of ALBERT's value matrices.
In Figure 10 we illustrate the eigenvalues of the value matrices $V_h$ for a couple of heads $h$ in a pre-trained ALBERT model. We focus on ALBERT-xlarge-v2 available online at

https://huggingface.co/albert-xlarge-v2.

This version uses 16 heads, with sequences of length $n = 256$ and tokens of dimension $d = 128$. While not all value matrices $V_h$ per head $h \in [16]$ satisfy the assumptions made in Section 4, we illustrate the eigenvalues of a couple of them which do.

<figure><img src="image_21.png" /><figcaption>**Figure 10.** The eigenvalues of $V_5$ and $V_{14}$ in the pre-trained ALBERT satisfy the eigenvalue assumption made in Definition 4.1. Furthermore, the second assumption made in Definition 4.1 is satisfied by $(Q_5, K_5)$ and $(Q_{14}, K_{14})$ (the inner products evaluated along the eigenvector of norm 1 equal 1.3060 and 0.6719 respectively). In other words, the triples $(Q_h, K_h, V_h)$ corresponding to heads $h = 5$ and $h = 14$ in ALBERT satisfy all the assumptions made in the statement of Theorem 4.2.</figcaption></figure>

## 11.3. Experiments related to Theorem 2.1.
We begin with the setup of Theorem 2.1, which we recall was proven to hold in the case $d = 1$. Herein we present a couple of examples (Figures 11 and 12) which elucidate the role that $d$ and $n$ appear to play in this fact.

Notably, as seen in Fig. 4, we believe that the conclusion of Theorem 2.1 could plausibly be extended to any $d > 1$, assuming $V > 0$.

## 11.4. Illustrating Theorem 4.2 in $\mathbb{R}^3$. 
To precisely illustrate the appearance of at most three hyperplanes in the setting of Theorem 4.2, we gave an example in $\mathbb{R}^2$. We expand on this and provide a couple of toy examples in $\mathbb{R}^3$ for the purpose of visualization (we recall that these are toy models, as Transformers in practice are high-dimensional), and namely focus in both examples on the case where the two latter eigenvalues are complex. In Fig. 14, we see the effect of having eigenvalues with a negative real part, and the complementary case is illustrated in Fig. 13.
<figure><img src="image_21.png" /><figcaption>**Figure 11.** We expand on Fig. 3—for the same setup, consider $n = 100$. The sequence length $n$ does not appear to influence the rank of $P(t)$, which is expected since the rank of $P$ corresponds to the number of leaders.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>**Figure 12.** We consider $n = 40$, $Q = K = I_d$ and a random matrix $V > 0$ in dimensions $d = 10$ (first row), $d = 40$ (second row), and $d = 80$ (third row). The conclusion of Theorem 2.1 appears to transfer to the higher dimensional case, and this would actually follow from Conjecture 4.3 (should it hold).</figcaption></figure>

11.5. **Complementing Figure 7.** In Figure 7, we illustrate the appearance of clustering in high-dimension (the ALBERT setup: $n = 256$ and $d = 128$) for generic random matrices $(Q, K, V)$. The value matrix $V$ in question has 65 positive eigenvalues, and we show the conjectured convergence of the 65 coordinates along the corresponding eigenvectors to one of possibly 3 (generically 2) real scalars. In Figure 15, we complement this illustration by showing the possible oscillatory and divergent behavior of the remaining coordinates.
<figure><img src="image_22.png" /><figcaption>**Figure 11.** We expand on Fig. 3—for the same setup, consider $n = 100$. The sequence length $n$ does not appear to influence the rank of $P(t)$, which is expected since the rank of $P$ corresponds to the number of leaders.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>**Figure 12.** We consider $n = 40$, $Q = K = I_d$ and a random matrix $V > 0$ in dimensions $d = 10$ (first row), $d = 40$ (second row), and $d = 80$ (third row). The conclusion of Theorem 2.1 appears to transfer to the higher dimensional case, and this would actually follow from Conjecture 4.3 (should it hold).</figcaption></figure>

11.5. **Complementing Figure 7.** In Figure 7, we illustrate the appearance of clustering in high-dimension (the ALBERT setup: $n = 256$ and $d = 128$) for generic random matrices $(Q, K, V)$. The value matrix $V$ in question has 65 positive eigenvalues, and we show the conjectured convergence of the 65 coordinates along the corresponding eigenvectors to one of possibly 3 (generically 2) real scalars. In Figure 15, we complement this illustration by showing the possible oscillatory and divergent behavior of the remaining coordinates.
<figure><img src="image_23.png" /><figcaption>**Figure 15.** We complement Figure 7 and plot the variance of the set $\{\varphi_j^*(z_i(t)): i \in [n]\}$ of all coordinates $j$ corresponding to negative eigenvalues of $V$. We also show the mean along tokens of a couple of coordinates (white lines). Coordinates diverge rapidly to $\pm\infty$ over time $t$; y-axis is in log scale.</figcaption></figure>

# Part 3. Discussion and open questions

## 12. OUTLOOK

Several important directions regarding the mathematical theory of Transformers remain unexplored. An important extension of our work would amount to studying multi-headed Transformers—borrowing the notation from Remark 3.4, they amount to:

$$x_i^{[k+1]} = x_i^{[k]} + \Delta t \sum_{h=1}^{H} \sum_{j=1}^{n} \left( \frac{e^{\langle Q_h x_i^{[k]}, K_h x_j^{[k]} \rangle}}{\sum_{\ell=1}^{n} e^{\langle Q_h x_i^{[k]}, K_h x_\ell(k) \rangle}} \right) V_h x_j^{[k]}, \quad k \in \mathbb{N}.$$

For each $h \in [H]$ (corresponding to a different head), the weight matrices $Q_h, K_h, V_h$ are constant. Proofs regarding clustering or convergence of the self-attention matrix for such dynamics is an open problem. Preliminary numerical investigations seem to indicate that interesting clustering phenomena also occur in this context. A characterization or properties of optimal weights by invoking the optimal control correspondence in the spirit of [Wei17] is also an interesting avenue for future research.

We hereby list a couple of additional numerical experiments suggesting generalizations of our results, which we leave as open problems.

12.1. **Beyond $Q^\top K > 0$ in Theorems 3.1 and 5.2.** As seen throughout all the presented proofs, assumptions on the value matrix $V$ are significantly more rigid than assumptions on the matrices $Q$ and $K$. For instance, should the eigenvalue $\lambda$ with the largest real part of $V$ be negative, all rescaled tokens will diverge to infinity. Should $\lambda$ be complex, we do not expect any clustering to occur (for the
<figure><img src="image_24.png" /><figcaption>Figure 15. We complement Figure 7 and plot the variance of the set {$\varphi_j^*(z_i(t)): i \in [n]$} of all coordinates j corresponding to negative eigenvalues of V. We also show the mean along tokens of a couple of coordinates (white lines). Coordinates diverge rapidly to $\pm\infty$ over time t; y-axis is in log scale.</figcaption></figure>

# Part 3. Discussion and open questions

## 12. OUTLOOK

Several important directions regarding the mathematical theory of Transformers remain unexplored. An important extension of our work would amount to studying multi-headed Transformers—borrowing the notation from Remark 3.4, they amount to:

$$x_i^{[k+1]} = x_i^{[k]} + \Delta t \sum_{h=1}^{H} \sum_{j=1}^{n} \left( \frac{e^{\langle Q_h x_i^{[k]}, K_h x_j^{[k]} \rangle}}{\sum_{\ell=1}^{n} e^{\langle Q_h x_i^{[k]}, K_h x_\ell(k) \rangle}} \right) V_h x_j^{[k]}, \quad k \in \mathbb{N}.$$

For each $h \in [H]$ (corresponding to a different head), the weight matrices $Q_h, K_h, V_h$ are constant. Proofs regarding clustering or convergence of the self-attention matrix for such dynamics is an open problem. Preliminary numerical investigations seem to indicate that interesting clustering phenomena also occur in this context. A characterization or properties of optimal weights by invoking the optimal control correspondence in the spirit of [Wei17] is also an interesting avenue for future research.

We hereby list a couple of additional numerical experiments suggesting generalizations of our results, which we leave as open problems.

12.1. **Beyond $Q^\top K > 0$ in Theorems 3.1 and 5.2.** As seen throughout all the presented proofs, assumptions on the value matrix $V$ are significantly more rigid than assumptions on the matrices $Q$ and $K$. For instance, should the eigenvalue $\lambda$ with the largest real part of $V$ be negative, all rescaled tokens will diverge to infinity. Should $\lambda$ be complex, we do not expect any clustering to occur (for the
rescaled tokens). Yet, none of the conclusions of Theorems 3.1 or 5.2 seem to change for generic choices of $Q^\top K$. This is illustrated in Figures 16 and 17 respectively.

$t = 0.0$

$t = 1.0$

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

$t = 2.0$

$t = 5.0$

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /><figcaption>**Figure 16.** Here, $V = I_d$, while $Q^\top K$ violates the PSD assumption—it is a random matrix (with entries drawn from the uniform distribution on $[-1, 1]$). Nonetheless, the clustering pattern entailed by Theorem 3.1 persists.</figcaption></figure>

### 12.2. Beyond pure self-attention: adding a feed-forward layer.
Practical implementations of the Transformer architecture combine the self-attention mechanism with a feed-forward neural network. While extending the mathematical analysis from this paper to such a broader setting would be challenging, we can offer some numerical insights into the expected outcomes.

The feed-forward neural network which can be adjoined to the Transformer dynamics in one of two ways. The first way consists in running the pure self-attention dynamics up to time $t \le T$ (or equivalently, for $O(T)$ layers), and then applying a pure feed-forward neural network to the concatenated vector of clustered features at time $T$. This amounts to seeing the feed-forward network as a map from $\mathbb{R}^{nd}$ to $\mathbb{R}^m$ (for some $m \ge 1$), which can be studied independently with existing theory.
rescaled tokens). Yet, none of the conclusions of Theorems 3.1 or 5.2 seem to change for generic choices of $Q^\top K$. This is illustrated in Figures 16 and 17 respectively.

$t = 0.0$

$t = 1.0$

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

$t = 2.0$

$t = 5.0$

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /><figcaption>**Figure 16.** Here, $V = I_d$, while $Q^\top K$ violates the PSD assumption—it is a random matrix (with entries drawn from the uniform distribution on $[-1, 1]$). Nonetheless, the clustering pattern entailed by Theorem 3.1 persists.</figcaption></figure>

### 12.2. Beyond pure self-attention: adding a feed-forward layer.
Practical implementations of the Transformer architecture combine the self-attention mechanism with a feed-forward neural network. While extending the mathematical analysis from this paper to such a broader setting would be challenging, we can offer some numerical insights into the expected outcomes.

The feed-forward neural network which can be adjoined to the Transformer dynamics in one of two ways. The first way consists in running the pure self-attention dynamics up to time $t \le T$ (or equivalently, for $O(T)$ layers), and then applying a pure feed-forward neural network to the concatenated vector of clustered features at time $T$. This amounts to seeing the feed-forward network as a map from $\mathbb{R}^{nd}$ to $\mathbb{R}^m$ (for some $m \ge 1$), which can be studied independently with existing theory.
$t = 0.0$

$t = 5.0$

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

$t = 10.0$

$t = 15.0$

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /><figcaption>**Figure 17.** Here, $V$ is paranormal, while $Q^\top K$ violates the PSD assumption—it is a random matrix (with entries drawn from the uniform distribution on $[-1, 1]$). Nonetheless, the clustering pattern entailed by Theorem 5.2 persists.</figcaption></figure>

The second way consists in using both the self-attention and feed-forward mechanisms in parallel at every layer $t$. In this case, clustering in the exact sense of Theorems 3.1 and Theorems 5.2 would be difficult to anticipate since the weights of the feed-forward network play the role of a value matrix $V$ (as they can be absorbed within $V$), and the conclusions of these theorems strongly depend on the identity-like structure.

In Figure 18, we focus on the second of the above-discussed examples, and illustrate a possible generalization of Theorem 4.2 to this setup. For simplicity, we focus on a 2-layer neural network: we apply a component-wise nonlinear activation function $\sigma$ (either the ReLU or tanh) to the self-attention dynamics, and then multiply by a weight matrix $W \in \mathbb{R}^{d \times d}$. Namely, we consider

$$ \dot{z}_i(t) = W\sigma \left( V \sum_{j=1}^{n} \left( \frac{e^{\langle Qe^{tV}z_i(t), Ke^{tV}z_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle Qe^{tV}z_i(t), Ke^{tV}z_k(t) \rangle}} \right) (z_j(t) - z_i(t)) \right) \quad (12.1) $$
$t = 0.0$

$t = 5.0$

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

$t = 10.0$

$t = 15.0$

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /><figcaption>**Figure 17.** Here, $V$ is paranormal, while $Q^\top K$ violates the PSD assumption—it is a random matrix (with entries drawn from the uniform distribution on $[-1, 1]$). Nonetheless, the clustering pattern entailed by Theorem 5.2 persists.</figcaption></figure>

The second way consists in using both the self-attention and feed-forward mechanisms in parallel at every layer $t$. In this case, clustering in the exact sense of Theorems 3.1 and Theorems 5.2 would be difficult to anticipate since the weights of the feed-forward network play the role of a value matrix $V$ (as they can be absorbed within $V$), and the conclusions of these theorems strongly depend on the identity-like structure.

In Figure 18, we focus on the second of the above-discussed examples, and illustrate a possible generalization of Theorem 4.2 to this setup. For simplicity, we focus on a 2-layer neural network: we apply a component-wise nonlinear activation function $\sigma$ (either the ReLU or tanh) to the self-attention dynamics, and then multiply by a weight matrix $W \in \mathbb{R}^{d \times d}$. Namely, we consider

$$ \dot{z}_i(t) = W\sigma \left( V \sum_{j=1}^{n} \left( \frac{e^{\langle Qe^{tV}z_i(t), Ke^{tV}z_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle Qe^{tV}z_i(t), Ke^{tV}z_k(t) \rangle}} \right) (z_j(t) - z_i(t)) \right) \quad (12.1) $$
for $i \in [n]$ and $t \ge 0$. A bias vector $b \in \mathbb{R}^d$ (whether inside or outside the activation function) can also be included to allow for translations. The clustering property appears to persist, the pattern depending on the weight matrix $W$ and on the activation function $\sigma$. We leave this problem open to further investigation.

<figure><img src="image_32.png" /><figcaption>**Figure 18.** The setup of Theorem 4.2 with a 2-layer neural network appended to the dynamics (i.e., (12.1)). Top: $\sigma = \text{ReLU}$ with $W = I_d$. Middle: $\sigma = \tanh$ with $W = I_d$. Bottom: $\sigma = \text{ReLU}$ with $W$ being a random matrix. In the first row, we see that the particles first evolve as to reach the upper right quadrant $(\mathbb{R}_{>0})^d$ (due to the ReLU). Once they reach it, every particle eventually follows one of three hyperplanes determined by the spectrum of $V$ and the projection onto $(\mathbb{R}_{>0})^d$. In the other two cases, all particles appear to collapse to 0.</figcaption></figure>

## REFERENCES

[ABV$^{+05}$] Juan A Acebrón, Luis L Bonilla, Conrad J Pérez Vicente, Félix Ritort, and Renato Spigler. The Kuramoto model: A simple paradigm for synchronization phenomena. *Reviews of modern physics*, 77(1):137, 2005.

[BM00] Paul S Bradley and Olvi L Mangasarian. K-plane clustering. *Journal of Global optimization*, 16:23–32, 2000.

[Che95] Yizong Cheng. Mean shift, mode seeking, and clustering. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 17(8):790–799, 1995.

[CHH$^{+16}$] Junghee Cho, Seung-Yeal Ha, Feimin Huang, Chunyin Jin, and Dongnam Ko. Emergence of bi-cluster flocking for the Cucker–Smale model. *Mathematical Models and Methods in Applied Sciences*, 26(06):1191–1218, 2016.

[CRBD18] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. *Advances in Neural Information Processing Systems*, 31, 2018.
for $i \in [n]$ and $t \ge 0$. A bias vector $b \in \mathbb{R}^d$ (whether inside or outside the activation function) can also be included to allow for translations. The clustering property appears to persist, the pattern depending on the weight matrix $W$ and on the activation function $\sigma$. We leave this problem open to further investigation.

<figure><img src="image_33.png" /><figcaption>**Figure 18.** The setup of Theorem 4.2 with a 2-layer neural network appended to the dynamics (i.e., (12.1)). Top: $\sigma = \text{ReLU}$ with $W = I_d$. Middle: $\sigma = \tanh$ with $W = I_d$. Bottom: $\sigma = \text{ReLU}$ with $W$ being a random matrix. In the first row, we see that the particles first evolve as to reach the upper right quadrant $(\mathbb{R}_{>0})^d$ (due to the ReLU). Once they reach it, every particle eventually follows one of three hyperplanes determined by the spectrum of $V$ and the projection onto $(\mathbb{R}_{>0})^d$. In the other two cases, all particles appear to collapse to 0.</figcaption></figure>

## REFERENCES

[ABV$^{+05}$] Juan A Acebrón, Luis L Bonilla, Conrad J Pérez Vicente, Félix Ritort, and Renato Spigler. The Kuramoto model: A simple paradigm for synchronization phenomena. *Reviews of modern physics*, 77(1):137, 2005.

[BM00] Paul S Bradley and Olvi L Mangasarian. K-plane clustering. *Journal of Global optimization*, 16:23–32, 2000.

[Che95] Yizong Cheng. Mean shift, mode seeking, and clustering. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 17(8):790–799, 1995.

[CHH$^{+16}$] Junghee Cho, Seung-Yeal Ha, Feimin Huang, Chunyin Jin, and Dongnam Ko. Emergence of bi-cluster flocking for the Cucker–Smale model. *Mathematical Models and Methods in Applied Sciences*, 26(06):1191–1218, 2016.

[CRBD18] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. *Advances in Neural Information Processing Systems*, 31, 2018.
[CS07] Felipe Cucker and Steve Smale. Emergent behavior in flocks. *IEEE Transactions on Automatic Control*, 52(5):852-862, 2007.

[DCL21] Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In *International Conference on Machine Learning*, pages 2793-2803. PMLR, 2021.

[Dob79] Roland L'vovich Dobrushin. Vlasov equations. *Funktsional'nyi Analiz i ego Prilozheniya*, 13(2):48-58, 1979.

[Gol13] François Golse. Mean field kinetic equations. *Course of Polytechnique*, 2013.

[HJ12] Roger A Horn and Charles R Johnson. *Matrix analysis*. Cambridge University Press, 2012.

[HK02] Rainer Hegselmann and Ulrich Krause. Opinion dynamics and bounded confidence: models, analysis and simulation. *Journal of Artificial Societies and Social Simulation (JASSS)*, 5(3), 2002.

[HKPZ19] Seung-Yeal Ha, Jeongho Kim, Jinyeong Park, and Xiongtao Zhang. Complete cluster predictability of the Cucker-Smale flocking model on the real line. *Archive for Rational Mechanics and Analysis*, 231:319-365, 2019.

[HR17] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. *In:verse problems*, 34(1), 2017.

[HysW$^{+22}$] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In *International Conference on Learning Representations*, 2022.

[HZRS16a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770-778, 2016.

[HZRS16b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In *Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14*, pages 630-645. Springer, 2016.

[JM14] Pierre-Emmanuel Jabin and Sebastien Motsch. Clustering and asymptotic behavior in opinion formation. *Journal of Differential Equations*, 257(11):4165-4187, 2014.

[Kra00] Ulrich Krause. A discrete nonlinear and non-autonomous model of consensus. In *Communications in Difference Equations: Proceedings of the Fourth International Conference on Difference Equations*, page 227. CRC Press, 2000.

[Kur75] Yoshiki Kuramoto. Self-entrainment of a population of coupled non-linear oscillators. In *International Symposium on Mathematical Problems in Theoretical Physics: January 23-29, 1975, Kyoto University, Kyoto/Japan*, pages 420-422. Springer, 1975.

[LCG$^{+20}$] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In *International Conference on Learning Representations*, 2020.

[LLH$^{+20}$] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. In *International Conference on Learning Representations*, 2020.

[LWLQ22] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. *AI Open*, 3:111-132, 2022.

[MT14] Sebastien Motsch and Eitan Tadmor. Heterophilious dynamics enhances consensus. *SIAM Review*, 56(4):577-621, 2014.

[PHD20] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. *Proceedings of the National Academy of Sciences*, 117(40):24652-24663, 2020.

[PR13] Benedetto Piccoli and Francesco Rossi. Transport equation with nonlocal velocity in Wasserstein spaces: convergence of numerical schemes. *Acta Applicandae Mathematicae*, 124:73-105, 2013.

[PR16] Benedetto Piccoli and Francesco Rossi. On properties of the generalized Wasserstein distance. *Archive for Rational Mechanics and Analysis*, 222:1339-1365, 2016.

[PRT15] Benedetto Piccoli, Francesco Rossi, and Emmanuel Trélat. Control to flocking of the kinetic Cucker-Smale model. *SIAM Journal on Mathematical Analysis*, 47(6):4685-4719, 2015.
[CS07] Felipe Cucker and Steve Smale. Emergent behavior in flocks. *IEEE Transactions on Automatic Control*, 52(5):852-862, 2007.

[DCL21] Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In *International Conference on Machine Learning*, pages 2793-2803. PMLR, 2021.

[Dob79] Roland L'vovich Dobrushin. Vlasov equations. *Funktsional'nyi Analiz i ego Prilozheniya*, 13(2):48-58, 1979.

[Gol13] François Golse. Mean field kinetic equations. *Course of Polytechnique*, 2013.

[HJ12] Roger A Horn and Charles R Johnson. *Matrix analysis*. Cambridge University Press, 2012.

[HK02] Rainer Hegselmann and Ulrich Krause. Opinion dynamics and bounded confidence: models, analysis and simulation. *Journal of Artificial Societies and Social Simulation (JASSS)*, 5(3), 2002.

[HKPZ19] Seung-Yeal Ha, Jeongho Kim, Jinyeong Park, and Xiongtao Zhang. Complete cluster predictability of the Cucker-Smale flocking model on the real line. *Archive for Rational Mechanics and Analysis*, 231:319-365, 2019.

[HR17] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. *In:verse problems*, 34(1), 2017.

[HysW$^{+22}$] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In *International Conference on Learning Representations*, 2022.

[HZRS16a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770-778, 2016.

[HZRS16b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In *Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14*, pages 630-645. Springer, 2016.

[JM14] Pierre-Emmanuel Jabin and Sebastien Motsch. Clustering and asymptotic behavior in opinion formation. *Journal of Differential Equations*, 257(11):4165-4187, 2014.

[Kra00] Ulrich Krause. A discrete nonlinear and non-autonomous model of consensus. In *Communications in Difference Equations: Proceedings of the Fourth International Conference on Difference Equations*, page 227. CRC Press, 2000.

[Kur75] Yoshiki Kuramoto. Self-entrainment of a population of coupled non-linear oscillators. In *International Symposium on Mathematical Problems in Theoretical Physics: January 23-29, 1975, Kyoto University, Kyoto/Japan*, pages 420-422. Springer, 1975.

[LCG$^{+20}$] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In *International Conference on Learning Representations*, 2020.

[LLH$^{+20}$] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. In *International Conference on Learning Representations*, 2020.

[LWLQ22] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. *AI Open*, 3:111-132, 2022.

[MT14] Sebastien Motsch and Eitan Tadmor. Heterophilious dynamics enhances consensus. *SIAM Review*, 56(4):577-621, 2014.

[PHD20] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. *Proceedings of the National Academy of Sciences*, 117(40):24652-24663, 2020.

[PR13] Benedetto Piccoli and Francesco Rossi. Transport equation with nonlocal velocity in Wasserstein spaces: convergence of numerical schemes. *Acta Applicandae Mathematicae*, 124:73-105, 2013.

[PR16] Benedetto Piccoli and Francesco Rossi. On properties of the generalized Wasserstein distance. *Archive for Rational Mechanics and Analysis*, 222:1339-1365, 2016.

[PRT15] Benedetto Piccoli, Francesco Rossi, and Emmanuel Trélat. Control to flocking of the kinetic Cucker-Smale model. *SIAM Journal on Mathematical Analysis*, 47(6):4685-4719, 2015.
[RS14] Brian Rider and Christopher D. Sinclair. Extremal laws for the real Ginibre ensemble. *The Annals of Applied Probability*, 24(4):1621–1651, 2014.

[SABP22] Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In *International Conference on Artificial Intelligence and Statistics*, pages 3515–3530. PMLR, 2022.

[VCBJ$^{+95}$] Tamás Vicsek, András Czirók, Eshel Ben-Jacob, Inon Cohen, and Ofer Shochet. Novel type of phase transition in a system of self-driven particles. *Physical Review Letters*, 75(6):1226, 1995.

[Vid11] René Vidal. Subspace clustering. *IEEE Signal Processing Magazine*, 28(2):52–68, 2011.

[VKF20] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. *Advances in Neural Information Processing Systems*, 33:21665–21674, 2020.

[VSP$^{+17}$] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. *Advances in Neural Information Processing Systems*, 30, 2017.

[Wei17] E Weinan. A proposal on machine learning via dynamical systems. *Communications in Mathematics and Statistics*, 1(5):1–11, 2017.

[WHL19] E Weinan, Jiequn Han, and Qianxiao Li. A mean-field optimal control formulation of deep learning. *Research in Mathematical Sciences*, 6(1):10, 2019.

[WLK$^{+20}$] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. *arXiv preprint arXiv:2006.04768*, 2020.

[YBR$^{+20}$] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In *International Conference on Learning Representations*, 2020.

**Borjan Geshkovski**

Department of Mathematics
MIT
Cambridge, MA
02139 USA

e-mail: borjan@mit.edu

**Cyril Letrouit**

Department of Mathematics
MIT
Cambridge, MA
02139 USA

e-mail: letrouit@mit.edu

**Yury Polyanskiy**

Department of EECS
MIT
Cambridge, MA
02139 USA

e-mail: yp@mit.edu

**Philippe Rigollet**

Department of Mathematics
MIT
Cambridge, MA
02139 USA

e-mail: rigollet@mit.edu
[RS14] Brian Rider and Christopher D. Sinclair. Extremal laws for the real Ginibre ensemble. *The Annals of Applied Probability*, 24(4):1621–1651, 2014.

[SABP22] Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In *International Conference on Artificial Intelligence and Statistics*, pages 3515–3530. PMLR, 2022.

[VCBJ$^{+95}$] Tamás Vicsek, András Czirók, Eshel Ben-Jacob, Inon Cohen, and Ofer Shochet. Novel type of phase transition in a system of self-driven particles. *Physical Review Letters*, 75(6):1226, 1995.

[Vid11] René Vidal. Subspace clustering. *IEEE Signal Processing Magazine*, 28(2):52–68, 2011.

[VKF20] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. *Advances in Neural Information Processing Systems*, 33:21665–21674, 2020.

[VSP$^{+17}$] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. *Advances in Neural Information Processing Systems*, 30, 2017.

[Wei17] E Weinan. A proposal on machine learning via dynamical systems. *Communications in Mathematics and Statistics*, 1(5):1–11, 2017.

[WHL19] E Weinan, Jiequn Han, and Qianxiao Li. A mean-field optimal control formulation of deep learning. *Research in Mathematical Sciences*, 6(1):10, 2019.

[WLK$^{+20}$] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. *arXiv preprint arXiv:2006.04768*, 2020.

[YBR$^{+20}$] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In *International Conference on Learning Representations*, 2020.

**Borjan Geshkovski**

Department of Mathematics
MIT
Cambridge, MA
02139 USA

e-mail: borjan@mit.edu

**Cyril Letrouit**

Department of Mathematics
MIT
Cambridge, MA
02139 USA

e-mail: letrouit@mit.edu

**Yury Polyanskiy**

Department of EECS
MIT
Cambridge, MA
02139 USA

e-mail: yp@mit.edu

**Philippe Rigollet**

Department of Mathematics
MIT
Cambridge, MA
02139 USA

e-mail: rigollet@mit.edu