SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \boldsymbol {g} [ i ] := Y \left(\boldsymbol {x} _ {T + 1} [ i ] - \boldsymbol {\alpha} [ i ]\right) (81) \\ = F \left(I - E ^ {\prime}\right) \left(I + \Theta\right) \bar {W} (N) \left(\boldsymbol {x} _ {T + 1} [ i ] - \boldsymbol {\alpha} [ i ]\right), \quad (\text {T h e o r e m} 5) (82) \\ = (1 + \sigma) \gamma * F (I - E ^ {\prime}) (I + \Theta) [ q _ {0}, \dots , q _ {1}, \dots , q _ {0} ] ^ {\top}, \quad (\text {L e m m a 5 , C o r o l l a r y 1}) (83) \\ = \gamma \left(\iota_ {n} \boldsymbol {f} _ {n} - \sum_ {n ^ {\prime} \neq n, n ^ {\prime} \in [ K ]} \beta_ {n n ^ {\prime}} \boldsymbol {f} _ {n ^ {\prime}}\right) (84) \\ \end{array}
$$
$$
\begin{array}{l} \gamma (t) := \frac {(M - 1) ^ {2} h (\lceil t / K \rceil)}{(M - 1) + \exp (M h (\lceil t / K \rceil))} > 0 (85) \\ \iota_ {n} := (1 + \sigma) \left[ q _ {1} \cdot \left(1 + \theta_ {n}\right) \left(1 - E _ {n n} ^ {\prime}\right) - q _ {0} \sum_ {k \neq n, k \in [ K ]} \left(1 + \theta_ {k}\right) E _ {k n} ^ {\prime} \right] (86) \\ = (1 + \sigma) \left[ \left(1 - E _ {n n} ^ {\prime}\right) \cdot \left(1 + \delta_ {1}\right) + \delta_ {2} \right] (87) \\ \end{array}
$$
$$
\begin{array}{l} \beta_ {n n ^ {\prime}} := (1 + \sigma) \left[ q _ {1} \cdot \left(1 + \theta_ {n}\right) E _ {n n ^ {\prime}} ^ {\prime} + q _ {0} \left(\left(1 + \theta_ {n ^ {\prime}}\right) + \sum_ {k \neq n, k \in [ K ]} \left(1 + \theta_ {k}\right) E _ {k n ^ {\prime}} ^ {\prime}\right)\right) (88) \\ = (1 + \sigma) \left[ E _ {n n ^ {\prime}} ^ {\prime} \cdot \left(1 + \delta_ {1}\right) + \delta_ {3} \right] (89) \\ \end{array}
$$
Here  $\sigma$  is defined in Cor. 1 and satisfies  $-1 < \sigma \ll \ln M$ .  $|\delta_1| \lesssim \frac{K}{\sqrt{N}} \sqrt{\ln(\frac{1}{\delta})} + \frac{1}{M} \ll 1$  and  $|\delta_2|, |\delta_3| \leq \frac{M}{(M - 1)^2} \times 2(1 + 3|\delta_1|) < \frac{3}{M}$ . Here we use the fact that  $|\theta|, |\theta_i| \lesssim \frac{K}{\sqrt{N}} \sqrt{\ln(\frac{1}{\delta})}$ ,  $\sum_{k \in [K]} \lambda_k u_{jk} u_{jn'} = E_{kn'}$  and the fact from Lemma 15:
$$
\left| E _ {k n} ^ {\prime} \right| \leq \max  _ {i \in [ K ]} \left| \lambda_ {i} \left(E ^ {\prime}\right) \right| \leq \frac {1}{1 - 1 / K} \left| \lambda_ {1} \right| \leq \frac {1}{K - 1} \tag {90}
$$
(a) Now let's prove that  $\xi_{n}(t) > 0$ . First from  $(I + E)(I - E^{\prime}) = I$  we have  $E - E^{\prime} - EE^{\prime} = O$ . Then use the symmetry of  $E$  and  $E^{\prime}$ , we get
$$
\left(E E ^ {\prime}\right) _ {n n} = \sum_ {k = 1} E _ {n k} E _ {k n} ^ {\prime} = \sum_ {k = 1} E _ {n k} E _ {n k} ^ {\prime} = \sum_ {k = 1} E _ {n k} E _ {n k} ^ {\prime} = \sum_ {k \neq n} E _ {n k} E _ {n k} ^ {\prime} + E _ {n n} E _ {n n} ^ {\prime} \tag {91}
$$
Note that  $F^{\top}F = I + E$ , we have  $E_{nn'} = \pmb{f}_n^\top \pmb{f}_{n'}, \forall n' \neq n$  and  $E_{nn} = 0$ . Then
$$
\left(E - E ^ {\prime} - E E ^ {\prime}\right) _ {n n} = O _ {n n} = 0 \Rightarrow \sum_ {k \neq n} E _ {n k} E _ {n k} ^ {\prime} = - E _ {n n} ^ {\prime} \tag {92}
$$
Note that  $|\lambda_i(E)| > 0, \forall i \in [K]$  in Assumption 2 implies that  $E_{ki} > 0$  holds for some  $k \neq i \in [K]$ . Then from (1) of Lemma 16 we get  $\sum_{k \neq n} E_{nn'}' \mathbf{f}_n^\top \mathbf{f}_{n'} > 0$ .
From Theorem 1 we have  $\beta_{nn'} = (1 + \sigma)[E_{nn'}' \cdot (1 + \delta_1) + \delta_3]$ . Note that  $0 < 1 + \sigma \ll \ln(M)$ , we have:
$$
\begin{array}{l} \sum_ {n ^ {\prime} \neq n} \beta_ {n n ^ {\prime}} \boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}} = (1 + \sigma) \left[ \sum_ {n ^ {\prime} \neq n} \left[ E _ {n n ^ {\prime}} ^ {\prime} \left(1 + \delta_ {1}\right) + \delta_ {3} \right] E _ {n n ^ {\prime}} \right] \\ = (1 + \sigma) [ - (1 + \delta_ {1}) E _ {n n} ^ {\prime} + \delta_ {3} \sum_ {n ^ {\prime} \neq n} E _ {n n ^ {\prime}} ] \\ = (1 + \sigma) \left[ \left(1 + \delta_ {1}\right) \sum_ {k = 1} ^ {K} \frac {\lambda_ {k} ^ {2}}{1 + \lambda_ {k}} u _ {n k} ^ {2} + \delta_ {3} \sum_ {n ^ {\prime} \neq n} E _ {n n ^ {\prime}} \right] \quad (\text {E q n .} 7 9) \\ \geq (1 + \sigma) \left[ \frac {1 + \delta_ {1}}{1 - | \lambda_ {1} |} \left(\min  _ {i} | \lambda_ {i} (E) | ^ {2}\right) - \frac {3}{M} \cdot K | \lambda_ {1} | \right], \quad (\text {E q n .} 9 0, | \delta_ {3} | <   \frac {3}{M}) \\ > (1 + \sigma) [ \frac {1}{2} (\min _ {i} | \lambda_ {i} (E) | ^ {2}) - \frac {3}{M} \cdot K | \lambda_ {1} | ], \quad (| \delta_ {1} | \ll 1, | \lambda_ {1} | <   \frac {1}{K} \ll 1) \\ > 0, \quad (\text {A s s u m p t i o n} 2) \tag {93} \\ \end{array}
$$
(b) We directly use Lemma 9, then we finish the proof.
Proof. Taking long sequence limit  $(T\to +\infty)$ , and summing over all possible choices of next token  $x_{T + 1} = n$ , plugging in the backpropagated gradient (Eqn. 9) into the dynamics of  $Z$  with query token  $m$  (Eqn. 4), we arrive at the following:
$$
\begin{array}{l} \dot {z} _ {m} = \eta_ {Z} \sum_ {n \in \psi^ {- 1} (m)} \operatorname {d i a g} \left(\boldsymbol {c} _ {n}\right) \frac {P _ {\boldsymbol {f} _ {n}} ^ {\perp}}{\| \boldsymbol {c} _ {n} \| _ {2}} Y \left(\boldsymbol {x} _ {T + 1} [ i ] - \boldsymbol {\alpha} [ i ]\right) (94) \\ = - \eta_ {Z} \gamma \sum_ {n \in \psi^ {- 1} (m)} \operatorname {d i a g} \left(\boldsymbol {f} _ {n}\right) P _ {\boldsymbol {f} _ {n}} ^ {\perp} \sum_ {n ^ {\prime} \neq n} \beta_ {n n ^ {\prime}} \boldsymbol {f} _ {n ^ {\prime}} (95) \\ = \eta_ {Z} \gamma \sum_ {n \in \psi^ {- 1} (m)} \operatorname {d i a g} \left(\boldsymbol {f} _ {n}\right) \left(\boldsymbol {f} _ {n} \boldsymbol {f} _ {n} ^ {\top} - I\right) \sum_ {n ^ {\prime} \neq n} \beta_ {n n ^ {\prime}} \boldsymbol {f} _ {n ^ {\prime}} (96) \\ \end{array}
$$
Note here we leverage the property that  $P_{\pmb{f}}^{\perp}\pmb{f} = 0$  and  $P_{c_n}^{\perp} = P_{\pmb{f}_n}^{\perp}$ .
Proof. For any token  $l$ , we have:
$$
\dot {z} _ {m l} = \eta_ {Z} \gamma \sum_ {n \in \psi^ {- 1} (m)} f _ {n l} \sum_ {n ^ {\prime} \neq n} \beta_ {n n ^ {\prime}} \left[ \left(\boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}}\right) f _ {n l} - f _ {n ^ {\prime} l} \right] \tag {97}
$$
Distinct token. For a token  $l$  distinct to  $n$ , by definition, for any  $n' \neq n$ ,  $\mathbb{P}(l|m,n') = 0$  and  $f_{n'l}(t) \propto \mathbb{P}(l|m,n')\exp (z_{ml}) \equiv 0$ . Therefore, we have:
$$
\dot {z} _ {m l} = \eta_ {Z} \gamma f _ {n l} ^ {2} \sum_ {n ^ {\prime} \neq n} \beta_ {n n ^ {\prime}} \boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}} = \eta_ {Z} f _ {n l} ^ {2} \xi_ {n} > 0 \tag {98}
$$
Note that  $\dot{z}_{ml} > 0$  is achieved by  $\xi_{n} > 0$  from Theorem 1.
Common token. For any query token  $m$ , consider  $n \in \psi^{-1}(m)$  and  $n' \neq n$ . If  $n$  and  $n'$  does not overlap then  $\mathrm{diag}(\pmb{f}_n)(\pmb{f}_n\pmb{f}_n^\top - I)\pmb{f}_{n'} = -\mathrm{diag}(\pmb{f}_n)\pmb{f}_{n'} = 0$ . When  $n$  and  $n'$  overlaps, let  $G_{CT}(n,n') := \{l : \mathbb{P}(l|n)\mathbb{P}(l|n') > 0\}$  be the subset of common tokens shared between  $n$  and  $n'$ , since  $|G_{CT}| = 1$  and  $\emptyset \neq G_{CT}(n,n') \subseteq G_{CT} := \bigcup_{n \neq n'} G_{CT}(n,n')$ , we have  $|G_{CT}(n,n')| = 1$  and  $l \in G_{CT}(n,n')$ , i.e., the common token  $l$  is the unique overlap. Then we have:
$$
f _ {n l} \left[ \left(\boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}}\right) f _ {n l} - f _ {n ^ {\prime} l} \right] = \left(\boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}}\right) f _ {n l} ^ {2} - \boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}} = - \left(1 - f _ {n l} ^ {2}\right) \left(\boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}}\right) \tag {99}
$$
$$
\dot {z} _ {m l} = - \eta_ {Z} \gamma \sum_ {n \in \psi^ {- 1} (m)} \left(1 - f _ {n l} ^ {2}\right) \sum_ {n ^ {\prime} \neq n} \beta_ {n n ^ {\prime}} \boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}} = - \eta_ {Z} \sum_ {n \in \psi^ {- 1} (m)} \left(1 - f _ {n l} ^ {2}\right) \xi_ {n} \leq 0 \tag {100}
$$
Since  $\xi_{n}(t) > 0$ , the only condition that will lead to  $\dot{z}_{ml} = 0$  is  $f_{nl}^{2} = 1$ . However, since at least one such  $n$  has another distinct token  $l'$ , and thus  $f_{nl'} > 0$ , by normalization condition,  $f_{nl} < 1$  and thus  $\dot{z}_{ml} < 0$ .
Note that for multiple common tokens, things can be quite involved. Here we prove a case when the symmetric condition holds.
Corollary 2 (Multiple CTs, symmetric case). If Assumption 2 holds and assume
- (1) Single query token  $m_0$ . For any next token  $n \in [K]$ ,  $\psi(n) = m_0$ .
(2) Symmetry. For any two next tokens  $n \neq n'$ , there exists a one-to-one mapping  $\phi$  that maps token  $l \in G_{DT}(n)$  to  $l' \in G_{DT}(n')$  so that  $\mathbb{P}(l|n) = \mathbb{P}(\phi(l)|n')$ ;
(3) Global common tokens with shared conditional probability: i.e., the global common token set  $G_{CT}$  satisfies the following condition: for any  $l \in G_{CT}$ ,  $\mathbb{P}(l|n) = \rho_l$ , which is independent of next token  $n$ ;
(4) The initial condition  $Z(0) = 0$ .
Then for any common token  $l \in G_{CT}$ ,  $\dot{z}_{m_0,l} < 0$ .
Proof. Since there is a global query token  $m_0$ , we omit the subscript  $m_0$  and let  $z_l \coloneqq z_{m_0, l}$ .
We want to prove the following induction hypothesis: for any  $t$  (a)  $z_{l}(t) = z_{\phi_{m}(l)}(t)$  for  $n, n'$  which are next tokens that the distinct token  $l$  (and  $l'$ ) belongs to, and (b) the normalization term  $o_{n}^{2}(t) \coloneqq \sum_{l} \tilde{c}_{l|n}^{2}(t) = o^{2}(t)$ , i.e., it does not depend on  $n$ .
We prove by induction on infinitesimal steps  $\delta t$ . First when  $t = 0$ , both conditions hold due to the initial condition  $Z(0) = 0$ . Then we assume that both conditions hold for time  $t$ , then by symmetry, we know that for any  $n_1$  and any distinct token  $l_1 \in G_{DT}(n_1)$ :
$$
\dot {z} _ {l _ {1}} (t) = \eta_ {Z} \gamma f _ {n _ {1} l _ {1}} ^ {2} \sum_ {n ^ {\prime} \neq n _ {1}} \beta_ {n _ {1} n ^ {\prime}} \boldsymbol {f} _ {n _ {1}} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}} = \eta_ {Z} \gamma f _ {n _ {2} l _ {2}} ^ {2} \sum_ {n ^ {\prime} \neq n _ {2}} \beta_ {n _ {2} n ^ {\prime}} \boldsymbol {f} _ {n _ {2}} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}} = \dot {z} _ {l _ {2}} (t) \tag {101}
$$
where  $l_{2} = \phi(l_{1})$  is the image of the distinct token  $l_{1}$ . This is because (1)  $\pmb{f}_{n_1}^\top \pmb{f}_{n'} = \sum_{l \in G_{CT}} \rho_l^2 \exp(2z_l(t)) / o^2(t)$  is independent of  $n_1$  and  $n'$  by inductive hypothesis, therefore,  $\beta$  is also independent of its subscripts. And (2)  $f_{n_1 l_1}^2 := \tilde{c}_{l_1 | n_1}^2 / o^2(t) = \tilde{c}_{l_2 | n_2}^2 / o^2(t) = f_{n_2 l_2}^2$ .
Therefore,  $\dot{z}_{l_1}(t) = \dot{z}_{l_2}(t)$ , which means that  $z_{l_1}(t') = z_{l_2}(t')$  for  $t' = t + \delta t$ .
Let  $G_{CT}(n_1, n_2) \coloneqq \{l : \mathbb{P}(l|n_1)\mathbb{P}(l|n_2) > 0\}$  be the subset of common tokens shared between  $n_1$  and  $n_2$ , then for their associated  $n_1$  and  $n_2$ , obviously  $G_{CT}(n_1, n_2) \subseteq G_{CT}$  and we have:
$$
\begin{array}{l} o _ {n _ {1}} \left(t ^ {\prime}\right) = \sum_ {l} \tilde {c} _ {l | n _ {1}} ^ {2} \left(t ^ {\prime}\right) = \sum_ {l} \mathbb {P} ^ {2} \left(l | n _ {1}\right) \exp \left(2 z _ {l} \left(t ^ {\prime}\right)\right) (102) \\ = \sum_ {l _ {1} \in G _ {D T} (n _ {1})} \mathbb {P} ^ {2} \left(l _ {1} \mid n _ {1}\right) \exp \left(2 z _ {l _ {1}} \left(t ^ {\prime}\right)\right) + \sum_ {l \in G _ {C T} \left(n _ {1}, n _ {2}\right)} \mathbb {P} ^ {2} \left(l \mid n _ {1}\right) \exp \left(2 z _ {l} \left(t ^ {\prime}\right)\right) (1 0 3) \\ = \sum_ {l _ {1} \in G _ {D T} (n _ {1})} \mathbb {P} ^ {2} \left(\phi \left(l _ {1}\right) \mid n _ {2}\right) \exp \left(2 z _ {\phi \left(l _ {1}\right)} \left(t ^ {\prime}\right)\right) + \sum_ {l \in G _ {C T} \left(n _ {1}, n _ {2}\right)} \rho_ {l} ^ {2} \exp \left(2 z _ {l} \left(t ^ {\prime}\right)\right) (1 0 4) \\ = \sum_ {l _ {2} \in G _ {D T} (n _ {2})} \mathbb {P} ^ {2} \left(l _ {2} \mid n _ {2}\right) \exp \left(2 z _ {l _ {2}} \left(t ^ {\prime}\right)\right) + \sum_ {l \in G _ {C T} \left(n _ {1}, n _ {2}\right)} \mathbb {P} ^ {2} \left(l \mid n _ {2}\right) \exp \left(2 z _ {l} \left(t ^ {\prime}\right)\right) (105) \\ = o _ {n _ {2}} \left(t ^ {\prime}\right) (106) \\ \end{array}
$$
So we prove the induction hypothesis holds for  $t' = t + \delta t$ . Let  $\delta t \to 0$  and we prove it for all  $t$ .
Now we check the dynamics of common token  $l \in G_{CT}$ . First we have for any  $n \neq n'$ ,  $f_{nl}^2(t) = \tilde{c}_{l|n}^2(t) / o^2(t) = \rho_l^2 \exp(2z_l(t)) / o^2(t) = \tilde{c}_{l|n'}^2(t) / o^2(t) = f_{n'l}^2(t)$  and thus  $f_{nl}(t) = f_{n'l}(t) := f_l(t) > 0$ , therefore:
$$
f _ {n l} \left[ \left(\boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}}\right) f _ {n l} - f _ {n ^ {\prime} l} \right] = - f _ {l} ^ {2} \left(1 - \boldsymbol {f} _ {n} ^ {\top} \boldsymbol {f} _ {n ^ {\prime}}\right) <   0 \tag {107}
$$
On the other hand, from the proof on induction hypothesis, we know all off-diagonal elements of  $E$  are the same and are positive. Then all the off-diagonal elements of  $E'$  are also the same and are positive. Following Theorem 1, we know  $\beta_{nn'} > 0$  and is independent of the subscripts. Therefore,  $\dot{z}_l < 0$ .
Theorem 3 (Growth of distinct tokens). For a next token  $n$  and its two distinct tokens  $l$  and  $l'$ , the dynamics of the relative gain  $r_{l / l' | n}(t) := f_{nl}^2(t) / f_{nl'}^2(t) - 1 = \tilde{c}_{l | n}^2(t) / \tilde{c}_{l' | n}^2(t) - 1$  has the following analytic form (here the query token  $m = \psi(n)$  and is uniquely determined by distinct token  $l$ ):
$$
e ^ {2 f _ {n l _ {0}} ^ {2} (0) B _ {n} (t)} \leq \chi l _ {0} (t) \leq e ^ {2 B _ {n} (t)} \tag {12}
$$
Proof. Let  $m = \psi(n)$  be the query token associated with next token  $n$ . For brievity, we omit subscript  $m$  in the proof and let  $z_l := z_{ml}$ .
First of all, for tokens  $l$  and  $l'$  that are both distinct for a specific next token  $n$ , from Eqn. 98, it is clear that
$$
\frac {\dot {z} _ {l}}{\dot {z} _ {l ^ {\prime}}} = r _ {l / l ^ {\prime} \mid n} (t) + 1 = \left(r _ {l / l ^ {\prime} \mid n} (0) + 1\right) \frac {e ^ {2 \left(z _ {l} (t) - z _ {l} (0)\right)}}{e ^ {2 \left(z _ {l ^ {\prime}} (t) - z _ {l ^ {\prime}} (0)\right)}} \tag {108}
$$
$$
e ^ {- 2 \left(z _ {l} - z _ {l} (0)\right)} \dot {z} _ {l} = \left(r _ {l / l ^ {\prime} | n} (0) + 1\right) e ^ {- 2 \left(z _ {l ^ {\prime}} - z _ {l ^ {\prime}} (0)\right)} \dot {z} _ {l ^ {\prime}} \tag {109}
$$
Integrate both side over time  $t$  and we get:
$$
e ^ {- 2 \left(z _ {l} (t) - z _ {l} (0)\right)} - 1 = \left(r _ {l / l ^ {\prime} | n} (0) + 1\right) \left[ e ^ {- 2 \left(z _ {l ^ {\prime}} (t) - z _ {l ^ {\prime}} (0)\right)} - 1 \right] \tag {110}
$$
From this we can get the close-form relationship between  $r_{l / l^{\prime}|n}(t)$  and  $z_{l}(t)$ :
$$
r _ {l / l ^ {\prime} | n} (t) = r _ {l / l ^ {\prime} | n} (0) e ^ {2 \left(z _ {l} (t) - z _ {l} (0)\right)} \tag {111}
$$
Now let  $l_0$  be the dominating distinct token that satisfies  $r_{l_0 / l|n}(0) > 0$  for any distinct token  $l$ , then
we have  $\dot{z}_{l_0} > 0$  due to Theorem 2.
- for any token  $l' \neq l_0$  that is distinct to  $n$ , we have:
$$
\dot {r} _ {l _ {0} / l ^ {\prime} | n} = r _ {l _ {0} / l ^ {\prime} | n} (0) e ^ {2 \left(z _ {l _ {0}} (t) - z _ {l _ {0}} (0)\right)} \dot {z} _ {l _ {0}} > 0 \tag {112}
$$
- for any common token  $l'$ , since  $\dot{z}_{l'} < 0$ , we have
$$
\dot {r} _ {l _ {0} / l ^ {\prime} \mid n} = \frac {\mathrm {d}}{\mathrm {d} t} \left(\frac {\tilde {c} _ {n l _ {0}} ^ {2}}{\tilde {c} _ {n l ^ {\prime}} ^ {2}}\right) = \frac {\mathbb {P} ^ {2} \left(l _ {0} \mid n\right)}{\mathbb {P} ^ {2} \left(l ^ {\prime} \mid n\right)} e ^ {2 \left(z _ {l _ {0}} - z _ {l ^ {\prime}}\right)} \cdot 2 \left(\dot {z} _ {l _ {0}} - \dot {z} _ {l ^ {\prime}}\right) > 0 \tag {113}
$$
$$
\frac {\mathrm {d}}{\mathrm {d} t} \left(f _ {n l _ {0}} ^ {2}\right) = \frac {\mathrm {d}}{\mathrm {d} t} \left(\frac {1}{M + \sum_ {l ^ {\prime} \neq l _ {0}} r _ {l ^ {\prime} / l _ {0} | n}}\right) > 0 \tag {114}
$$
Therefore,  $f_{nl_0}^2 (t)$  is monotonously increasing. Combined with the fact  $f_{nl_0}^2 (t)\leq 1$  due to normalization condition  $\| \pmb {f}_n\| _2 = 1$ , we have:
$$
\xi_ {n} (t) \geq \frac {1}{\eta_ {Z}} \dot {z} _ {l _ {0}} = f _ {n l _ {0}} ^ {2} (t) \xi_ {n} (t) \geq f _ {n l _ {0}} ^ {2} (0) \xi_ {n} (t) \tag {115}
$$
Integrate over time and we have:
$$
B (t) \geq \int_ {0} ^ {t} \dot {z} _ {l _ {0}} \left(t ^ {\prime}\right) \mathrm {d} t ^ {\prime} = z _ {l _ {0}} (t) - z _ {l _ {0}} (0) \geq f _ {n l _ {0}} ^ {2} (0) B (t) \tag {116}
$$
where  $B(t) \coloneqq \eta_Z\int_0^t\xi_n(t')\mathrm{d}t'$ . Plugging that into Eqn. 111, and we have:
$$
e ^ {2 f _ {n l _ {0}} ^ {2} (0) B (t)} \leq \chi_ {l _ {0}} (t) \leq e ^ {2 B (t)} \tag {117}
$$
F Estimation in Sec. 6
- Attention scanning. At the beginning of the training,  $\gamma(t) = O(\eta_Y t / K)$  and  $B_n(t) \approx \frac{1}{4} \ln K^{-1} (\rho_0^4 + 2\eta_Y \eta_Z t^2) = O(\ln t)$ . This means that the growth factor for dominant token  $l_0$  is (sub-)linear:  $\chi_{l_0}(t) \geq e^{2f_{nl_0}^2(0) B_n(t)} \approx [K^{-1} (\rho_0^4 + 2\eta_Y \eta_Z t^2)]^{0.5 f_{nl_0}^2(0)}$ , and the attention on less co-occurred token drops gradually.
- Attention snapping. When  $t \geq t_0 \coloneqq 2(1 + \delta') K \ln M / \eta_Y$  with  $\delta' = \Theta(\frac{\ln \ln M}{\ln M})$ ,  $\gamma(t) = O\left(\frac{K \ln (\eta_Y t / K)}{\eta_Y t}\right)$  and  $B_n(t) = O(\ln \ln t)$ . Therefore, while  $B_n(t)$  still grows to infinite, the growth factor  $\chi_{l_0}(t) = O(\ln t)$  grows at a much slower logarithmic rate.
Proof. Since every next token  $n$  shares the same query token  $m$ , we omit the subscript  $m$  and let  $z_{l} := z_{ml}$ .
We start from the two following assumptions:
$$
\dot {z} _ {l} = - C _ {0} ^ {- 1} \eta_ {Z} \gamma (t) \exp \left(4 z _ {l}\right) \tag {118}
$$
$$
\xi_ {n} (t) = C _ {0} ^ {- 1} \gamma (t) \exp \left(4 z _ {l}\right) \tag {119}
$$
Given that, we can derive the dynamics of  $z_{l}(t)$  and  $\xi_{n}(t)$ :
$$
\exp \left(- 4 z _ {l}\right) \dot {z} _ {l} = - C _ {0} ^ {- 1} \eta_ {Z} \gamma (t) \tag {120}
$$
$$
\mathrm {d} \exp (- 4 z _ {l}) = 4 C _ {0} ^ {- 1} \eta_ {Z} \gamma (t) \mathrm {d} t \tag {121}
$$
$$
\exp (- 4 z _ {l}) = 4 C _ {0} ^ {- 1} \eta_ {Z} \int_ {0} ^ {t} \gamma \left(t ^ {\prime}\right) \mathrm {d} t ^ {\prime} + 1 \quad (\text {u s e} z _ {l} (0) = 0) \tag {122}
$$
Let  $\Gamma(t) \coloneqq \eta_Z \int_0^t \gamma(t') \, \mathrm{d}t'$ , then  $\Gamma(0) = 0$  and  $\mathrm{d}\Gamma(t) = \eta_Z \gamma(t) \, \mathrm{d}t$ . Therefore, we have
$$
\xi_ {n} (t) = C _ {0} ^ {- 1} \gamma (t) \exp (4 z _ {l}) = \frac {\gamma (t)}{C _ {0} + 4 \Gamma (t)} \tag {123}
$$
and thus  $B_{n}(t)\coloneqq \eta_{Z}\int_{0}^{t}\xi_{n}(t^{\prime})\mathrm{d}t^{\prime}$  can be integrated analytically, regardless of the specific form of  $\gamma (t)$ :
$$
B _ {n} (t) = \eta_ {Z} \int_ {0} ^ {t} \frac {\gamma \left(t ^ {\prime}\right) \mathrm {d} t ^ {\prime}}{C _ {0} + 4 \Gamma (t)} = \int_ {0} ^ {t} \frac {\mathrm {d} \Gamma}{C _ {0} + 4 \Gamma} = \frac {1}{4} \ln \left(C _ {0} + 4 \Gamma (t)\right) \tag {124}
$$
Recall that  $\gamma(t) = \frac{(M-1)^2h(t/K)}{M-1+\exp(Mh(t/K))}$  (Theorem 1). Note that  $h$  (if treated in continuous time step) is strictly monotonically increasing and satisfies  $h(0) = 0, \mathrm{d}h(t/K) = \eta_Y(M-1 + \exp(Mh(t/K)))^{-1} \mathrm{d}t/K$  (Lemma 6 and Lemma 7), we can let  $\gamma(h) := \frac{(M-1)^2h}{M-1+\exp(Mh)}$  and get:
$$
\begin{array}{l} \Gamma (t) := \eta_ {Z} \int_ {t = 0} ^ {t} \gamma \left(t ^ {\prime}\right) \mathrm {d} t ^ {\prime} (125) \\ = \eta_ {Z} K \int_ {h (0)} ^ {h (t / K)} \gamma \left(h ^ {\prime}\right) \cdot \frac {M - 1 + \exp \left(M h ^ {\prime}\right)}{\eta_ {Y}} \cdot d h ^ {\prime} (126) \\ = \frac {\eta_ {Z}}{\eta_ {Y}} K (M - 1) ^ {2} \int_ {h (0)} ^ {h (t / K)} h ^ {\prime} \mathrm {d} h ^ {\prime} (127) \\ = \frac {\eta_ {Z}}{\eta_ {Y}} \cdot \frac {K (M - 1) ^ {2}}{2} h ^ {2} (t / K) (128) \\ \end{array}
$$
Therefore,  $B_{n}(t)$  has a close form with respect to  $h$ , regardless of the specific form of  $h(t)$ :
$$
B _ {n} (t) = \frac {1}{4} \ln \left(C _ {0} + 2 \frac {\eta_ {Z}}{\eta_ {Y}} K (M - 1) ^ {2} h ^ {2} (t / K)\right) \tag {129}
$$
(1) When  $t < t_0' \coloneqq K \ln (M) / \eta_Y$ , from Lemma 9 we have  $h(t / K) = (1 + o(1)) \cdot \eta_Y t / (MK)$ . We neglect the  $o(1)$  term and denote  $\nu \coloneqq \eta_Y / \eta_Z$ , then we have when  $t \leq t_0'$ :
$$
B _ {n} (t) = \frac {1}{4} \ln \left(C _ {0} + \frac {2 (M - 1) ^ {2}}{\nu K M ^ {2}} \eta_ {Y} ^ {2} t ^ {2}\right) \tag {130}
$$
And  $B_{n}(t_{0}^{\prime}) = \frac{1}{4}\ln \left(C_{0} + 2K(M - 1)^{2}M^{-2}\nu^{-1}\ln^{2}(M)\right)$
Figure 9: Numerical simulation of  $B_{n}(t)$  with changing  $\eta_{Z}$  and fixed  $\nu = \eta_{Z} / \eta_{Y}$ . The dotted line denotes the transition time  $t_0$ , and  $B_{n}(t_{0})$  marked with the solid dot is independent of  $\eta_{Z}$ .
(2) Similarly, when  $t > t_0 \coloneqq 2(1 + \omega_1)K\ln M / \eta_Y$  with  $\omega_{1} = \Theta (\ln \ln M / \ln M)$  is defined in Lemma 8, from Lemma 9 we have  $h(t / K) = (1 + o(1))\ln (M\eta_Yt / K) / M$ . We neglect the  $o(1)$  term and get when  $t > t_0$ :
$$
B _ {n} (t) = \frac {1}{4} \ln \left(C _ {0} + \frac {2 K (M - 1) ^ {2}}{\nu M ^ {2}} \ln^ {2} \left(M \eta_ {Y} t / K\right)\right) \tag {131}
$$
From this we know  $B_{n}(t_{0}) = \frac{1}{4}\ln (C_{0} + 2K(M - 1)^{2}M^{-2}\nu^{-1}\ln^{2}(2(1 + \omega_{1})M\ln M))$ . It's interesting to find that  $B_{n}(t_{0})$  just depends on  $K, M$  and  $\nu$ , and thus fixing  $\nu$  and changing  $\eta_{Z}$  will not influence the value of  $B_{n}(t_{0})$ , which means that the main difference between  $B_{n}$  is arises at the stage  $t > t_{0}$ . This matches the results in Fig. 9.
(3) Finally, we estimate  $B_{n}(t)$  when  $t$  is large. When  $\nu$  is fixed and  $t\gg (M\eta_Y)^{-1}\exp (1 / \sqrt{2\nu})$  we have
$$
\begin{array}{l} B _ {n} (t) = \left(1 + o (1)\right) \cdot \left[ \frac {1}{2} \ln \ln \left(M \eta_ {Y} t / K\right) + \frac {1}{4} \ln \left(2 K (M - 1) ^ {2} M ^ {- 2} \nu^ {- 1}\right) \right] (132) \\ = \Theta \left(\ln \ln \left(\frac {M \eta_ {Z} \nu t}{K}\right) - \ln \left(\frac {\nu}{K}\right)\right) (133) \\ \end{array}
$$
Therefore, from Eqn. 133 we get:
(a) Fix  $\nu$ , larger  $\eta_Z$  result in larger  $B_n(t)$  and sparser attention map.
(b) Fix  $\eta_Z$ , larger  $\nu$  (i.e., larger  $\eta_Y$ ) result in smaller  $B_n(t)$  and denser attention map since  $\ln \ln(x)$  is much slower than  $\ln(x)$ .
These match our experimental results in the main paper (Fig. 6).
We use WikiText [25] dataset to verify our theoretical findings. This includes two datasets, WikiText2 and WikiText103. We train both on 1-layer transformer with SGD optimizer. Instead of using reparameterization  $Y$  and  $Z$  (Sec. 3.1), we choose to keep the original parameterization with token embedding  $U$  and train with a unified learning rate  $\eta$  until convergence. Fig. 10 shows that the averaged entropy of the self-attention map evaluated in the validation set indeed drops with when the learning rate  $\eta$  becomes larger.
Note that in the original parameterization, it is not clear how to set  $\eta_{Y}$  and  $\eta_Z$  properly and we leave it for future work.
Furthermore, we use the recall-threshold relationship to reshow that smaller  $\eta_{Y} / \eta_{Z}$  and larger  $\eta_{Z}$  will result in a sparser self-attention map as Fig. 11 and Fig. 12. Here we use some thresholds to retag every entry of the attention as a 0-1 value based on its softmax value for every query token, and then calculate the recall value by the average value of the proportion of the distinct tokens with new labels