SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\lim _ {t \to \infty} \left(\mathbb {E} \left[ \left(\boldsymbol {v} ^ {2} (t)\right) ^ {\alpha} \right] + \mathbb {E} \left[ \left(\tilde {\boldsymbol {v}} ^ {2} (t)\right) ^ {\alpha} \right]\right) \to \infty ,
$$
where  $\tilde{\pmb{v}}^2 (t)\coloneqq e^{-t}\intop_0^t e^s\pmb {v}^2 (s)\mathrm{d}s$  is the exponential moving average.
A similar computation for  $\mathbf{u}$  will yield,
$$
\lim  _ {t \rightarrow \infty} \mathbb {E} \left[\left(\boldsymbol {u} ^ {2} (t)\right) ^ {\alpha} \right]\rightarrow \infty .
$$
Therefore for the limit of  $\pmb{a}_i, \| \pmb{w}_i \|$ , we have,
$$
\lim _ {t \to \infty} \mathbb {E} \left[ \left\| \pmb {w} _ {i} (t) \right\| ^ {2 \alpha} \right] \to \infty , \quad \lim _ {t \to \infty} \left(\mathbb {E} \left[ \left(\pmb {a} _ {i} ^ {2} (t)\right) ^ {\alpha} \right] + \mathbb {E} \left[ \left(\bar {\pmb {a}} _ {i} ^ {2} (t)\right) ^ {\alpha} \right]\right) \to \infty .
$$
Now we proceed to combine the above results and obtain the result on  $\mathbf{a}$ ,  $\mathbf{W}$ . Note that for  $0 < \alpha < 1$ ,  $x^{\alpha}$  is concave. Further using the Jensen's inequality, we have,
$$
\left(\left\| \boldsymbol {W} \right\| ^ {2}\right) ^ {\alpha} = \left(\sum_ {i = 1} ^ {d} \left\| \boldsymbol {w} _ {i} \right\| ^ {2}\right) ^ {\alpha} \geq d ^ {1 - \alpha} \sum_ {i = 1} ^ {d} \left\| \boldsymbol {w} _ {i} (t) \right\| ^ {2 \alpha}.
$$
Similar expression for a and taking the limit, we obtain, the result
$$
\lim  _ {t \to \infty} \mathbb {E} \left[ \left\| \boldsymbol {W} (t) \right\| ^ {\alpha} \right] \to \infty , \quad \lim  _ {t \to \infty} \mathbb {E} \left[ \left\| \mathbf {a} (t) \right\| ^ {\alpha} + \left\| \bar {\mathbf {a}} (t) \right\| ^ {\alpha} \right] \to \infty .
$$
Alignment. Let  $z_1, z_2 \ldots z_d$  be the rows of the matrix  $W$ . Now the evolution of the rows can be written as
$$
\dot {\boldsymbol {z}} _ {i} = \mathbf {a} (\mathrm {d} \boldsymbol {B} _ {t} ^ {i}),
$$
$$
\mathrm {d} \mathbf {a} = \sum_ {j = 1} ^ {d} z _ {j} \mathrm {d} B _ {t} ^ {j}.
$$
For any two matrices, with same dimensions define  $[u,v]\stackrel {\mathrm{def}}{=}uv^{\top} - vu^{\top}$
$$
\mathrm {d} [ \boldsymbol {z} _ {i}, \mathbf {a} ] = \mathrm {d} (\boldsymbol {z} _ {i} \mathbf {a} ^ {\top}) - \mathrm {d} (\mathbf {a} \boldsymbol {z} _ {i} ^ {\top}),
$$
Using the Itô chain rule,
$$
\begin{array}{l} \mathrm {d} \left(z _ {i} \mathbf {a} ^ {\top}\right) = \mathrm {d} z _ {i} \mathbf {a} ^ {\top} + z _ {i} \mathrm {d} b ^ {\top} + \mathrm {d} z _ {i} \mathrm {d} \mathbf {a}, \\ = \mathbf {a a} ^ {\top} (\mathrm {d} B _ {t} ^ {i}) + z _ {i} \left(\sum_ {j = 1} ^ {d} z _ {j} ^ {\top} \mathrm {d} B _ {t} ^ {j}\right) + \left(\mathbf {a} (\mathrm {d} B _ {t} ^ {i})\right) \left(\sum_ {j = 1} ^ {d} z _ {j} ^ {\top} \mathrm {d} B _ {t} ^ {j}\right), \\ = \mathbf {a a} ^ {\top} (\mathrm {d} B _ {t} ^ {i}) + \sum_ {j = 1} ^ {d} z _ {i} z _ {j} ^ {\top} \mathrm {d} B _ {t} ^ {j} + \mathbf {a} z _ {i} ^ {\top} \mathrm {d} t, \\ \end{array}
$$
$$
\mathrm {d} \left(\mathbf {a} \boldsymbol {z} _ {i} ^ {\top}\right) = \mathbf {a} \mathbf {a} ^ {\top} \left(\mathrm {d} \boldsymbol {B} _ {t} ^ {i}\right) + \boldsymbol {z} _ {i} \mathbf {a} ^ {\top} \mathrm {d} t + \sum_ {j = 1} ^ {d} \boldsymbol {z} _ {j} \boldsymbol {z} _ {i} ^ {\top} \mathrm {d} \boldsymbol {B} _ {t} ^ {j},
$$
$$
\mathrm {d} [ \boldsymbol {z} _ {i}, \mathbf {a} ] = - [ \boldsymbol {z} _ {i}, \mathbf {a} ] \mathrm {d} t + \sum_ {i = 1} ^ {d} [ \boldsymbol {z} _ {i}, \boldsymbol {z} _ {j} ] \mathrm {d} B _ {t} ^ {j},
$$
From the above evolution, we have that,
$$
\mathrm {d} \mathbb {E} \left[ \left[ \boldsymbol {z} _ {i}, \mathbf {a} \right] \right] = - \mathbb {E} \left[ \left[ \boldsymbol {z} _ {i}, \mathbf {a} \right] \right] \mathrm {d} t.
$$
$$
\mathbb {E} \left[ \left[ z _ {i}, \mathbf {a} \right] \right] = \left[ z _ {i} (0), \mathbf {a} (0) \right] e ^ {- t}.
$$
Let  $e_i \stackrel{\mathrm{def}}{=} [z_i, \mathbf{a}] [k, l]$ , be any  $(kl)^{th}$  entry of the matrix. Similarly, let  $c_{ij} \stackrel{\mathrm{def}}{=} [z_i, z_j] [k, l]$ .
$$
\mathrm {d} e _ {i} = - e _ {i} \mathrm {d} t + \sum_ {j} c _ {i j} \mathrm {d} B _ {t} ^ {j},
$$
$$
\mathrm {d} e _ {i} ^ {2} = - 2 e _ {i} \left(e _ {i} \mathrm {d} t + \sum_ {j} c _ {i j} \mathrm {d} B _ {t} ^ {j}\right) + \sum_ {j} c _ {i j} ^ {2} \mathrm {d} t,
$$
$$
\mathrm {d} e _ {i} ^ {2} = - 2 e _ {i} ^ {2} \mathrm {d} t + \sum_ {j} c _ {i j} ^ {2} \mathrm {d} t - 2 e _ {i} \sum_ {j} c _ {i j} \mathrm {d} B _ {t} ^ {j}.
$$
Again using the Ito formula and computing  $(e_i^2)^\alpha$  for some  $\alpha \in (0,1)$ , we get,
$$
\mathrm {d} \left(e _ {i} ^ {2}\right) ^ {\boldsymbol {\alpha}} = \boldsymbol {\alpha} \left(e _ {i} ^ {2}\right) ^ {\boldsymbol {\alpha} - 1} \mathrm {d} e _ {i} ^ {2} + \frac {1}{2} \boldsymbol {\alpha} (\boldsymbol {\alpha} - 1) \left(e _ {i} ^ {2}\right) ^ {\boldsymbol {\alpha} - 2} \mathrm {d} e _ {i} ^ {2} \mathrm {d} e _ {i} ^ {2},
$$
$$
\begin{array}{l} \mathrm {d} \left(e _ {i} ^ {2}\right) ^ {\boldsymbol {\alpha}} = \boldsymbol {\alpha} \left(e _ {i} ^ {2}\right) ^ {\boldsymbol {\alpha} - 1} \left[ - 2 e _ {i} ^ {2} \mathrm {d} t + \sum_ {j} c _ {i j} ^ {2} \mathrm {d} t - 2 e _ {i} \sum_ {j} c _ {i j} \mathrm {d} \boldsymbol {B} _ {t} ^ {j} \right] + \frac {1}{2} \boldsymbol {\alpha} (\boldsymbol {\alpha} - 1) \left(e _ {i} ^ {2}\right) ^ {\boldsymbol {\alpha} - 2} 4 e _ {i} ^ {2} \left[ \sum_ {j} c _ {i j} ^ {2} \right], \\ = - 2 \alpha \left(e _ {i} ^ {2}\right) ^ {\alpha} \mathrm {d} t + \alpha \left(e _ {i} ^ {2}\right) ^ {\alpha - 1} \sum_ {j} c _ {i j} ^ {2} \mathrm {d} t - 2 \alpha \left(e _ {i} ^ {2}\right) ^ {1 - \alpha} e _ {i} \sum_ {j} c _ {i j} \mathrm {d} B _ {t} ^ {j} + \frac {1}{2} \alpha (\alpha - 1) \left(e _ {i} ^ {2}\right) ^ {\alpha - 2} 4 e _ {i} ^ {2} \left[ \sum_ {j} c _ {i j} ^ {2} \right] \mathrm {d} t, \\ = - 2 \alpha \left(e _ {i} ^ {2}\right) ^ {\alpha} \mathrm {d} t + \alpha (2 \alpha - 1) \left(e _ {i} ^ {2}\right) ^ {\alpha - 1} \sum_ {j} c _ {i j} ^ {2} \mathrm {d} t - 2 \alpha \left(e _ {i} ^ {2}\right) ^ {\alpha - 1} e _ {i} \sum_ {j} c _ {i j} \mathrm {d} B _ {t} ^ {j}, \\ \end{array}
$$
Taking  $\alpha = 0.5$
$$
\mathrm {d} \left| e _ {i} \right| = - \left| e _ {i} \right| \mathrm {d} t - \left| e _ {i} \right| ^ {- 1} e _ {i} \sum_ {j} c _ {i j} \mathrm {d} B _ {t} ^ {j}.
$$
Taking expectation, we get,
$$
\mathrm {d} \mathbb {E} \left[ | e _ {i} | \right] = - \mathbb {E} \left[ | e _ {i} | \right] \mathrm {d} t.
$$
$$
\mathbb {E} \left[ | [ \boldsymbol {z} _ {i}, \mathbf {a} ] | \right] = | [ \boldsymbol {z} _ {i} (0), \mathbf {a} (0) ] | e ^ {- t}.
$$