SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \dot {\mathbf {D}} _ {\alpha} = - 2 \left(\tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2}\right) ^ {- 2} \mathbf {D} _ {\beta} \operatorname {d i a g} \left(\mathbf {R} ^ {\prime}\right) ^ {\top} \mathbf {D} _ {\beta} + \left(\tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2}\right) ^ {- 1} \dot {\mathbf {D}} _ {\beta} (C.38) \\ = - 2 \left(\tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2}\right) ^ {- 2} \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2} \operatorname {d i a g} \left(\mathbf {R} ^ {\prime}\right) + \left(\tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2}\right) ^ {- 1} \dot {\mathbf {D}} _ {\boldsymbol {\beta}}. (C.39) \\ \end{array}
$$
Finally, for dealing with the RHS of (C.36), we recall equation (C.12) which is implies that  $2\gamma (\tilde{\mathbf{D}}_{Z_1}^2)^{-1} = I_d - \tilde{\mathbf{D}}_\alpha^2$ . Putting together (C.39) and (C.36), we get that:
$$
- 2 \left(\tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2}\right) ^ {- 2} \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2} \operatorname {d i a g} \left(\mathbf {R} ^ {\prime}\right) + \left(\tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2}\right) ^ {- 1} \dot {\tilde {\mathbf {D}}} _ {\boldsymbol {\beta}} = 2 \gamma \left(\tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2}\right) ^ {- 1} \operatorname {d i a g} \left(\mathbf {R} ^ {\prime}\right), \tag {C.40}
$$
Therefore, we have the following string of implications:
$$
\begin{array}{l} \left(\mathrm {C}. 4 0\right) \Longleftrightarrow \frac {1}{2} \left[ \gamma I + \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2} \left(\tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2}\right) ^ {- 1} \right] ^ {- 1} \dot {\mathbf {D}} _ {\boldsymbol {\beta}} = \operatorname {d i a g} \left(\mathbf {R} ^ {\prime}\right) \\ \Longleftrightarrow \frac {1}{2} \tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2} \left[ \gamma (\tilde {\mathbf {D}} _ {\mathbf {Z} _ {1}} ^ {2}) + \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2} \right] ^ {- 1} \dot {\mathbf {D}} _ {\boldsymbol {\beta}} = \operatorname {d i a g} \left(\mathbf {R} ^ {\prime}\right) \\ \Longleftrightarrow \frac {1}{2} \left(\gamma I _ {d} + \sqrt {\gamma^ {2} I _ {d} + \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2}}\right) \left[ \gamma^ {2} I _ {d} + \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2} + \gamma \sqrt {\gamma^ {2} I _ {d} + \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2}} \right] ^ {- 1} \dot {\mathbf {D}} _ {\boldsymbol {\beta}} = \operatorname {d i a g} \left(\mathbf {R} ^ {\prime}\right) \\ \Longleftrightarrow \frac {1}{2} \left[ \sqrt {\gamma^ {2} I _ {d} + \tilde {\mathbf {D}} _ {\boldsymbol {\beta}} ^ {2}} \right] ^ {- 1} \dot {\mathbf {D}} _ {\boldsymbol {\beta}} = \operatorname {d i a g} \left(\mathbf {R} ^ {\prime}\right) \\ \Longrightarrow \frac {d \frac {1}{2} \sinh^ {- 1} (\frac {1}{\gamma} \mathbf {D} _ {\beta})}{d t} = \operatorname {d i a g} \left(\mathbf {R} ^ {\prime}\right). \\ \end{array}
$$
As a final step, we remark that  $\operatorname{diag}(\mathbf{R}') = -\nabla_{\mathbf{D}_{\beta}}\mathcal{L}(\beta)$  and that  $\nabla_{\mathbf{D}_{\beta}}\operatorname{tr}\left(\frac{1}{2}\mathbf{D}_{\beta}\sinh^{-1}(\mathbf{D}_{\beta}/\gamma) - \sqrt{\mathbf{D}_{\beta}^2 + \gamma^2}\right) = \sinh^{-1}(\frac{1}{\gamma}\mathbf{D}_{\beta})$
C.1 Extensions and further experiments
In this subsection, we describe how the insights from our analysis can be extended to relaxed assumptions on initialization, discrete gradient descent and non-linear activations.
Perturbations from assumption on initialization. Our analysis is contingent upon two assumptions regarding the initialization shape. The first assumption concerns the orthogonality of the initial feature layer  $\mathbf{W}_1(0)$ . In Figure 1a, we explored a scenario where the feature layer is initialized with a random Gaussian matrix, yet the evolution of singular values closely aligns with our theoretical analysis for orthogonal initialization. In Figure 4, we demonstrate the impact of zero initialization for the weight layer  $\mathbf{W}_2$ . We maintain the same experimental setup as in Figure 1a, but employ a random Gaussian initialization for  $\mathbf{W}_2$  with variance scales of  $10^{-2}$  and  $10^{-3}$ , while initializing  $\mathbf{W}_1$  with a variance of  $10^{-3}$ . In this context, the evolution of singular values continues to adhere to the predicted trend from our analysis.
Figure 4: The time evolution of singular values of the hidden layer weights of a 2-layer linear network when trained with gradient flow initialized with Gaussian random variables and non-zero  $W_{2}$ .
Discrete step size. Here we present a simpler problem to show how we can go beyond continuous time analysis. Consider the problem with  $l = d, k = 1$ ,  $W_{1}, W_{2} = W$ ,  $\mathbf{a}$  and  $W$  is initialized with
$I$ . The evolution of  $W, a$  with a learning rate  $\eta$  can be written as
$$
\boldsymbol {W} _ {t + 1} = \boldsymbol {W} _ {t} + \eta \boldsymbol {R} _ {t} \mathbf {a} _ {t} ^ {\top},
$$
$$
\mathbf {a} _ {t + 1} = \mathbf {a} _ {t} + \eta \boldsymbol {W} _ {t} ^ {\top} \mathbf {a} _ {t},
$$
$$
\boldsymbol {R} _ {t + 1} = \boldsymbol {R} _ {t} - \eta (X ^ {\top} X) (\boldsymbol {W} _ {t} \boldsymbol {W} _ {t} ^ {\top} + \| \mathbf {a} _ {t} \| ^ {2}) R _ {t} - \eta^ {2} (\mathbf {a} _ {t} ^ {\top} \boldsymbol {W} _ {t} \boldsymbol {R} _ {t}) (X ^ {\top} X) \boldsymbol {R} _ {t}.
$$
If we further assume that  $X^{\top}X = I$ , then it can be shown that  $R_{t},\mathbf{a}_{t}$  only grow in norm and do not change in direction. So the update of  $W_{t}$  is always aligned with the rank-1 matrix  $R_0\mathbf{a}_0^\top$ . Hence for small initialization, the final  $W_{\infty}$  is approximately a rank-1 matrix. This presents a way forward for the discrete step-size case and orthogonal data. With further analysis, we think this can be generalized to any data matrix satisfying the RIP conditions. This is not included in the main paper due to restrictive assumptions on the data but we will include these comments in the appendix.
Non-linear activations. We consider the same teacher-student setup as in the Figure 2. To completely characterize the dynamics and the final weight parameters is a challenging problem. The characterization of the dynamics at the small scale of initialization is also absent (for any general data matrix), Boursier et al. [2022] solves this in the case of orthogonal data. To study even this simple case of two neurons, one has to study the dynamics in two phases where one jumps from zero initialization (a saddle point) to another saddle and then further converge to zero train loss as seen in the Figure 5a. It is difficult to precisely characterize this intermediate saddle. With some careful additional work, we believe that our analysis can capture Phase 1 (where you jump to the first saddle) of the dynamics where you approximately learn rank 1 matrix, see Figure 5b, for general data matrices extending the current understanding. We hope this briefly sketches a way forward for ReLU networks.
(b)
Figure 5: (a) The training curve of the teacher-student network which follows a saddle-to-saddle dynamics. (b) The time evolution of singular values of the hidden layer weights of a 2-layer ReLU network when trained with gradient flow. The plot represents Phase 1 of the training where you first learn a (approximately) rank-1 hidden layer.
D Noise Dynamics
Noise Model. Here we consider the scalar case ( $k = 1$ ), abusing the notation  $\mathbf{W} = \mathbf{W}_1, \mathbf{a} = \mathbf{W}_2$ . The gradients of the loss  $\mathcal{L}(\mathbf{W}, \mathbf{a})$  are
$$
\nabla_ {\boldsymbol {W}} \mathcal {L} = - X ^ {\top} (Y - X \boldsymbol {W} \mathbf {a}) \mathbf {a} ^ {\top}, \quad \nabla_ {\mathbf {a}} \mathcal {L} = - \boldsymbol {W} ^ {\top} X ^ {\top} (Y - X \boldsymbol {W} \mathbf {a}).
$$
When the labels or outputs are doped with a noise of magnitude  $\delta > 0$ , i.e., adding  $\varepsilon \sim \delta \mathrm{N}(0, \mathbf{I}_n)$ . Now the gradients computed after doping with this label noise are
$$
\nabla_ {\boldsymbol {W}} \tilde {\mathcal {L}} = - X ^ {\top} (Y + \varepsilon - X \boldsymbol {W} \mathbf {a}) \mathbf {a} ^ {\top}, \quad \nabla_ {\mathbf {a}} \tilde {\mathcal {L}} = - \boldsymbol {W} ^ {\top} X ^ {\top} (Y + \varepsilon - X \boldsymbol {W} \mathbf {a}).
$$
$$
\nabla_ {\boldsymbol {W}} \tilde {\mathcal {L}} = \nabla_ {\boldsymbol {W}} \mathcal {L} - X ^ {\top} \varepsilon \mathbf {a} ^ {\top}, \quad \nabla_ {\mathbf {a}} \tilde {\mathcal {L}} = \nabla_ {\mathbf {a}} \mathcal {L} - \boldsymbol {W} ^ {\top} X ^ {\top} \varepsilon .
$$
Thus label noise gradient descent with step size  $\eta$  and with added label noise  $\varepsilon_{t}$  at each iteration writes
$$
\boldsymbol {W} _ {t + 1} = \boldsymbol {W} _ {t} - \eta \left(\nabla_ {\boldsymbol {W}} \mathcal {L} \left(\boldsymbol {W} _ {t}, \mathbf {a} _ {t}\right) - X ^ {\top} \varepsilon_ {t} \mathbf {a} _ {t} ^ {\top}\right).
$$
The continuous time version of this SDE writes,
$$
\mathrm {d} \boldsymbol {W} = - \eta \nabla_ {\boldsymbol {W}} \mathcal {L} (\boldsymbol {W}, \mathbf {a}) \mathrm {d} t + \eta \delta X ^ {\top} \mathrm {d} \boldsymbol {B} _ {t} \mathbf {a} ^ {\top}.
$$
Now, we consider the large noise regime, where the dominating term in the SDE is the diffusion term. Therefore we consider the SDE,
$$
\mathrm {d} \boldsymbol {W} = \eta \delta X ^ {\top} \mathrm {d} \boldsymbol {B} _ {t} \mathbf {a} ^ {\top},
$$
where  $B_{t}$  is a  $n$ -dimensional Brownian motion. Note that we can get rid of the term  $\eta \delta$  by re-scaling time by a constant factor. Similar steps for the evolution of  $\mathbf{a}$  gives the SDE,
$$
\mathrm {d} \boldsymbol {W} = X ^ {\top} \mathrm {d} \boldsymbol {B} _ {t} \mathbf {a} ^ {\top}, \quad \mathrm {d} \mathbf {a} = \boldsymbol {W} ^ {\top} X ^ {\top} \mathrm {d} \boldsymbol {B} _ {t}.
$$
Now consider the compact SVD decomposition of  $X$ , i.e.,  $X = UDV^{\top}$ , where  $U, D \in \mathbb{R}^{n \times n}, V \in \mathbb{R}^{d \times n}$  and  $UU^{\top} = U^{\top}U = V^{\top}V = \mathbf{I}_n$ .
$$
\mathrm {d} \boldsymbol {W} = V D U ^ {\top} \mathrm {d} \boldsymbol {B} _ {t} \mathbf {a} ^ {\top}, \quad \mathrm {d} \mathbf {a} = \boldsymbol {W} ^ {\top} V D U ^ {\top} \mathrm {d} \boldsymbol {B} _ {t},
$$
$$
\mathrm {d} V ^ {\top} \boldsymbol {W} = D \left(U ^ {\top} \mathrm {d} \boldsymbol {B} _ {t}\right) \mathbf {a} ^ {\top}, \quad \mathrm {d} \mathbf {a} = \left(V ^ {\top} \boldsymbol {W}\right) ^ {\top} D \left(U ^ {\top} \mathrm {d} \boldsymbol {B} _ {t}\right).
$$
Using Levy's characterization  $\mathbf{Y}_t = U^\top \mathbf{B}_t$  is a Brownian motion, since  $U$  is an orthogonal matrix. Let  $\bar{\mathbf{W}} = V^\top \mathbf{W}$ , then
$$
\mathrm {d} \tilde {\boldsymbol {W}} = D \mathrm {d} \boldsymbol {Y} _ {t} \mathbf {a} ^ {\top}, \quad \mathrm {d} \mathbf {a} = \tilde {\boldsymbol {W}} ^ {\top} D \mathrm {d} \boldsymbol {Y} _ {t}.
$$
Here we consider  $D = \mathbf{I}$  and change the notation to get the SDE 4.1 below. Our results can be extended to any diagonal matrix  $D$ .
$$
\mathrm {d} \boldsymbol {W} = (\mathrm {d} \boldsymbol {B} _ {t}) \mathbf {a} ^ {\top}, \quad \mathrm {d} \mathbf {a} = \boldsymbol {W} ^ {\top} \mathrm {d} \boldsymbol {B} _ {t}.
$$
$$
\lim  _ {t \rightarrow \infty} \mathbb {E} \left[\left\| \boldsymbol {W} (t) \right\| ^ {2} \right]\rightarrow \infty , \quad \lim  _ {t \rightarrow \infty} \mathbb {E} \left[\left\| \boldsymbol {\mathsf {a}} (t) \right\| ^ {2} \right]\rightarrow \infty .
$$
$$
\lim _ {t \to \infty} \mathbb {E} \left[ \left\| \boldsymbol {W} (t) \right\| ^ {\alpha} \right] \to \infty , \lim _ {t \to \infty} \mathbb {E} \left[ \left\| \mathbf {a} (t) \right\| ^ {\alpha} + \left\| \bar {\mathbf {a}} (t) \right\| ^ {\alpha} \right] \to \infty .
$$
$$
\lim  _ {t \rightarrow \infty} \mathbb {E} \left[\left|\left[ \boldsymbol {w} _ {i}, \mathbf {a} \right]\right|\right]\rightarrow 0.
$$
Proof. Consider the noise model Eq. (4.1),
$$
\mathrm {d} \boldsymbol {W} = \left(\mathrm {d} \boldsymbol {B} _ {t}\right) \mathbf {a} ^ {\top}, \quad \mathrm {d} \mathbf {a} = \boldsymbol {W} ^ {\top} \mathrm {d} \boldsymbol {B} _ {t}, \tag {D.1}
$$
Variance. Let  $\boldsymbol{w}_i$  be the  $i^{th}$  column of  $\mathbf{W}$  and  $\boldsymbol{a}_i$  be the  $i^{th}$  coordinate of  $\mathbf{a}$ . For any  $i \in [n]$ , the diffusion of the quantities can be separately written as
$$
\mathrm {d} \boldsymbol {w} _ {i} = \boldsymbol {a} _ {i} \mathrm {d} \boldsymbol {B} _ {t}, \quad \mathrm {d} \boldsymbol {a} _ {i} = \langle \boldsymbol {w} _ {i}, \mathrm {d} \boldsymbol {B} _ {t} \rangle .
$$
Using the Ito chain rule,
$$
\begin{array}{l} \left. \mathrm {d} \left\| \boldsymbol {w} _ {i} \right\| ^ {2} = 2 \langle \boldsymbol {w} _ {i}, \mathrm {d} \boldsymbol {w} _ {i} \rangle + \langle \mathrm {d} \boldsymbol {w} _ {i}, \mathrm {d} \boldsymbol {w} _ {i} \rangle , \right. \\ = d \left\| \boldsymbol {a} _ {i} \right\| ^ {2} \mathrm {d} t + 2 \boldsymbol {a} _ {i} \left\langle \boldsymbol {w} _ {i}, \mathrm {d} \boldsymbol {B} _ {t} \right\rangle . \\ \end{array}
$$
$$
\mathrm {d} \boldsymbol {a} _ {i} ^ {2} = 2 \boldsymbol {a} _ {i} \mathrm {d} \boldsymbol {a} _ {i} + \mathrm {d} \boldsymbol {a} _ {i} \mathrm {d} \boldsymbol {a} _ {i} = 2 \boldsymbol {a} _ {i} \left\langle \boldsymbol {w} _ {i}, \mathrm {d} \boldsymbol {B} _ {t} \right\rangle + \left\| \boldsymbol {w} _ {i} \right\| ^ {2} \mathrm {d} t.
$$
Note that  $\langle \pmb{w}_i, \mathrm{d}\pmb{B}_t \rangle \sim \| \pmb{w}_i\| \mathrm{d}\tilde{\pmb{B}}_t$  for some one-dimensional Brownian motion  $(\tilde{B}_t)_{t \geq 0}$ .
$$
\mathrm {d} \boldsymbol {a} _ {i} ^ {2} = \left\| \boldsymbol {w} _ {i} \right\| ^ {2} \mathrm {d} t + 2 \boldsymbol {a} _ {i} \left\| \boldsymbol {w} _ {i} \right\| \mathrm {d} \tilde {\boldsymbol {B}} _ {t}, \quad \mathrm {d} \left\| \boldsymbol {w} _ {i} \right\| ^ {2} = d \left\| \boldsymbol {a} _ {i} \right\| ^ {2} \mathrm {d} t + 2 \boldsymbol {a} _ {i} \left\| \boldsymbol {w} _ {i} \right\| \mathrm {d} \tilde {\boldsymbol {B}} _ {t}.
$$
Let  $\pmb{v} \coloneqq \pmb{a}_i$ ,  $\pmb{u} \coloneqq \left\| \pmb{w}_i \right\|$ , using this notation we get,
$$
\mathrm {d} \boldsymbol {u} ^ {2} = d \boldsymbol {v} ^ {2} \mathrm {d} t + 2 \boldsymbol {u} \boldsymbol {v} \mathrm {d} \tilde {\boldsymbol {B}} _ {t}, \quad \mathrm {d} \boldsymbol {v} ^ {2} = \boldsymbol {u} ^ {2} \mathrm {d} t + 2 \boldsymbol {u} \boldsymbol {v} \mathrm {d} \tilde {\boldsymbol {B}} _ {t}. \tag {D.2}
$$
Let  $\pmb{u}_0\coloneqq \mathbb{E}\left[\pmb {u}^2\right],\pmb {v}_0\coloneqq \mathbb{E}\left[\pmb {v}^2\right]$ . Using the Dynkins formula, taking the expectation, we get,
$$
\mathrm {d} \mathbb {E} [ \boldsymbol {u} ^ {2} ] = d \mathbb {E} [ \boldsymbol {v} ^ {2} ] \mathrm {d} t, \quad \mathrm {d} \mathbb {E} [ \boldsymbol {v} ^ {2} ] = \mathbb {E} [ \boldsymbol {u} ^ {2} ],
$$
$$
\mathrm {d} \boldsymbol {u} _ {0} = d \boldsymbol {v} _ {0} \mathrm {d} t, \quad \mathrm {d} \boldsymbol {v} _ {0} = \boldsymbol {u} _ {0} \mathrm {d} t.
$$
This system can be transformed into,
$$
\mathrm {d} \left(\boldsymbol {u} _ {0} + \sqrt {d} \boldsymbol {v} _ {0}\right) = \sqrt {d} \left(\boldsymbol {u} _ {0} + \sqrt {n} \boldsymbol {v} _ {0}\right) \mathrm {d} t.
$$
Solving the above ODE we get,
$$
\left(\boldsymbol {u} _ {0} + \sqrt {d} \boldsymbol {v} _ {0}\right) = c _ {0} e ^ {\sqrt {d t}}, \quad \text {w h e r e} c _ {0} = \boldsymbol {u} _ {0} (0) + \sqrt {d} \boldsymbol {v} _ {0} (0).
$$
$$
\mathrm {d} \left(\boldsymbol {u} _ {0} - \sqrt {d} \boldsymbol {v} _ {0}\right) = - \sqrt {d} \left(\boldsymbol {u} _ {0} - \sqrt {d} \boldsymbol {v} _ {0}\right) \mathrm {d} t,
$$
$$
\left(\boldsymbol {u} _ {0} - \sqrt {d} \boldsymbol {v} _ {0}\right) = c _ {1} e ^ {- \sqrt {d} t}, \quad \text {w h e r e} c _ {1} = \boldsymbol {u} _ {0} (0) - \sqrt {d} \boldsymbol {v} _ {0} (0).
$$
$$
\boldsymbol {u} _ {0} = \frac {1}{2} \left[ c _ {0} e ^ {\sqrt {d t}} + c _ {1} e ^ {- \sqrt {d t}} \right], \quad \boldsymbol {v} _ {0} = \frac {1}{2 \sqrt {d}} \left[ c _ {0} e ^ {\sqrt {d t}} - c _ {1} e ^ {- \sqrt {d t}} \right].
$$
Taking the limit proves the first part of the result.
Scale. From Eq. D.2, we have,
$$
\mathrm {d} \left(\boldsymbol {u} ^ {2} + \sqrt {n} \boldsymbol {v} ^ {2}\right) = \sqrt {n} \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) + 2 (\sqrt {d} + 1) \boldsymbol {u v} \mathrm {d} \tilde {\boldsymbol {B}} _ {t}.
$$
Again using the Itô chain rule, we get,
$$
\begin{array}{l} \mathrm {d} \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\alpha} = \alpha \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\alpha - 1} \mathrm {d} \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) \\ + \frac {1}{2} \boldsymbol {\alpha} (\boldsymbol {\alpha} - 1) \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\alpha - 2} d \left(\boldsymbol {u} ^ {2} + \sqrt {n} \boldsymbol {v} ^ {2}\right) d \left(\boldsymbol {u} ^ {2} + \sqrt {n} \boldsymbol {v} ^ {2}\right), \\ = \boldsymbol {\alpha} \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\boldsymbol {\alpha} - 1} \left[ \sqrt {d} \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) d t + 2 \left(\sqrt {d} + 1\right) \boldsymbol {u v} d \tilde {\boldsymbol {B}} _ {t} \right] \\ + \frac {1}{2} \boldsymbol {\alpha} (\boldsymbol {\alpha} - 1) \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\boldsymbol {\alpha} - 2} \left(4 \left(\sqrt {d} + 1\right) ^ {2} \boldsymbol {u} ^ {2} \boldsymbol {v} ^ {2}\right) d t, \\ \end{array}
$$
The drift term is
$$
\begin{array}{l} \boldsymbol {\alpha} \sqrt {n} \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\boldsymbol {\alpha}} + 2 \boldsymbol {\alpha} (\boldsymbol {\alpha} - 1) \left(\sqrt {d} + 1\right) ^ {2} \left(\boldsymbol {u} ^ {2} + \sqrt {n} \boldsymbol {v} ^ {2}\right) ^ {\boldsymbol {\alpha} - 2} \boldsymbol {u} ^ {2} \boldsymbol {v} ^ {2} \\ = \boldsymbol {\alpha} \left(\boldsymbol {u} ^ {2} + \sqrt {n} \boldsymbol {v} ^ {2}\right) ^ {\boldsymbol {\alpha}} \left[ \sqrt {d} + 2 (\boldsymbol {\alpha} - 1) \left(\sqrt {d} + 1\right) ^ {2} \frac {\boldsymbol {u} ^ {2} \boldsymbol {v} ^ {2}}{\left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {2}} \right]. \\ \end{array}
$$
Again using the Dynkins formula for the evolution of expectation, we have,
$$
\mathrm {d} \mathbb {E} \left[ \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\alpha} \right] = \mathbb {E} \left[ \boldsymbol {\alpha} \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\alpha} \left[ \sqrt {d} + 2 (\boldsymbol {\alpha} - 1) (\sqrt {d} + 1) ^ {2} \frac {\boldsymbol {u} ^ {2} \boldsymbol {v} ^ {2}}{\left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {2}} \right] \right] \mathrm {d} t. \tag {D.3}
$$
For any function,
$$
g (y) = \frac {y}{\left(y + \sqrt {d}\right) ^ {2}},
$$
attains its maximum value at  $y = \sqrt{d}$ , i.e.,  $g(\sqrt{n}) = 1 / (4\sqrt{d})$ . Note that  $\alpha < 1$  and we have
$$
2 (\boldsymbol {\alpha} - 1) \left(\sqrt {n} + 1\right) ^ {2} \frac {\boldsymbol {u} ^ {2} \boldsymbol {v} ^ {2}}{\left(\boldsymbol {u} ^ {2} + \sqrt {n} \boldsymbol {v} ^ {2}\right) ^ {2}} \geq 2 (\boldsymbol {\alpha} - 1) \left(\sqrt {n} + 1\right) ^ {2} g (\sqrt {n}) = (\boldsymbol {\alpha} - 1) \frac {\left(\sqrt {n} + 1\right) ^ {2}}{2 \sqrt {n}}.
$$
The drift can be lower bounded as the following,
$$
\begin{array}{l} \boldsymbol {\alpha} \sqrt {n} \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\boldsymbol {\alpha}} + 2 \boldsymbol {\alpha} (\boldsymbol {\alpha} - 1) \left(\sqrt {d} + 1\right) ^ {2} \left(\boldsymbol {u} ^ {2} + \sqrt {n} \boldsymbol {v} ^ {2}\right) ^ {\boldsymbol {\alpha} - 2} \boldsymbol {u} ^ {2} \boldsymbol {v} ^ {2} \\ \geq \boldsymbol {\alpha} \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\boldsymbol {\alpha}} \left[ \sqrt {d} + (\boldsymbol {\alpha} - 1) \frac {\left(\sqrt {d} + 1\right) ^ {2}}{2 \sqrt {d}} \right]. \\ \end{array}
$$
For  $d \geq 5$  and  $0 < \alpha < 1$ , we have  $c_0 > 0$ ,
$$
\left[ \sqrt {d} + (\boldsymbol {\alpha} - 1) \frac {(\sqrt {d} + 1) ^ {2}}{2 \sqrt {d}} \right] > c _ {0}.
$$
Using the above expression in Eq. D.3,
$$
\mathrm {d} \mathbb {E} \left[ \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\alpha} \right] \geq \alpha c _ {0} \mathbb {E} \left[ \left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\alpha} \right] \mathrm {d} t.
$$
Taking the limit,
$$
\lim  _ {t \rightarrow \infty} \mathbb {E} \left[\left(\boldsymbol {u} ^ {2} + \sqrt {d} \boldsymbol {v} ^ {2}\right) ^ {\boldsymbol {\alpha}} \right]\rightarrow \infty .
$$
From the SDE (D.2), we obtain the following process with only diffusion,
$$
\mathrm {d} \left(\boldsymbol {u} ^ {2} - \boldsymbol {v} ^ {2}\right) = \left(d \boldsymbol {v} ^ {2} - \boldsymbol {u} ^ {2}\right) \mathrm {d} t,
$$
$$
\mathrm {d} \left(\boldsymbol {u} ^ {2} - \boldsymbol {v} ^ {2}\right) + \left(\boldsymbol {u} ^ {2} - \boldsymbol {v} ^ {2}\right) \mathrm {d} t = (n - 1) \boldsymbol {v} ^ {2},
$$
$$
e ^ {t} \mathrm {d} (\boldsymbol {u} ^ {2} - \boldsymbol {v} ^ {2}) + e ^ {t} (\boldsymbol {u} ^ {2} - \boldsymbol {v} ^ {2}) \mathrm {d} t = (n - 1) e ^ {t} \boldsymbol {v} ^ {2} \mathrm {d} t,
$$
$$
\mathrm {d} e ^ {t} \left(\boldsymbol {u} ^ {2} - \boldsymbol {v} ^ {2}\right) = (n - 1) e ^ {t} \boldsymbol {v} ^ {2} \mathrm {d} t,
$$
$$
\left(\boldsymbol {u} ^ {2} \left(t _ {2}\right) - \boldsymbol {v} ^ {2} \left(t _ {2}\right)\right) = e ^ {- \left(t _ {2} - t _ {1}\right)} \left(\boldsymbol {u} ^ {2} \left(t _ {1}\right) - \boldsymbol {v} ^ {2} \left(t _ {1}\right)\right) + (n - 1) \int_ {t _ {1}} ^ {t _ {2}} e ^ {- \left(t _ {2} - t\right)} \boldsymbol {v} ^ {2} (t) d t,
$$
$$
\left(\boldsymbol {u} ^ {2} \left(t _ {2}\right) + \sqrt {n} \boldsymbol {v} ^ {2} \left(t _ {2}\right)\right) = e ^ {- \left(t _ {2} - t _ {1}\right)} \left(\boldsymbol {u} ^ {2} \left(t _ {1}\right) - \boldsymbol {v} ^ {2} \left(t _ {1}\right)\right) + \sqrt {n} \boldsymbol {v} ^ {2} \left(t _ {2}\right) + (n - 1) e ^ {- t _ {2}} \int_ {t _ {1}} ^ {t _ {2}} e ^ {t} \boldsymbol {v} ^ {2} (t) d t,
$$
Denote  $C(t) \coloneqq \pmb{u}^2(t) + \sqrt{d}\pmb{v}^2(t)$ , using the fact that  $\pmb{u}^2(t_1) - \pmb{v}^2(t_1) \leq C(t_1)$ ,
$$
C \left(t _ {2}\right) \leq e ^ {- \left(t _ {2} - t _ {1}\right)} C \left(t _ {1}\right) + \sqrt {d} v ^ {2} \left(t _ {2}\right) + (d - 1) e ^ {- t _ {2}} \int_ {t _ {1}} ^ {t _ {2}} e ^ {t} v ^ {2} (t),
$$
With  $t_1 = 0$ , using the notation  $\tilde{\pmb{v}}^2(t) \coloneqq e^{-t} \int_{0}^{t} e^s \pmb{v}^2(s) \, \mathrm{d}s$ , to denote the exponential moving average.
For any time  $t > 0$ , we have,
$$
C (t) \leq e ^ {- (t)} C (0) + d (\boldsymbol {v} ^ {2} (t) + \tilde {\boldsymbol {v}} ^ {2} (t)).
$$
Raising to the power of  $\alpha$
$$
C (t) ^ {\alpha} \leq \left[ e ^ {- t} C (0) + d (\boldsymbol {v} ^ {2} (t) + \tilde {\boldsymbol {v}} ^ {2} (t)) \right] ^ {\alpha}.
$$
For  $a, b > 0$  and  $0 < p < 1$ , we have  $(a + b)^p \leq a^p + b^p$ . Using the inequality,
$$
C (t) ^ {\alpha} \leq e ^ {- \alpha t} C (0) ^ {\alpha} + d ^ {\alpha} (\boldsymbol {v} ^ {2} (t)) ^ {\alpha} + d ^ {\alpha} (\tilde {\boldsymbol {v}} ^ {2} (t)) ^ {\alpha}.
$$
Taking the expectation,
$$
\mathbb {E} \left[ C (t) ^ {\alpha} \right] \leq e ^ {- \alpha t} \mathbb {E} \left[ C (0) ^ {\alpha} \right] + d ^ {\alpha} \mathbb {E} \left[ (\boldsymbol {v} ^ {2} (t)) ^ {\alpha} \right] + d ^ {\alpha} \mathbb {E} \left[ (\tilde {\boldsymbol {v}} ^ {2} (t)) ^ {\alpha} \right].
$$
Now, we proceed by taking the limit,
$$
\lim  _ {t \rightarrow \infty} \mathbb {E} \left[ C (t) ^ {\alpha} \right] \leq e ^ {- \alpha t} \lim  _ {t \rightarrow \infty} \mathbb {E} \left[ C (0) ^ {\alpha} \right] + d ^ {\alpha} \lim  _ {t \rightarrow \infty} \left[ \mathbb {E} \left[ (\boldsymbol {v} ^ {2} (t)) ^ {\alpha} \right] + \mathbb {E} \left[ (\tilde {\boldsymbol {v}} ^ {2} (t)) ^ {\alpha} \right]\right].
$$
Thus, we obtain,