SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {d}{d t} L \left(\rho_ {t}\right) = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho_ {t}} (x) - y (x)\right) \frac {d}{d t} f _ {\rho_ {t}} (x) \right] (D.327) \\ = - \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \underset {u \sim \rho_ {t}} {\mathbb {E}} \left[ \left(f _ {\rho_ {t}} (x) - y (x)\right) \sigma^ {\prime} (u \cdot x) x \cdot \operatorname {g r a d} _ {u} L (\rho_ {t}) \right] (D.328) \\ = - \underset {u \sim \rho_ {t}} {\mathbb {E}} \left[ \operatorname {g r a d} _ {u} L \left(\rho_ {t}\right) \cdot \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho_ {t}} (x) - y (x)\right) \sigma^ {\prime} (u \cdot x) x \right] \right] (D.329) \\ = - \underset {u \sim \rho_ {t}} {\mathbb {E}} \left[ \operatorname {g r a d} _ {u} L \left(\rho_ {t}\right) \cdot \nabla_ {u} L \left(\rho_ {t}\right) \right] (D.330) \\ = - \underset {u \sim \rho_ {t}} {\mathbb {E}} \left[ \| \operatorname {g r a d} _ {u} L (\rho_ {t}) \| _ {2} ^ {2} \right]. \\ (\mathrm {B . c .} \operatorname {g r a d} _ {u} = (I - u u ^ {\top}) \nabla_ {u} L (\rho_ {t}) \text {a n d} (I - u u ^ {\top}) ^ {2} = (I - u u ^ {\top})) \\ \end{array}
$$
The above lemma implies that  $L(\rho_t)$  is always decreasing. Note that frequently, when we apply the above lemma, we will only consider the first coordinate of  $\mathrm{grad}_u L(\rho_t)$ .
Proposition D.40. For any distribution  $\rho$  on  $\mathbb{R}$
$$
\underset {w, w ^ {\prime} \sim \rho} {\mathbb {E}} [ | w - w ^ {\prime} | ^ {2} ] = 2 \operatorname {V a r} _ {w \sim \rho} [ w ] \tag {D.331}
$$
Proof of Proposition D.40. This follows from a short calculation:
$$
\underset {w, w ^ {\prime} \sim \rho} {\mathbb {E}} [ | w - w ^ {\prime} | ^ {2} ] = \underset {w, w ^ {\prime} \sim \rho} {\mathbb {E}} \left(w ^ {2} - 2 w w ^ {\prime} + (w ^ {\prime}) ^ {2}\right) = 2 \underset {w \sim \rho} {\mathbb {E}} [ w ^ {2} ] - 2 \left(\underset {w \sim \rho} {\mathbb {E}} [ w ]\right) ^ {2} = 2 \underset {w \sim \rho} {\operatorname {V a r}} [ w ] \tag {D.332}
$$
E Proofs for Finite-Samples and Finite-Width Dynamics
In this section we give the proof for results related to the empirical dynamics, where the number of samples and network width are both polynomial in  $d$ . The end goal is to prove Theorem 3.4, the proof of which is given below.
Proof of Theorem 3.4. We have
$$
L \left(\hat {\rho} _ {T _ {*}, \epsilon}\right) = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\hat {\rho} _ {T _ {*}, \epsilon}} (x) - y (x)\right) ^ {2} \right] \tag {E.1}
$$
$$
\begin{array}{l} \leq 3 \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\hat {\rho} _ {T _ {*}, \epsilon}} (x) - f _ {\bar {\rho} _ {T _ {*}, \epsilon}} (x)) ^ {2} ] + 3 \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\bar {\rho} _ {T _ {*}, \epsilon}} (x) - f _ {\rho_ {T _ {*}, \epsilon}} (x)) ^ {2} ] (E.2) \\ + 3 \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho_ {T _ {*}, \epsilon}} (x) - y (x)\right) ^ {2} \right] (E.3) \\ \end{array}
$$
$$
\lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \frac {1}{d ^ {\frac {\mu - 1}{2}}} + \frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) d ^ {2} (\log d) ^ {O (1)}}{m} + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \epsilon^ {2}. \tag {E.4}
$$
where the first inequality is by Cauchy-Schwarz inequality, and the second inequality is by Lemma 5.1, Lemma 5.2 and Theorem 3.3. Since  $\epsilon = \frac{1}{\log\log d}$ , the last term dominates because the conditions of Theorem 3.4 imply that  $m \gtrsim d^3$ , and because  $\frac{\mu - 1}{2} > 1$ , meaning that the first term is at most  $\frac{\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2}{d}$ .
To complete the proof, we give proofs for results in Section 5. We first give a proof for Lemma 5.1:
Proof of Lemma 5.1. For the first statement, for any fixed  $t \leq T$ , by Lemma E.13, we have
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho_ {t}} (x) - f _ {\bar {\rho} _ {t}} (x)\right) ^ {2} \right] \lesssim \frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) \left(d ^ {2} + \log \frac {1}{\delta}\right) (\log d) ^ {C}}{m} + \frac {\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}}{d ^ {\Omega (\log d)}} \tag {E.5}
$$
with probability  $1 - \delta$  — we will choose  $\delta$  shortly. In addition, for any  $t, t' \in [0, T]$ , by Lemma E.14, we have that  $\| u_t(\chi) - u_{t'}(\chi) \|_2 \lesssim (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2) d^4 |t - t'|$ , which by Lemma E.12 implies that for any  $x \in \mathbb{S}^{d-1}$ ,
$$
\left| f _ {\rho_ {t}} (x) - f _ {\rho_ {t ^ {\prime}}} (x) \right| \leq \underset {\chi \sim \rho_ {0}} {\mathbb {E}} \left| \sigma \left(u _ {t} (\chi) \cdot x\right) - \sigma \left(u _ {t ^ {\prime}} (\chi) \cdot x\right) \right| \tag {E.6}
$$
$$
\lesssim \left(| \hat {\sigma} _ {2, d} | \sqrt {N _ {2 , d}} + | \hat {\sigma} _ {4, d} | \sqrt {N _ {4 , d}}\right) \cdot \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {4} | t - t ^ {\prime} | \tag {E.7}
$$
In particular, if  $|t - t'| \leq \frac{1}{(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)d^{\Omega(\log d)}}$ , then for all  $x \in \mathbb{S}^{d-1}$  we have
$$
\left| f _ {\rho_ {t}} (x) - f _ {\rho_ {t ^ {\prime}}} (x) \right| \lesssim \frac {\left| \hat {\sigma} _ {2 , d} \right| + \left| \hat {\sigma} _ {4 , d} \right|}{d ^ {\Omega (\log d)}} \tag {E.8}
$$
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho_ {t}} (x) - f _ {\rho_ {t ^ {\prime}}} (x)\right) ^ {2} \right] \lesssim \frac {\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}}{d ^ {\Omega (\log d)}} \tag {E.9}
$$
By the same argument, we have that
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\bar {\rho} _ {t}} (x) - f _ {\bar {\rho} _ {t ^ {\prime}}} (x)\right) ^ {2} \right] \lesssim \frac {\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}}{d ^ {\Omega (\log d)}} \tag {E.10}
$$
Now, we perform a union bound over all times  $t$  such that  $t \leq T$  and  $t$  is an integer multiple of  $\frac{1}{(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)d^{H\log d}}$  for some sufficiently large absolute constant  $H > 0$ . Since  $T \leq \frac{d^C}{\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2}$ , the number of such times is at most  $d^{C + H\log d}$ . Thus, with probability at least  $1 - \delta \cdot d^{C + H\log d}$ , we have
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\rho_ {t}} (x) - f _ {\bar {\rho} _ {t}} (x)) ^ {2} ] \lesssim \frac {(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}) (d ^ {2} + \log \frac {1}{\delta}) (\log d) ^ {O (1)}}{m} + \frac {\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}}{d ^ {\Omega (\log d)}} \tag {E.11}
$$
for all  $t \in [0, T]$ . In particular, in order to have exponentially small failure probability, we set  $\delta = \frac{1}{e^{d^2} d^{C + H \log d}}$ , and by our assumption that  $m \leq d^{O(\log d)}$ , we obtain
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho_ {t}} (x) - f _ {\bar {\rho} _ {t}} (x)\right) ^ {2} \right] \lesssim \frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) d ^ {2} (\log d) ^ {O (1)}}{m} \tag {E.12}
$$
for all  $t \leq T$ , with probability at least  $1 - e^{-d^2}$ , since  $\log \frac{1}{\delta} = d^2 + O((\log d)^2)$ . This completes the proof of the first statement of the lemma. Additionally, by Lemma E.16, we have
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\hat {\rho} _ {t}} (x) - f _ {\bar {\rho} _ {t}} (x)\right) ^ {2} \right] \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \underset {(\bar {u}, \hat {u}) \sim \Gamma} {\mathbb {E}} \left[ \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} \right] \tag {E.13}
$$
which proves the second statement of the lemma.
The rest of the section is dedicated to prove Lemma 5.2. To begin with, we first introduce the following lemma which formalizes the decomposition of  $\frac{d}{dt} \| \hat{u}_t - \bar{u}_t \|_2^2$  discussed in Section 5.
Lemma E.1 (Decomposition of  $\frac{d}{dt}\| \hat{u}_t - \bar{u}_t\| _2^2$ ). Suppose we are in the setting of Theorem 3.4. Then,  $\frac{d}{dt}\| \hat{u}_t - \bar{u}_t\| _2^2 = A_t + B_t + C_t$ , where  $A_{t} = -2\langle \mathrm{grad}_{\hat{u}}L(\rho_{t}) - \mathrm{grad}_{\bar{u}}L(\rho_{t}),\hat{u}_{t} - \bar{u}_{t}\rangle$ ,  $B_{t} = -2\langle \mathrm{grad}_{\hat{u}}L(\hat{\rho}_{t}) - \mathrm{grad}_{\hat{u}}L(\rho_{t}),\hat{u}_{t} - \bar{u}_{t}\rangle$ , and  $C_t = -2\langle \mathrm{grad}_{\hat{u}}\widehat{L} (\hat{\rho}_t) - \mathrm{grad}_{\hat{u}}L(\hat{\rho}_t),\hat{u}_t - \bar{u}_t\rangle$
Proof of Lemma E.1. By the chain rule,
$$
\frac {d}{d t} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} = 2 \left\langle \frac {d}{d t} \hat {u} _ {t} - \frac {d}{d t} \bar {u} _ {t}, \hat {u} _ {t} - \bar {u} _ {t} \right\rangle = - 2 \left\langle \operatorname {g r a d} _ {\hat {\rho}} \widehat {L} \left(\hat {\rho} _ {t}\right) - \operatorname {g r a d} _ {\bar {u}} L \left(\rho_ {t}\right), \hat {u} _ {t} - \bar {u} _ {t} \right\rangle \tag {E.14}
$$
We can expand the difference in gradients as
$$
\begin{array}{l} \operatorname {g r a d} _ {\hat {\rho}} \widehat {L} (\hat {\rho} _ {t}) - \operatorname {g r a d} _ {\bar {u}} L (\rho_ {t}) = \left(\operatorname {g r a d} _ {\hat {u}} \widehat {L} (\hat {\rho} _ {t}) - \operatorname {g r a d} _ {\hat {u}} L (\hat {\rho} _ {t})\right) + \left(\operatorname {g r a d} L _ {\hat {u}} (\hat {\rho} _ {t}) - \operatorname {g r a d} L _ {\bar {u}} (\rho_ {t})\right) (E.15) \\ + \left(\operatorname {g r a d} _ {\hat {u}} L \left(\rho_ {t}\right) - \operatorname {g r a d} _ {\bar {u}} L \left(\rho_ {t}\right)\right) (E.16) \\ \end{array}
$$
$$
\begin{array}{l} \frac {d}{d t} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} = - 2 \left\langle \operatorname {g r a d} _ {\hat {u}} \widehat {L} \left(\hat {\rho} _ {t}\right) - \operatorname {g r a d} _ {\hat {u}} L \left(\hat {\rho} _ {t}\right), \hat {u} _ {t} - \bar {u} _ {t} \right\rangle - 2 \left\langle \operatorname {g r a d} L _ {\hat {u}} \left(\hat {\rho} _ {t}\right) - \operatorname {g r a d} L _ {\hat {u}} \left(\rho_ {t}\right), \hat {u} _ {t} - \bar {u} _ {t} \right\rangle (E.17) \\ - 2 \left\langle \operatorname {g r a d} _ {\bar {u}} L \left(\rho_ {t}\right) - \operatorname {g r a d} _ {\bar {u}} L \left(\rho_ {t}\right), \hat {u} _ {t} - \bar {u} _ {t} \right\rangle (E.18) \\ = C _ {t} + B _ {t} + A _ {t} (E.19) \\ \end{array}
$$
Next, we give a proof of Lemma 5.4, which gives an upper bound for each of  $A_{t}$ ,  $B_{t}$  and  $C_{t}$ .
Proof of Lemma 5.4. The proof directly follows from Lemma E.8, Lemma E.7, Lemma E.9, Lemma E.10 and Lemma E.11.  $\square$
Combining the upper bounds in Lemma 5.4 and the decomposition in Lemma E.1, we have the following Lemma E.2 which upper bounds the growth of  $\| \hat{u}_t - \bar{u}_t\|$ . Recall that  $T_{*,\epsilon}$  is the total runtime for the algorithm (see Theorem 3.3), and  $T_{1}$  is the time such that  $w_{T_1}(\iota_{\mathrm{U}}) = \frac{1}{\log d}$  (see Definition 4.5).
Lemma E.2 (Growth Rate of  $\| \delta_t\| _2^2$  With Dependency on  $n$ ). In the setting of Theorem 3.4, suppose that the inductive hypothesis in Assumption 5.3 holds for some  $\phi$  and  $\psi$  up until  $t$  for some  $t\leq T_{*,\epsilon}$ . Further assume that  $\psi < \frac{1}{2}$  and  $\phi >\frac{1}{2}$ . Then, when  $t\leq T_1$  we have
$$
\begin{array}{l} \frac {d}{d t} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} \leq \left(4 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} |\right) \left(1 + O \left(\frac {1}{\log d}\right)\right) \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} (E.20) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \cdot \frac {d (\log d) ^ {O (1)}}{\sqrt {m}} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} (E.21) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (\log d) ^ {O (1)} \left(\frac {1}{d ^ {\phi}} + \sqrt {\frac {d}{n}} + \frac {d ^ {2 . 5 - 2 \psi - 2 \phi}}{n}\right) \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} (E.22) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (\log d) ^ {O (1)} \left(\frac {d ^ {2 - 2 \psi - \phi}}{n} + \frac {d ^ {4 - 4 \psi - 2 \phi}}{n}\right) \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} (E.23) \\ \end{array}
$$
$$
\begin{array}{l} \frac {d}{d t} \bar {\Delta} _ {t} ^ {2} \leq \left(4 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} |\right) \left(1 + O \left(\frac {1}{\log d}\right)\right) \bar {\Delta} _ {t} ^ {2} (E.24) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \cdot \frac {d (\log d) ^ {O (1)}}{\sqrt {m}} \bar {\Delta} _ {t} (E.25) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (\log d) ^ {O (1)} \left(\sqrt {\frac {d}{n}} + \frac {d ^ {2 . 5 - 2 \psi - 2 \phi}}{n}\right) \bar {\Delta} _ {t} (E.26) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (\log d) ^ {O (1)} \left(\frac {1}{d ^ {\phi}} + \frac {d ^ {2 - 2 \psi - \phi}}{n} + \frac {d ^ {4 - 4 \psi - 2 \phi}}{n}\right) \bar {\Delta} _ {t} ^ {2}. (E.27) \\ \end{array}
$$
When  $t\in [T_1,T_{*,\epsilon}]$  , we have
$$
\begin{array}{l} \frac {d}{d t} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \gamma_ {2} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} (E.28) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \cdot \frac {d (\log d) ^ {O (1)}}{\sqrt {m}} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} (E.29) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (\log d) ^ {O (1)} \left(\frac {1}{d ^ {\phi}} + \sqrt {\frac {d}{n}} + \frac {d ^ {2 . 5 - 2 \psi - 2 \phi}}{n}\right) \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} (E.30) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (\log d) ^ {O (1)} \left(\frac {d ^ {2 - 2 \psi - \phi}}{n} + \frac {d ^ {4 - 4 \psi - 2 \phi}}{n}\right) \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} (E.31) \\ \end{array}
$$
$$
\begin{array}{l} \frac {d}{d t} \bar {\Delta} _ {t} ^ {2} \leq \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \gamma_ {2} \bar {\Delta} _ {t} ^ {2} (E.32) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \cdot \frac {d (\log d) ^ {O (1)}}{\sqrt {m}} \bar {\Delta} _ {t} (E.33) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (\log d) ^ {O (1)} \left(\sqrt {\frac {d}{n}} + \frac {d ^ {2 . 5 - 2 \psi - 2 \phi}}{n}\right) \bar {\Delta} _ {t} (E.34) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (\log d) ^ {O (1)} \left(\frac {1}{d ^ {\phi}} + \frac {d ^ {2 - 2 \psi - \phi}}{n} + \frac {d ^ {4 - 4 \psi - 2 \phi}}{n}\right) \bar {\Delta} _ {t} ^ {2}. (E.35) \\ \end{array}
$$
Proof of Lemma E.2. This result follows directly from Lemma 5.4 and the inductive hypothesis. By the inductive hypothesis, we have  $\overline{\Delta}_t \leq d^{-\phi}$ . We apply this to the bound for  $C_t$  in Lemma 5.4 to obtain the desired bounds above. Note that in our bounds for  $\frac{d}{dt} \overline{\Delta}_t^2$ , it is important that we use the upper bound for  $\mathbb{E}[B_t]$  in Lemma 5.4 instead of the upper bound for  $B_t$ .
Using this bound on the growth rate, we can show that the inductive hypothesis is maintained.
Lemma E.3 (IH is Maintained). Suppose we are in the setting of Theorem 3.4. Suppose  $\psi >0$  and  $1 > \phi >\frac{1}{2} +\psi$ . Let  $n = d^{\mu}(\log d)^{C}$  where  $\mu \geq \max \{2 + 2\phi ,4 - 2\phi -4\psi \}$  and  $C$  is a sufficiently large universal constant. Suppose the inductive hypothesis in Assumption 5.3 holds for all  $t\le T$  where  $T < T_{*,\epsilon}$ . Then, when  $m\gtrsim d^{3 + 2\phi}(\log d)^{\Omega (1)}$  and  $m\leq d^{C}$  for a sufficiently large universal constant  $C$ , we have that the inductive hypothesis holds for all  $t\le T + c_{\mathrm{IH}}$  where  $c_{\mathrm{IH}} > 0$  is a constant that only depends on  $d$ ,  $\phi$ ,  $\psi$ ,  $\hat{\sigma}_{2,d}$ ,  $\hat{\sigma}_{4,d}$ .
Proof. Suppose the inductive hypothesis holds for all  $t \leq T$  where  $T \leq T_1$ . Then, by Lemma E.2 we have
$$
\begin{array}{l} \frac {d}{d t} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} \leq \left(4 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | \left(1 + O \left(\frac {1}{\log d}\right)\right)\right) \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} (E.36) \\ + \frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) d (\log d) ^ {O (1)}}{\sqrt {m}} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} (E.37) \\ + O \left(\frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) (\log d) ^ {O (1)}}{d ^ {\phi}} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2}\right) (E.38) \\ \leq \left(4 \hat {\sigma} _ {2, d} ^ {2} \left| D _ {2, t} \right| \left(1 + O \left(\frac {1}{\log d}\right)\right)\right) \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} (E.39) \\ + O \left(\frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) (\log d) ^ {O (1)}}{d ^ {\phi}} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2}\right). (E.40) \\ \end{array}
$$
Here we have used the fact that  $n = d^{\mu}(\log d)^{O(1)}$  where  $\mu \geq \max (4 - 4\psi -2\phi ,2 + 2\phi)$  as well as the fact that  $\phi >\frac{1}{2}$ , as follows:
- In the third term in the bound for  $\frac{d}{dt} \| \hat{u}_t - \bar{u}_t \|_2^2$  from Lemma E.2, only the  $\frac{1}{d^\phi}$  term is important — this term dominates the  $\sqrt{\frac{d}{n}}$  and  $\frac{d^{2.5 - 2\psi - 2\phi}}{n}$  terms, using the fact that  $\mu > 2 + 2\phi$ .
- In the fourth term in the bound for  $\frac{d}{dt} \| \hat{u}_t - \bar{u}_t \|_2^2$ , the  $\frac{d^{2-2\psi-\phi}}{n} \| \hat{u}_t - \bar{u}_t \|_2^2$  term is dominated by the  $\frac{1}{d^\phi} \| \hat{u}_t - \bar{u}_t \|_2$  term — this is because  $\mu \geq 2 + 2\phi$ .
- Since  $\mu \geq 4 - 4\psi - 2\phi$ , if we increase  $n$  by a  $(\log d)^{O(1)}$  factor, then the  $\frac{d^{4 - 4\psi - 2\phi}}{n} \| \hat{u}_t - \bar{u}_t \|_2^2$  term can be absorbed into the first term, as it is dominated by the  $4\hat{\sigma}_{2,d}^2 |D_{2,t}|O\left(\frac{1}{\log d}\right)\| \hat{u}_t - \bar{u}_t \|_2^2$  term. (Recall that by Lemma D.8,  $|D_{2,t}| \gtrsim \gamma_2$  during Phase 1).
In the second inequality above, we have used the fact that  $m \gtrsim d^{3 + 2\phi}(\log d)^{\Omega(1)}$  — this implies that the term involving  $m$  is dominated by the  $\frac{(\log d)^{O(1)}}{d^{\phi}}\|\hat{u}_t - \bar{u}_t\|_2$  term.
Additionally, by Lemma E.2, we have
$$
\begin{array}{l} \frac {d}{d t} \bar {\Delta} _ {t} ^ {2} \leq \left(4 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | \left(1 + O \left(\frac {1}{\log d}\right)\right)\right) \bar {\Delta} _ {t} ^ {2} + \frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) \cdot d (\log d) ^ {O (1)}}{\sqrt {m}} \bar {\Delta} _ {t} (E.41) \\ + O \left(\frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right)}{d ^ {0 . 5 + \phi} (\log d) ^ {O (1)}} \bar {\Delta} _ {t}\right) (E.42) \\ \leq \left(4 \hat {\sigma} _ {2, d} ^ {2} \left| D _ {2, t} \right| \left(1 + O \left(\frac {1}{\log d}\right)\right)\right) \bar {\Delta} _ {t} ^ {2} + O \left(\frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right)}{d ^ {0 . 5 + \phi} (\log d) ^ {O (1)}} \bar {\Delta} _ {t}\right). (E.43) \\ \end{array}
$$
Here, we apply the bound for  $\frac{d}{dt}\overline{\Delta}_t^2$  from Lemma E.2. In the third term in this bound, the  $\sqrt{\frac{d}{n}}$  term dominates the  $\frac{d^{2.5 - 2\psi - 2\phi}}{n}$  term, using the fact that  $\phi >\frac{1}{2}$  and  $\mu \geq 2 + 2\phi >3$ , meaning  $\frac{d^{2.5 - 2\psi - 2\phi}}{n}\leq \frac{d^{1.5}}{n}\leq \frac{1}{\sqrt{n}}$ . In the fourth term, we deal with the  $\frac{d^{4 - 4\psi - 2\phi}}{n}$  term similarly to our calculations for the bound for  $\frac{d}{dt}\| \hat{u}_t - \bar{u}_t\| _2^2$ . We can also deal with the  $\frac{d^{2 - 2\psi - \phi}}{n}$  term similarly, since by our definition of  $\mu$ , this will be at most  $\frac{1}{d^3\phi}\leq \frac{1}{(\log d)^{O(1)}}$ , and will thus be absorbed into the first term involving  $|D_{2,t}|$ . Using  $\mu \geq 2 + 2\phi$ , we then obtain  $\sqrt{\frac{d}{n}}\leq \frac{1}{d^{0.5 + \phi}}$  as the coefficient of the  $\overline{\Delta}_t$  term (i.e. the first-order term involving  $\overline{\Delta}_t$ ). Note that here we need to have  $m\gtrsim d^{3 + 2\phi}(\log d)^{\Omega (1)}$  in order to absorb the term involving  $m$  into the  $\frac{1}{d^{0.5 + \phi}}\overline{\Delta}_t$  term.
Now using Lemma I.12 we have
$$
\overline {{\Delta}} _ {t} ^ {2} \lesssim \frac {1}{(\log d) ^ {O (1)}} \Big (\frac {(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}) O (\frac {1}{d ^ {0 . 5 + \phi}})}{4 \hat {\sigma} _ {2 , d} ^ {2} \min _ {t} | D _ {2 , t} |} \Big) ^ {2} \exp \Big (\int_ {0} ^ {T} 4 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | \Big (1 + O \Big (\frac {1}{\log d} \Big) \Big) d t \Big) (E. 4 4)
$$
$$
\Delta_ {\max, T} ^ {2} \lesssim (\log d) ^ {O (1)} \Big (\frac {(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}) O (\frac {1}{d ^ {\phi}})}{4 \hat {\sigma} _ {2 , d} ^ {2} \min _ {t} | D _ {2 , t} |} \Big) ^ {2} \exp \Big (\int_ {t} ^ {T} 4 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | \Big (1 + O \Big (\frac {1}{\log d} \Big) \Big) d t \Big) \tag {E.45}
$$
By Lemma D.5 we have
$$
\exp \left(\int_ {t = 0} ^ {T} 4 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | d t\right) \lesssim d \tag {E.46}
$$
and by Lemma D.8 we know that  $|D_{2,t}| \gtrsim \gamma_2$  during Phase 1. Thus,
$$
\Delta_ {\max , T} ^ {2} \leq O \left(\frac {(\log d) ^ {O (1)}}{\gamma_ {2} ^ {2} d ^ {2 \phi - 1}}\right). \tag {E.47}
$$
Since  $2\phi - 1 > 2\psi$  and  $d$  is chosen to be sufficiently large after  $\gamma_{2}$  is chosen (Assumption 3.2), we have that  $\Delta_{\max,T} < \sqrt{\frac{1}{d^{\psi + \phi - 1/2}}} = \frac{1}{d^{\frac{1}{2}\phi + \frac{1}{2}\psi - \frac{1}{4}}} < \frac{1}{d^{\psi}}$ . Note that  $\Delta_{\max,t}$  is continuous in  $t$  and we have
$$
\begin{array}{l} \frac {d}{d t} \Delta_ {\max , t} \leq \max  _ {\chi} \frac {d}{d t} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} (E.48) \\ = \max  _ {x} \frac {1}{\| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2}} \frac {d}{d t} \| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2} ^ {2} (E.49) \\ \leq \max  _ {x} \frac {1}{\| \hat {u} _ {t} - \bar {u} _ {t} \| _ {2}} | \langle \hat {u} _ {t} - \bar {u} _ {t}, \operatorname {g r a d} _ {\hat {u} _ {t}} \widehat {L} (\hat {u} _ {t}) - \operatorname {g r a d} _ {\bar {u} _ {t}} L (\bar {u} _ {t}) \rangle | (E.50) \\ \leq \max  _ {\chi} \| \operatorname {g r a d} _ {\hat {u} _ {t}} \widehat {L} (\hat {u} _ {t}) - \operatorname {g r a d} _ {\bar {u} _ {t}} L (\bar {u} _ {t}) \| _ {2} \quad \text {(B y C a u c h y - S c h w a r z I n e q u a l i t y)} \\ \leq \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {4}. \quad \text {(B y} \\ \end{array}
$$
Thus, for  $\mu_{1} = \left(\frac{1}{d^{\psi}} -\frac{1}{d^{\frac{1}{2}\phi + \frac{1}{2}\psi - \frac{1}{4}}}\right)\cdot \frac{1}{(\hat{\sigma}_{2,d}^{2} + \hat{\sigma}_{4,d}^{2})d^{4}}$  we still have the inductive hypothesis for  $\Delta_{\mathrm{max}}$  until time  $T + \mu_1$ , that is,  $\Delta_{\mathrm{max},T + \mu_1}\leq \frac{1}{d^\psi}$ . Similarly, our calculation above using Lemma I.12 gives us
$$
\bar {\Delta} _ {T} ^ {2} \leq O \left(\frac {1}{\gamma_ {2} ^ {2} d ^ {2 \phi} (\log d) ^ {O (1)}}\right) \tag {E.51}
$$
Thus, we have  $\overline{\Delta}_T < \frac{1}{d^\phi(\log d)^{O(1)}} < \frac{1}{d^\phi}$  (where in the middle expression, we choose a smaller power of  $(\log d)^{O(1)}$  so that the inequality holds). Observe that  $\overline{\Delta}_t$  is continuous in  $t$ , and