SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Furthermore, since  $|D_{4,t}| \geq \xi |D_{2,t}|$ , this means that  $|D_{4,t}| \gtrsim \xi \epsilon$ , since otherwise,  $|D_{2,t}| \lesssim \epsilon$  and  $|D_{4,t}| \lesssim \xi \epsilon$ , and the loss would be less than  $(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2$ . Thus, if time  $T_{\mathrm{esc}}$  is spent in this case, then by Gronwall's inequality (Fact I.13), this case contributes a factor of at least
$$
\exp \left(T _ {\text {e s c}} \cdot \hat {\sigma} _ {4, d} ^ {2} D _ {4, t}\right) \geq \exp \left(T _ {\text {e s c}} \cdot \hat {\sigma} _ {4, d} ^ {2} \xi \epsilon\right) \tag {D.280}
$$
to  $1 - w_{\mathbb{R}}$
Summary of 4 Cases. In summary, by Eq. (D.274), Eq. (D.275), Eq. (D.276) and Eq. (D.280),
$$
\begin{array}{l} 1 - w _ {\mathrm {R}} \geq \left(1 - \frac {1}{\log d}\right) \cdot \exp \left(- \frac {1}{\xi^ {6} \kappa^ {1 2}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right)\right) \exp \left(- \frac {C}{\xi^ {4} \kappa^ {4}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right)\right) (D.281) \\ \exp \left(- \frac {C}{\xi^ {4} \kappa^ {6}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right)\right) \exp \left(T _ {\mathrm {e s c}} \cdot \hat {\sigma} _ {4, d} ^ {2} \xi \epsilon\right) (D.282) \\ \end{array}
$$
where  $T_{\mathrm{esc}}$  is the amount of time spent so far in the case where  $|D_{4,t}| \geq \xi |D_{2,t}|$ ,  $r(t) \geq \frac{1}{2}$  and  $w_{\mathbb{R}} \geq 1 - \xi$ . This is in fact a lower bound on  $1 - w_{\mathbb{R}}$  when  $L(\rho_t) \geq (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2$ , since at all other times when  $L(\rho_t) \geq (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2$ , we have that  $1 - w_{\mathbb{R}}$  is increasing (i.e.  $w_{\mathbb{R}}$  is moving away from 1). In particular, if
$$
T _ {\text {e s c}} \gtrsim \frac {1}{\hat {\sigma} _ {4 , d} ^ {2} \xi \epsilon} \left(\frac {1}{\xi^ {6} \kappa^ {1 2}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) + \frac {C}{\xi^ {4} \kappa^ {4}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) + \frac {C}{\xi^ {4} \kappa^ {6}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right)\right) \gtrsim \frac {1}{\hat {\sigma} _ {4 , d} ^ {2} \epsilon \xi^ {7} \kappa^ {1 2}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) \tag {D.283}
$$
then  $1 - w_{\mathrm{R}}$  will be at least  $\xi$ .
Finally, the following lemma deals with the last case, where the positive root  $r$  of  $P_{t}$  is at most  $\frac{1}{2}$ , and additionally,  $w_{t}(\iota_{\mathbb{R}}) \leq 1 - \xi$ .
Lemma D.32. Suppose we are in the setting of Theorem 3.3. Assume  $D_{4,T_2} = 0$  and  $D_{2,T_2} < 0$ , i.e. Phase 3 Case 2 holds. Then, the amount of time  $t \geq T_2$  that elapses while the following conditions simultaneously hold:
1.  $|D_{4,t}| \geq \xi |D_{2,t}|$
2.  $r\leq \frac{1}{2}$
3.  $w_{t}(\iota_{\mathbb{R}})\leq 1 - \xi$
4.  $L(\rho_t)\geq (\hat{\sigma}_{2,d}^2 +\hat{\sigma}_{4,d}^2)\epsilon^2$
is at most  $\frac{1}{\hat{\sigma}_{4,d}^2\gamma_2^{11 / 2}\xi^8\kappa^4}\log \left(\frac{\gamma_2}{\epsilon}\right)$
Proof of Lemma D.32. For convenience and ease of presentation, in this proof we define  $\tau \coloneqq \sqrt{\gamma_2}$  and  $\beta \geq 1.1$  such that  $\gamma_4 = \beta \tau^4$ . (Note that by Assumption 3.2,  $\beta$  is at most a universal constant.) We first show that there is always a large fraction of particles which are far away from the positive root  $r$  of  $P_t(w)$ . We consider two cases: (i)  $r \leq \tau^{3/2}$  and (ii)  $r \geq \tau^{3/2}$ .
Case 1:  $r \leq \tau^{3/2}$ . Assume for the sake of contradiction that
$$
\mathbb {P} _ {w \sim \rho_ {t}} \left(\left| w \right| \geq r + \tau^ {3 / 2}\right) \leq \tau^ {4} \tag {D.284}
$$
$$
\begin{array}{l} \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {4} \right] \leq \mathbb {P} _ {w \sim \rho_ {t}} \left(\left| w \right| \geq r + \tau^ {3 / 2}\right) + \mathbb {P} _ {w \sim \rho_ {t}} \left(\left| w \right| \leq r + \tau^ {3 / 2}\right) \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {4} \mid | w | \leq r + \tau^ {3 / 2} \right] (D.285) \\ = \tau^ {4} + (r + \tau^ {3 / 2}) ^ {4} (D.286) \\ \leq \tau^ {4} + (2 \tau^ {3 / 2}) ^ {4} \quad (\text {B y a s s u m p t i o n t h a t} r \leq \tau^ {3 / 2}) \\ = \tau^ {4} + 1 6 \tau^ {6} (D.287) \\ \end{array}
$$
$$
\begin{array}{l} D _ {4, t} = \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ P _ {4, d} (w) \right] - \beta \tau^ {4} (D.288) \\ \leq \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {4} \right] + O \left(\frac {1}{d}\right) - \beta \tau^ {4} (ByEq.(C.4)) \\ \leq \tau^ {4} + 1 6 \tau^ {6} + O \left(\frac {1}{d}\right) - \beta \tau^ {4} (D.289) \\ = 1 6 \tau^ {6} + O \left(\frac {1}{d}\right) - (\beta - 1) \tau^ {4} (D.290) \\ \leq 1 7 \tau^ {6} - (\beta - 1) \tau^ {4} \quad \text {(B . c .} d \text {i s s u f f i e n t l y l a r g e b y A s s u m p t i o n 3 . 2)} \\ <   0 \quad (\mathrm {B . c .} (\beta - 1) > 1 7 \tau^ {2} \text {b y A s s u m p t i o n 3 . 2}) \\ \end{array}
$$
which contradicts the assumption that we are in Phase 3, Case 2 since  $D_{4,t} > 0$  by Lemma D.28. Thus, if  $r \leq \tau^{3/2}$ , then
$$
\mathbb {P} _ {w \sim \rho_ {t}} \left(\left| w \right| \geq r + \tau^ {3 / 2}\right) \geq \tau^ {4} \tag {D.291}
$$
Case 2:  $r \geq \tau^{3/2}$ . Suppose that
$$
\mathbb {P} _ {w \sim \rho_ {t}} \left(r - \tau^ {3 / 2} \leq | w | \leq r + \tau^ {3 / 2}\right) \geq 1 - \tau^ {5} \tag {D.292}
$$
$$
\begin{array}{l} \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {2} \right] \geq \mathbb {P} _ {w \sim \rho_ {t}} \left(r - \tau^ {3 / 2} \leq | w | \leq r + \tau^ {3 / 2}\right) \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {2} \mid | w | \in \left[ r - \tau^ {3 / 2}, r + \tau^ {3 / 2} \right] \right] (D.293) \\ \geq \left(1 - \tau^ {5}\right) \cdot \left(r - \tau^ {3 / 2}\right) ^ {2} (D.294) \\ \end{array}
$$
Since  $D_{2,t} < 0$  by Lemma D.28, this implies that  $\mathbb{E}_{w\sim \rho_t}[w^2 ]\leq \tau^2 +O(1 / d)$  by Proposition D.38. Thus,
$$
(1 - \tau^ {5}) (r - \tau^ {3 / 2}) ^ {2} \leq \tau^ {2} + O (1 / d) \tag {D.295}
$$
and taking square roots gives
$$
(1 - \tau^ {5}) ^ {1 / 2} (r - \tau^ {3 / 2}) \leq \tau + O (1 / \sqrt {d}) \tag {D.296}
$$
$$
r \leq \tau^ {3 / 2} + \frac {1}{(1 - \tau^ {5}) ^ {1 / 2}} \cdot \left(\tau + O \left(\frac {1}{\sqrt {d}}\right)\right) \tag {D.297}
$$
$$
\begin{array}{l} \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {4} \right] \leq \mathbb {P} _ {w \sim \rho_ {t}} \left(\left| w \right| \in \left[ r - \tau^ {3 / 2}, r + \tau^ {3 / 2} \right]\right) \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {4} \mid | w | \in \left[ r - \tau^ {3 / 2}, r + \tau^ {3 / 2} \right] \right] (D.298) \\ + \mathbb {P} _ {w \sim \rho_ {t}} \left(\left| w \right| \notin \left[ r - \tau^ {3 / 2}, r + \tau^ {3 / 2} \right]\right) (D.299) \\ \leq (r + \tau^ {3 / 2}) ^ {4} + \tau^ {5} \quad (\text {B y}) \\ \end{array}
$$
and since  $D_{4,t} \geq 0$  by Lemma D.28,  $\mathbb{E}_{w \sim \rho_t}[w^4] \geq \beta \tau^4 - O(1 / d)$  by Eq. (C.4) and because  $w \sim \rho_t$  are bounded by 1 in absolute value. Thus, by the above equation,
$$
\left(r + \tau^ {3 / 2}\right) ^ {4} + \tau^ {5} \geq \beta \tau^ {4} - O \left(\frac {1}{d}\right) \tag {D.300}
$$
$$
\left(r + \tau^ {3 / 2}\right) ^ {4} + \tau^ {5} + O \left(\frac {1}{d}\right) \geq \beta \tau^ {4} \tag {D.301}
$$
and taking fourth roots gives
$$
r + \tau^ {3 / 2} + \tau^ {5 / 4} + O \left(\frac {1}{d ^ {1 / 4}}\right) \geq \beta^ {1 / 4} \tau \tag {D.302}
$$
(by the inequality  $\sqrt{x + y} \leq \sqrt{x} + \sqrt{y}$ ). Therefore,
$$
r \geq \beta^ {1 / 4} \tau - \tau^ {3 / 2} - \tau^ {5 / 4} - O \left(\frac {1}{d ^ {1 / 4}}\right) \tag {D.303}
$$
Combining Eq. (D.297) and Eq. (D.303) gives
$$
\beta^ {1 / 4} \tau - \tau^ {3 / 2} - \tau^ {5 / 4} - O \left(\frac {1}{d ^ {1 / 4}}\right) \leq \tau^ {3 / 2} + \frac {1}{(1 - \tau^ {5}) ^ {1 / 2}} \cdot \left(\tau + O \left(\frac {1}{\sqrt {d}}\right)\right) \tag {D.304}
$$
Rearranging this equation gives
$$
\left(\beta^ {1 / 4} - \frac {1}{\sqrt {1 - \tau^ {5}}}\right) \tau \leq 2 \tau^ {3 / 2} + \tau^ {5 / 4} + O \left(\frac {1}{d ^ {1 / 4}}\right) + \frac {1}{\sqrt {1 - \tau^ {5}}} \cdot O \left(\frac {1}{\sqrt {d}}\right) \tag {D.305}
$$
$$
\leq 3 \tau^ {5 / 4} + O \left(\frac {1}{d ^ {1 / 4}}\right) + \frac {1}{\sqrt {1 - \tau^ {5}}} \cdot O \left(\frac {1}{\sqrt {d}}\right)
$$
$$
(\mathrm {B . c .} \tau \leq 1, \text {w e h a v e} \tau^ {3 / 2} \leq \tau^ {5 / 4})
$$
$$
\leq 4 \tau^ {5 / 4} + \frac {1}{\sqrt {1 - \tau^ {5}}} \cdot O \left(\frac {1}{\sqrt {d}}\right)
$$
(B.c.  $d$  is sufficiently large by Assumption 3.2)
$$
\leq 4 \tau^ {5 / 4} + O \left(\frac {1}{\sqrt {d}}\right) \quad (\mathrm {B . c .} \tau \leq \frac {1}{2} \text {b y A s s u m p t i o n 3 . 2})
$$
$$
\leq 5 \tau^ {5 / 4} \quad (\text {B . c .} d \text {i s s u f f i e n t l y l a r g e b y A s s u m p t i o n 3 . 2})
$$
Further rearranging gives
$$
\beta^ {1 / 4} - \frac {1}{\sqrt {1 - \tau^ {5}}} \leq 5 \tau^ {1 / 4} \tag {D.306}
$$
This contradicts Assumption 3.2 — since  $\beta \geq 1.1$ , the left-hand side is larger than an absolute constant, while the right-hand side can be made sufficiently small (since  $\gamma_{2} \leq c_{2}$  and  $c_{2}$  is chosen to be sufficiently small). Thus, in the case that  $r \geq \tau^{3/2}$ , we obtain
$$
\mathbb {P} _ {w \sim \rho_ {t}} \left(\left| w \right| \notin \left[ r - \tau^ {3 / 2}, r + \tau^ {3 / 2} \right]\right) \geq \tau^ {5} \tag {D.307}
$$
In summary, whether  $r \geq \tau^{3/2}$  or  $r \leq \tau^{3/2}$ , we can conclude that
$$
\mathbb {P} _ {w \sim \rho_ {t}} (| w | \notin [ r - \tau^ {3 / 2}, r + \tau^ {3 / 2} ]) \geq \tau^ {5} \tag {D.308}
$$
Further assume that  $w_{t}(\iota_{\mathrm{R}})\leq 1 - \xi$  , and note that by Lemma D.28, for all  $\iota >\iota_{\mathrm{L}}$  , we have  $w_{t}(\iota)\geq$ $\xi^2\kappa^2$  . Thus, by Proposition D.6, with probability  $1 - O(\kappa)$  under  $\rho_t$  , we have  $\xi^2\kappa^2\leq |w|\leq 1 - \xi$  By a union bound,
$$
\mathbb {P} _ {w \sim \rho_ {t}} \left(\left| \left| w \right| - r \right| \geq \tau^ {3 / 2} \text {a n d} \xi^ {2} \kappa^ {2} \leq | w | \leq 1 - \xi\right) \geq \tau^ {5} - O (\kappa) \geq \frac {\tau^ {5}}{2} \tag {D.309}
$$
Suppose  $w$  satisfies  $||w| - r| \geq \tau^{3/2}$  and  $\xi^2\kappa^2 \leq |w| \leq 1 - \xi$ , i.e. the conditions mentioned in Eq. (D.309), and write
$$
P _ {t} (w) = 2 \hat {\sigma} _ {2, d} ^ {2} D _ {2, t} w + 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} w ^ {3} = 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} w (w - r) (w + r) \tag {D.310}
$$
$$
\begin{array}{l} \left| v _ {t} (w) \right| \gtrsim \left| 1 - w ^ {2} \right| \cdot \left| P _ {t} (w) + Q _ {t} (w) \right| (By Lemma 4.4) \\ \gtrsim | 1 + w | \cdot | 1 - w | \cdot \left| P _ {t} (w) + Q _ {t} (w) \right| (D.311) \\ \gtrsim \xi | P _ {t} (w) + Q _ {t} (w) | \quad (\mathrm {B . c .} | w | \leq 1 - \xi) \\ \gtrsim \xi \cdot \left| 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} w (w - r) (w + r) - O \left(\frac {\hat {\sigma} _ {2 , d} ^ {2} | D _ {2 , t} | + \hat {\sigma} _ {4 , d} ^ {2} | D _ {4 , t} |}{d}\right) | w | \right| (By Lemma 4.4) \\ \begin{array}{l} \gtrsim \xi \Big (4 \hat {\sigma} _ {4, d} ^ {2} | D _ {4, t} | \xi^ {2} \kappa^ {2} \cdot \tau^ {3 / 2} \cdot \tau^ {3 / 2} - O \Big (\frac {\hat {\sigma} _ {2 , d} ^ {2} | D _ {2 , t} | + \hat {\sigma} _ {4 , d} ^ {2} | D _ {4 , t} |}{d} \Big) \Big) \\ \quad \text {(B y t r i a n g l e i n e q u a l i t y a n d b . c .} | w | \geq \xi^ {2} \kappa^ {2} \text {a n d} | | w | - r | \geq \tau^ {3 / 2}) \end{array} \\ \begin{array}{l} \gtrsim \xi \left(4 \hat {\sigma} _ {4, d} ^ {2} | D _ {4, t} | \tau^ {3} \xi^ {2} \kappa^ {2} - O \left(\frac {\hat {\sigma} _ {4 , d} ^ {2} | D _ {4 , t} |}{\xi d}\right)\right) \\ \quad (\text {B . c .} | D _ {2, t} | \leq | D _ {4, t} | / \xi \text {a n d} \hat {\sigma} _ {2, d} ^ {2} \lesssim \hat {\sigma} _ {4, d} ^ {2} \text {b y A s s u m p t i o n 3 . 2}) \end{array} \\ \begin{array}{l} \gtrsim \xi \cdot 4 \hat {\sigma} _ {4, d} ^ {2} | D _ {4, t} | \tau^ {3} \xi^ {2} \kappa^ {2} \\ \hskip 1 4. 2 2 6 3 7 8 p t (\text {B . c .} 1 / (\xi d) = \frac {\log \log d}{d} \text {a n d} d \text {i s s u f f i c i e n t l y l a r g e (A s s u m p t i o n 3 . 2)}) \end{array} \\ \stackrel {>} {_ \sim} \hat {\sigma} _ {4, d} ^ {2} \tau^ {3} \xi^ {3} \kappa^ {2} | D _ {4, t} | (D.312) \\ \end{array}
$$
and the average velocity is at least
$$
\begin{array}{l} \underset {w \sim \rho_ {t}} {\mathbb {E}} | v _ {t} (w) | ^ {2} \gtrsim \mathbb {P} _ {w \sim \rho_ {t}} \left(\left| \left| w \right| - r \right| \geq \tau^ {3 / 2} \text {a n d} \xi^ {2} \kappa^ {2} \leq | w | \leq 1 - \xi\right) \cdot \hat {\sigma} _ {4, d} ^ {4} \tau^ {6} \xi^ {6} \kappa^ {4} | D _ {4, t} | ^ {2} (ByEq.(D.312)) \\ \gtrsim \tau^ {5} \cdot \hat {\sigma} _ {4, d} ^ {4} \tau^ {6} \xi^ {6} \kappa^ {4} | D _ {4, t} | ^ {2} \quad (\text {B y E q . (D . 3 0 9)}) \\ \gtrsim \hat {\sigma} _ {4, d} ^ {4} \tau^ {1 1} \xi^ {6} \kappa^ {4} | D _ {4, t} | ^ {2} (D.313) \\ \begin{array}{c} \gtrsim \hat {\sigma} _ {4, d} ^ {2} \tau^ {1 1} \xi^ {6} \kappa^ {4} (\xi^ {2} L (\rho_ {t})) \\ \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad (B. c. | D _ {4, t} | \geq \xi | D _ {2, t} | a n d \hat {\sigma} _ {4, d} ^ {2} \gtrsim \hat {\sigma} _ {2, d} ^ {2} b y A s s u m p t i o n 3. 2) \end{array} \\ \gtrsim \hat {\sigma} _ {4, d} ^ {2} \tau^ {1 1} \xi^ {8} \kappa^ {4} L \left(\rho_ {t}\right) (D.314) \\ \end{array}
$$
$$
\frac {d L \left(\rho_ {t}\right)}{d t} \lesssim - \hat {\sigma} _ {4, d} ^ {2} \tau^ {1 1} \xi^ {8} \kappa^ {4} L \left(\rho_ {t}\right) \tag {D.315}
$$
Since the initial loss is  $L(\rho_0) \lesssim (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\gamma_2^2$  by Lemma D.8, we can calculate using Gronwall's inequality (Fact I.13) that the time spent when the conditions in the statement of this lemma simultaneously hold is at most  $\frac{1}{\hat{\sigma}_{4,d}^2\tau^{11}\xi^8\kappa^4}\log \left(\frac{\gamma_2}{\epsilon}\right)$ , as desired.
To conclude, we show Lemma D.33 which gives a bound on the running time during Phase 3, Case 2.
Lemma D.33 (Phase 3, Case 2 Summary). Suppose we are in the setting of Theorem 3.3. Assume
$D_{4,T_2} = 0$  and  $D_{2,T_2} < 0$ , i.e. Phase 3 Case 2 holds. Then, at most  $O\left(\frac{1}{\tilde{\sigma}_{2,d}^2\epsilon\gamma_2^{11/2}\xi^{8}\kappa^{12}}\log \left(\frac{\gamma_2}{\epsilon}\right)\right)$
time  $t \geq T_2$  elapses while  $L(\rho_t) \gtrsim (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2$ .
Proof of Lemma D.33. By combining Lemma D.29, Lemma D.30, Lemma D.31, and Lemma D.32, we find that the overall running time during Phase 3 Case 2 is at most
$$
O \left(\frac {1}{\hat {\sigma} _ {2 , d} ^ {2} \xi^ {4} \kappa^ {6}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) + \frac {1}{\hat {\sigma} _ {2 , d} ^ {2} \xi^ {4} \kappa^ {4}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) + \frac {1}{\hat {\sigma} _ {4 , d} ^ {2} \epsilon \xi^ {7} \kappa^ {1 2}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) + \frac {1}{\hat {\sigma} _ {4 , d} ^ {2} \gamma_ {2} ^ {1 1 / 2} \xi^ {8} \kappa^ {4}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right)\right) \tag {D.316}
$$
and this can be upper bounded by  $O\left(\frac{1}{\hat{\sigma}_{2,d}^{2}\epsilon\gamma_{2}^{11/2}\xi^{8}\kappa^{12}} \log \left(\frac{\gamma_{2}}{\epsilon}\right)\right)$ , where we have used the fact that  $\hat{\sigma}_{2,d}^{2} \asymp \hat{\sigma}_{4,d}^{2}$  (see Assumption 3.2).
D.7 Summary of Phase 3
The following lemma provides a bound on the running time after  $T_{2}$  that is required for  $L(\rho_t) \lesssim (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2$ .
Lemma D.34 (Phase 3). Suppose we are in the setting of Theorem 3.3. Then,  $T_{*,\epsilon} - T_2 \lesssim \frac{(\log\log d)^{20}}{\hat{\sigma}_{2,d}^2\epsilon\gamma_2^8}\log \left(\frac{\gamma_2}{\epsilon}\right)$ .
Proof of Lemma D.34. By combining Lemma D.22 and Lemma D.33, we find that the overall running time during Phase 3 is at most
$$
\left. \max  \left(O \left(\frac {1}{\hat {\sigma} _ {4 , d} ^ {2} \gamma_ {2} ^ {8} \xi^ {4} \kappa^ {6}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right)\right), O \left(\frac {1}{\hat {\sigma} _ {2 , d} ^ {2} \epsilon \gamma_ {2} ^ {1 1 / 2} \xi^ {8} \kappa^ {1 2}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right)\right)\right) \right. \tag {D.317}
$$
and this is at most  $O\left(\frac{1}{\hat{\sigma}_{2,d}^{2}\epsilon^{\gamma_{2}^{8}\zeta^{8}\kappa^{12}}}\log \left(\frac{\gamma_{2}}{\epsilon}\right)\right)$ , where we have used the fact that  $\hat{\sigma}_{2,d}^{2} \asymp \hat{\sigma}_{4,d}^{2}$  (see Assumption 3.2)
D.8 Proof of Main Theorem for Population Case
Finally, we prove Theorem 3.3:
Proof of Theorem 3.3. We combine Lemma D.5, Lemma D.10 and Lemma D.34 to find that the total runtime is at most
$$
T _ {*, \epsilon} \lesssim \frac {1}{\hat {\sigma} _ {2 , d} ^ {2} \gamma_ {2}} \log d + \frac {\left(\log \log d\right) ^ {1 8}}{\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right)} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) + \frac {\left(\log \log d\right) ^ {2 0}}{\hat {\sigma} _ {2 , d} ^ {2} \epsilon \gamma_ {2} ^ {8}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) \tag {D.318}
$$
and we can further simplify the above bound to obtain:
$$
T _ {*, \epsilon} \lesssim \frac {1}{\hat {\sigma} _ {2 , d} ^ {2} \gamma_ {2}} \log d + \frac {\left(\log \log d\right) ^ {2 0}}{\hat {\sigma} _ {2 , d} ^ {2} \epsilon \gamma_ {2} ^ {8}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) \tag {D.319}
$$
where we have used the fact that  $\hat{\sigma}_{2,d}^2\asymp \hat{\sigma}_{4,d}^2$  by Assumption 3.2.
D.9 Additional Lemmas
Lemma D.35. Suppose Assumption 3.2 holds. There exists some  $r$ , with  $r \gtrsim \frac{1}{\log \log d}$  and  $r \leq 1$  such that if  $\rho$  is the singleton distribution under which  $r$  has probability 1, then the velocity at  $r$  is 0.
Proof. Let  $\rho$  be such that  $r$  has probability 1 under  $\rho$ , for some  $r \in (0,1)$ . Applying the definitions of  $D_{2}$  and  $D_{4}$  to  $\rho$ , we have  $D_{2} = P_{2,d}(r) - \gamma_{2}$  and  $D_{4} = P_{4,d}(r) - \gamma_{4}$ . By Lemma D.3, the velocity of  $r$  is
$$
v (r) = - \left(1 - r ^ {2}\right) \left(\hat {\sigma} _ {2, d} ^ {2} D _ {2} P _ {2, d} ^ {\prime} (r) + \hat {\sigma} _ {4, d} ^ {2} D _ {4} P _ {4, d} ^ {\prime} (r)\right) \tag {D.320}
$$
The second factor is equal to
$$
\hat {\sigma} _ {2, d} ^ {2} \left(P _ {2, d} (r) - \gamma_ {2}\right) P _ {2, d} ^ {\prime} (r) + \hat {\sigma} _ {4, d} ^ {2} \left(P _ {4, d} (r) - \gamma_ {4}\right) P _ {4, d} ^ {\prime} (r) \tag {D.321}
$$
and we denote this polynomial of  $r$  by  $F(r)$ . Note that  $P_{2,d}(1) = P_{4,d}(1) = 1$  (see Equation (2.20) of Atkinson and Han [12]). Thus,  $F(1) > 0$  by Assumption 3.2, since  $P_{2,d}'(1) > 0$  and  $P_{4,d}'(1) > 0$  by Eq. (C.4). On the other hand, suppose  $r \asymp \frac{1}{\log \log d}$ . Then,
$$
\begin{array}{l} \left| P _ {2, d} (r) \right| = \left| \frac {d}{d - 1} r ^ {2} - \frac {1}{d - 1} \right| (ByEq.(C.4)) \\ \lesssim \frac {1}{(\log \log d) ^ {2}} (D.322) \\ \end{array}
$$
Similarly,  $|P_{4,d}(r)| \lesssim \frac{1}{(\log\log d)^4}$ . Thus,  $P_{2,d}(r) - \gamma_2 < 0$  and  $P_{4,d}(r) - \gamma_4 < 0$  since  $d$  can be chosen to be sufficiently large according to Assumption 3.2. However, by Eq. (C.4), one can show that  $P_{2,d}'(r) \gtrsim \frac{1}{\log\log d}$  and  $P_{4,d}'(r) \gtrsim \frac{1}{(\log\log d)^3}$  as long as  $d$  is sufficiently large. Thus,  $F(r) < 0$  for  $r \asymp \frac{1}{\log\log d}$ . In summary,  $F$  has a root between  $\frac{C}{\log\log d}$  and 1 for some universal constant  $C$ , and for this value of  $r$ , the velocity of  $r$  is 0.
Lemma D.36. In the setting of Theorem 3.3,  $T_{*,\epsilon} - T_1 \lesssim \frac{(\log \log d)^{20}}{\hat{\sigma}_{2,d}^2 \epsilon \gamma_2^8} \log \left( \frac{\gamma_2}{\epsilon} \right)$ . Thus,  $(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2) \gamma_2 (T_{*,\epsilon} - T_1) \lesssim \frac{(\log \log d)^{20}}{\epsilon \gamma_2^7} \log \left( \frac{\gamma_2}{\epsilon} \right)$ .
Proof. This follows from Lemma D.10 and Lemma D.34.  $\square$
D.10 Helper Lemmas
In this subsection, we give some convenient facts that we use in the following subsections.
Proposition D.37. If  $0 \leq \iota_1 < \iota_2$ , then for all  $t \geq 0$ ,  $w_t(\iota_1) < w_t(\iota_2)$ .
Proof of Proposition D.37. Assume this is not the case, and let  $t$  be the infimum of all times  $s$  such that  $w_{s}(\iota_{1}) \geq w_{s}(\iota_{2})$ . Then,  $w_{t}(\iota_{1}) = w_{t}(\iota_{2})$ , meaning that for all  $t' \geq t$ ,  $w_{t'}(\iota_{1}) = w_{t'}(\iota_{2})$ . For convenience, let  $w_{0} = w_{t}(\iota_{1}) = w_{t}(\iota_{2})$ . Note that  $t > 0$  since  $\iota_{1} \neq \iota_{2}$ . However, this implies that for  $t' \in (0, t)$ ,  $w_{t'}(\iota_{1}) < w_{t'}(\iota_{2})$ , meaning that for any  $\delta > 0$ ,  $w_{s}(\iota_{1})$  and  $w_{s}(\iota_{2})$  are non-identical solutions to the initial value problem on the time interval  $(t - \delta, t + \delta)$  given by the conditions  $f(t) = w_{0}$  and  $\frac{df}{dt} = v_{t}(f(t))$ . This contradicts the Picard-Lindelof theorem, and thus there cannot exist a time  $t$  such that  $w_{t}(\iota_{1}) \geq w_{t}(\iota_{2})$ .
Proposition D.38. Suppose we are in the setting of Theorem 3.3, and let  $t \geq 0$  such that  $D_{2,t} < 0$ . Then,  $\mathbb{E}_{w \sim \rho_t}[w^2] \leq \gamma_2 + O\left(\frac{1}{d}\right)$ .
Proof of Proposition D.37. Suppose  $D_{2,t}\leq 0$ . Then,  $\mathbb{E}_{w\sim \rho_t}[P_{2,d}(w)]\leq \gamma_2$ , which implies that
$$
\frac {d}{d - 1} \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {2} \right] - \frac {1}{d - 1} \leq \gamma_ {2} \tag {D.323}
$$
$$
\underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {2} \right] \leq \frac {d - 1}{d} \gamma_ {2} + \frac {1}{d} \leq \gamma_ {2} + O \left(\frac {1}{d}\right) \tag {D.324}
$$
Lemma D.39 (Decrease in Loss and Average Velocity). Suppose we are in the setting of Theorem 3.3. Then,  $\frac{dL}{dt} = -\mathbb{E}_{u\sim \rho_t}[\| \mathrm{grad}_uL(\rho_t)\| _2^2 ]$
We note that a similar result was shown in Lemma E.11 of Wei et al. [75], but for gradient flow rather than projected gradient flow.
Proof of Lemma D.39. First, we can calculate
$$
\frac {d}{d t} L \left(\rho_ {t}\right) = \frac {1}{2} \frac {d}{d t} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho_ {t}} (x) - y (x)\right) ^ {2} \right] = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho_ {t}} (x) - y (x)\right) \frac {d}{d t} f _ {\rho_ {t}} (x) \right]. \tag {D.325}
$$
Furthermore, we can expand  $\frac{d}{dt} f_{\rho_t}(x)$  as
$$
\frac {d}{d t} \underset {u \sim \rho_ {t}} {\mathbb {E}} [ \sigma (u \cdot x) ] = - \underset {u \sim \rho_ {t}} {\mathbb {E}} \left[ \sigma^ {\prime} (u \cdot x) x \cdot \operatorname {g r a d} _ {u} L (\rho_ {t}) \right]. \tag {D.326}
$$