SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {g r a d} _ {w} L (\rho) = - \left(1 - w ^ {2}\right) \cdot \left(\hat {\sigma} _ {2, d} ^ {2} D _ {2} (t) P _ {2, d} ^ {\prime} (w) + \hat {\sigma} _ {4, d} ^ {2} D _ {4} (t) P _ {4, d} ^ {\prime} (w)\right) \tag {D.40}
$$
where we use  $\mathrm{grad}_wL(\rho)$  to denote the first coordinate of  $\mathrm{grad}_uL(\rho)$ . For convenience, during the rest of the proof of this lemma, let  $\eta_d^{(1)} = \frac{d}{d - 1}$ ,  $\eta_d^{(2)} = \frac{d^2 + 6d + 8}{d^2 - 1}$  and  $\eta_d^{(3)} = \frac{6d + 12}{d^2 - 1}$ . Then, by Eq. (C.4),
$$
\begin{array}{l} \operatorname {g r a d} _ {w} L (\rho) = - \left(1 - w ^ {2}\right) \cdot \left(\hat {\sigma} _ {2, d} ^ {2} D _ {2} (t) P _ {2, d} ^ {\prime} (w) + \hat {\sigma} _ {4, d} ^ {2} D _ {4} (t) P _ {4, d} ^ {\prime} (w)\right) (D.41) \\ = - (1 - w ^ {2}) \cdot \left(\hat {\sigma} _ {2, d} ^ {2} D _ {2} (t) \cdot 2 \eta_ {d} ^ {(1)} w + \hat {\sigma} _ {4, d} ^ {2} D _ {4} (t) \cdot \left(4 \eta_ {d} ^ {(2)} w ^ {3} - 2 \eta_ {d} ^ {(3)} w\right)\right) (D.42) \\ = - \left(1 - w ^ {2}\right) \cdot \left(P _ {t} (w) + 2 \hat {\sigma} _ {2, d} ^ {2} D _ {2} (t) \cdot \left(\eta_ {d} ^ {(1)} - 1\right) w \right. (D.43) \\ \left. + 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4} (t) \cdot \left(\eta_ {d} ^ {(2)} - 1\right) w ^ {3} - 2 \hat {\sigma} _ {4, d} ^ {2} D _ {4} (t) \eta_ {d} ^ {(3)} w\right) (D.44) \\ = - \left(1 - w ^ {2}\right) \cdot \left(P _ {t} (w) + Q _ {t} (w)\right) (D.45) \\ \end{array}
$$
Here we have chosen  $\lambda_{d}^{(1)} = 2\hat{\sigma}_{2,d}^{2}(\eta_{d}^{(1)} - 1)D_{2}(t) - 2\hat{\sigma}_{4,d}^{2}D_{4}(t)\eta_{d}^{(3)}$  and  $\lambda_{d}^{(3)} = 4\hat{\sigma}_{4,d}^{2}D_{4}(t)(\eta_{d}^{(2)} - 1)$ . Observe that  $|\lambda_{d}^{(1)}|, |\lambda_{d}^{(3)}| \leq O\left(\frac{\hat{\sigma}_{2,d}^{2}|D_{2,t}| + \hat{\sigma}_{4,d}^{2}|D_{4,t}|}{d}\right)$ , as desired.
D.3 Analysis of Phase 1
Recall the formal definition of Phase 1 in Definition 4.5. Intuitively, during Phase 1, the particles are all growing at a similar rate — because the particles  $w$  are all less than  $w_{\max} = \frac{1}{\log d}$  in magnitude, the cubic term involving  $w^3$  in the velocity of  $w$  does not have a significant effect on the growth of  $w$ . We formalize this in the following lemma.
Lemma D.4 (Uniform Growth Lemma). Suppose we are in the setting of Theorem 3.3. Consider a time interval  $[t_0, t_1] \in [0, T_2]$  in Phase 1 or 2. We are interested in quantifying the (relative) growth rate of the neurons indexed (or initialized) at  $\iota_1$  and  $\iota_2$  (where  $0 < \iota_1 < \iota_2$ ), defined as
$$
F _ {1} \triangleq \frac {w _ {t _ {1}} (\iota_ {1})}{w _ {t _ {0}} (\iota_ {1})}
$$
$$
F _ {2} \triangleq \frac {w _ {t _ {1}} (\iota_ {2})}{w _ {t _ {0}} (\iota_ {2})}
$$
Let  $\delta \in (0,1)$  such that  $\frac{1}{\sqrt{d}} \lesssim \delta \leq \frac{1}{\log F_2} - \text{here}$ , we implicitly assume that  $F_2 \leq e^d$ . We also assume that  $\delta \leq \frac{\gamma_2^2}{16}$ . Further assume that  $w_{t_1}(\iota_2) = \delta$ . Finally assume that  $\mathbb{P}(|\iota| \geq |\iota_2|) \leq \frac{\gamma_2^2}{16}$ . Then,
$$
F _ {1} \asymp F _ {2} \tag {D.46}
$$
and additionally,
$$
F _ {1} \asymp \exp \left(\int_ {t _ {0}} ^ {t _ {1}} 2 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | d t\right) \tag {D.47}
$$
$$
t _ {1} - t _ {0} \lesssim \frac {1}{\hat {\sigma} _ {2 , d} ^ {2} \gamma_ {2}} \log F _ {2} \tag {D.48}
$$
We note that  $F_{1}$  and  $F_{2}$  are both at least 1.
Note that this lemma also holds for  $\iota_2 < \iota_1 < 0$ , and with  $w_{t_1}(\iota_2) = -\delta$ , by the symmetry of  $\rho_t$ .
Proof of Lemma D.4. In this proof, we define  $\tau = \sqrt{\gamma_2}$  — then, by Assumption 3.2, we can write  $\gamma_4 = \beta \cdot \tau^4$  where  $1.1 \leq \beta$  and  $\beta$  is less than some universal constant  $c_1$ . Suppose at time  $t_1$ ,  $w_{t_1}(\iota_2) = \delta$ . One useful observation, which we use in the rest of this proof, is that for all  $t \leq t_1$ ,  $w_t(\iota_2) \leq \delta$ , since for  $w \gtrsim \frac{1}{\sqrt{d}}$ , we have  $\frac{dw}{dt} \geq 0$  by Lemma 4.4 (here we used the assumption that  $\delta \gtrsim \frac{1}{\sqrt{d}}$ ). Thus,
$$
\underset {w \sim \rho} {\mathbb {E}} [ P _ {2, d} (w) ] = \mathbb {P} (| \iota | > | \iota_ {2} |) \cdot \underset {w \sim \rho} {\mathbb {E}} [ P _ {2, d} (w) | | \iota | > | \iota_ {2} | ] + \mathbb {P} (| \iota | <   | \iota_ {2} |) \cdot \underset {w \sim \rho} {\mathbb {E}} [ P _ {2, d} (w) | | \iota | <   | \iota_ {2} | ] \tag {D.49}
$$
$$
\leq \frac {\gamma_ {2} ^ {2}}{1 6} + \underset {w \sim \rho} {\mathbb {E}} \left[ P _ {2, d} (w) \mid | \iota | <   | \iota_ {2} | \right]
$$
(B.c.  $|P_{2,d}(w)| \leq 1$  by Eq. (2.116) of Atkinson and Han [12] and  $\mathbb{P}(|\iota| > |\iota_2|) \leq \delta$ )
$$
\leq \frac {\gamma_ {2} ^ {2}}{1 6} + \delta^ {2} + O \left(\frac {1}{d}\right)
$$
(By Eq. (C.4) and b.c.  $|w_{t}(\iota)| \leq \delta$  if  $|\iota| \leq |\iota_{2}|$  by Proposition D.37)
$$
\leq \frac {\tau^ {4}}{8} + O \left(\frac {1}{d}\right)
$$
$$
(\mathrm {B . c .} \delta \leq \frac {\gamma_ {2} ^ {2}}{1 6} \text {a n d} \gamma_ {2} = \tau^ {2})
$$
Thus, writing  $\gamma_{2} = \tau^{2}$ , we have
$$
\left| D _ {2, t} \right| = \left| \underset {w \sim \rho} {\mathbb {E}} \left[ P _ {2, d} (w) \right] - \tau^ {2} \right| \geq \left| \frac {\tau^ {4}}{8} + O \left(\frac {1}{d}\right) - \tau^ {2} \right| \geq \frac {\tau^ {2}}{2} \tag {D.50}
$$
where the last inequality is by Assumption 3.2 b.c.  $d \geq c_{3}$  and  $\gamma_{4} \lesssim \gamma_{2}$ . By a similar argument,
$$
\left| D _ {4, t} \right| = \left| \underset {w \sim \rho} {\mathbb {E}} \left[ P _ {4, d} (w) \right] - \beta \tau^ {4} \right| \tag {D.51}
$$
$$
\leq \beta \tau^ {4} + \frac {\gamma_ {2} ^ {2}}{1 6} + \delta^ {4} + O \left(\frac {1}{d}\right)
$$
(Using  $|P_{4,d}(w)| \leq 1$ ,  $\mathbb{P}(|\iota| > |\iota_2|) \leq \frac{\gamma_2^2}{16}$ ,  $\delta \leq \frac{\tau^4}{16}$  and Eq. (C.4))
$$
\leq (\beta + 1 / 8) \tau^ {4} \quad \left(\mathrm {B . c .} \delta \leq \frac {\gamma_ {2} ^ {2}}{1 6} \leq \frac {\beta \tau^ {4}}{1 6}\right)
$$
$$
\leq \left| D _ {2, t} \right| \tag {D.52}
$$
Thus, for any time  $t \in [t_0, t_1]$ , for any  $\iota \in [\iota_1, \iota_2]$ , by Lemma 4.4 (and because  $w_t(\iota) \leq \delta$  for any  $\iota \in [\iota_1, \iota_2]$  by Proposition D.37), we have
$$
v _ {t} \left(w _ {t} (\iota)\right) = - \left(1 - w _ {t} (\iota) ^ {2}\right) \cdot \left(2 \hat {\sigma} _ {2, d} ^ {2} D _ {2, t} w _ {t} (\iota) \cdot (1 \pm O (\delta)) + Q _ {t} \left(w _ {t} (\iota)\right)\right)
$$
(B.c.  $\hat{\sigma}_{4,d}^2\lesssim \hat{\sigma}_{2,d}^2$  by Assumption 3.2,  $|D_{4,t}|\leq |D_{2,t}|$  , and  $w_{t}(\iota)\leq \delta$
$$
= 2 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | w _ {t} (\iota) \cdot (1 \pm O (\delta)) - O \left(\frac {\hat {\sigma} _ {2 , d} ^ {2} | D _ {2 , t} |}{d}\right) | w _ {t} (\iota) |
$$
(B.c.  $|w_{t}(\iota)|\leq \delta$  and by bound on coefficients of  $Q_{t}$  from Lemma 4.4)
$$
= 2 \hat {\sigma} _ {2, d} ^ {2} \left| D _ {2, t} \right| w _ {t} (\iota) \cdot (1 \pm O (\delta)) \quad \left(\mathrm {B . c .} \delta \geq \frac {1}{d}\right)
$$
In summary,
$$
v _ {t} \left(w _ {t} (\iota)\right) = 2 \hat {\sigma} _ {2, d} ^ {2} \left| D _ {2, t} \right| w _ {t} (\iota) \cdot (1 \pm O (\delta)) \tag {D.53}
$$
$$
\frac {v \left(w _ {t} \left(\iota_ {2}\right)\right)}{w _ {t} \left(\iota_ {2}\right)} \leq \frac {1 + O (\delta)}{1 - O (\delta)} \frac {v \left(w _ {t} \left(\iota_ {1}\right)\right)}{w _ {t} \left(\iota_ {1}\right)} \leq (1 + O (\delta)) \cdot \frac {v \left(w _ {t} \left(\iota_ {1}\right)\right)}{w _ {t} \left(\iota_ {1}\right)} \tag {D.54}
$$
and  $w_{t}(\iota_{2})$  grows by a factor
$$
F _ {2} = \exp \left(\int_ {t _ {0}} ^ {t _ {1}} \frac {v \left(w _ {t} \left(\iota_ {2}\right)\right)}{w _ {t} \left(\iota_ {2}\right)} d t\right) \leq \exp \left((1 + O (\delta)) \int_ {t _ {0}} ^ {t _ {1}} \frac {v \left(w _ {t} \left(\iota_ {1}\right)\right)}{w _ {t} \left(\iota_ {1}\right)} d t\right) = F _ {1} ^ {1 + O (\delta)} \tag {D.55}
$$
Rearranging gives
$$
F _ {1} \geq F _ {2} ^ {1 - O (\delta)} = \frac {F _ {2}}{F _ {2} ^ {O (\delta)}} \tag {D.56}
$$
Finally, observe that  $F_2^{O(\delta)} \leq O(1)$ , since by the assumption that  $\delta \leq \frac{1}{\log F_2}$ ,
$$
F _ {2} ^ {\delta} \leq F _ {2} ^ {\frac {1}{\log F _ {2}}} \tag {D.57}
$$
and for any  $x > 0$ ,  $x^{\frac{1}{\ln x}} = e$ , meaning that  $F_2^\delta \leq O(1)$ . This implies that  $F_1 \gtrsim F_2$ . From the above calculations, we can also obtain
$$
\begin{array}{l} \exp \left(\int_ {t _ {0}} ^ {t _ {1}} 2 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | d t\right) \gtrsim \exp \left(\int_ {t _ {0}} ^ {t _ {1}} \frac {1}{1 + O (\delta)} \cdot \frac {v _ {t} \left(w _ {t} \left(\iota_ {2}\right)\right)}{w _ {t} \left(\iota_ {2}\right)} d t\right) (ByEq.(D.53)) \\ \gtrsim \exp \left((1 - O (\delta)) \cdot \int_ {t _ {0}} ^ {t _ {1}} \frac {v _ {t} \left(w _ {t} \left(\iota_ {2}\right)\right)}{w _ {t} \left(\iota_ {2}\right)} d t\right) (D.58) \\ \gtrsim F _ {2} ^ {1 - O (\delta)} (D.59) \\ \stackrel {>} {_ \sim} F _ {2} \quad (\text {B y E q . (D . 5 7)}) \\ \end{array}
$$
as desired. Similarly, we can obtain
$$
\begin{array}{l} \exp \left(\int_ {t _ {0}} ^ {t _ {1}} 2 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | d t\right) \lesssim \exp \left(\int_ {t _ {0}} ^ {t _ {1}} (1 + O (\delta)) \cdot \frac {v _ {t} \left(w _ {t} \left(\iota_ {2}\right)\right)}{w _ {t} \left(\iota_ {2}\right)} d t\right) (D.60) \\ \lesssim F _ {2} ^ {1 + O (\delta)} (D.61) \\ \lesssim F _ {2} \quad (\mathrm {B . c .} \delta \leq \frac {1}{\log F _ {2}}) \\ \end{array}
$$
$$
\begin{array}{l} F _ {1} \lesssim \exp \left(\int_ {t _ {0}} ^ {t _ {1}} (1 + O (\delta)) \cdot \frac {v _ {t} \left(w _ {t} \left(\iota_ {2}\right)\right)}{w _ {t} \left(\iota_ {2}\right)} d t\right) (D.62) \\ \lesssim F _ {2} ^ {1 + O (\delta)} (D.63) \\ \lesssim F _ {2} \quad (\mathrm {B . c .} \delta \leq \frac {1}{\log F _ {2}}) \\ \end{array}
$$
as desired. Finally, we show the running time bound. Since  $\delta \leq \frac{\gamma_2^2}{16}$  and  $\gamma_{2}$  is less than a sufficiently small universal constant by Assumption 3.2, for any time  $t$  during this interval,  $v(w_{t}(\iota_{2}))\gtrsim \hat{\sigma}_{2,d}^{2}D_{2}(t)w_{t}(\iota_{2})\gtrsim \hat{\sigma}_{2,d}^{2}\tau^{2}w_{t}(\iota_{2})$  (where the second inequality is by Eq. (D.50)). Thus, by Gronwall's inequality (Fact I.13), the time elapsed when  $w_{t}(\iota_{2})$  grows by a factor of  $F_{2}$  is at most  $\frac{1}{\hat{\sigma}_{2,d}^{2}\tau^{2}}\log F_{2}$ .
D.3.1 Summary of Phase 1
As a direct consequence of the above lemma, we can show Lemma D.5, which characterizes how fast the particles  $w_{t}$  grow during Phase 1.
Lemma D.5 (Phase 1). Suppose we are in the setting of Theorem 3.3. At the end of phase 1, i.e. at time  $T_{1}$ , for all  $\iota \in (0, \iota_{\mathrm{U}})$ , we have  $w_{T_1}(\iota) \asymp \frac{\sqrt{d}}{(\log d)^2}\iota$ . Furthermore,  $T_{1} \lesssim \frac{1}{\hat{\sigma}_{2,d}^{2}\gamma_{2}}\log d$ . Additionally, we have  $\exp \left(\int_0^{T_1}2\hat{\sigma}_{2,d}^2 |D_{2,t}|dt\right) \asymp \frac{\sqrt{d}}{(\log d)^2}$ .
Proof of Lemma D.5. To show the lower bound on  $w_{T_1}(\iota)$ , we apply Lemma D.4, with  $t_0 = 0$  and  $t_1 = T_1$ , with  $\delta = \frac{1}{\log d}$ , and with  $\iota_1 = \iota$  and  $\iota_2 = \iota_{\mathrm{U}}$ . Let us verify that the assumptions of Lemma D.4 hold:
- Observe that  $\frac{w_{T_1}(\iota_{\mathrm{U}})}{\iota_{\mathrm{U}}} = \frac{1 / \log d}{\log d / \sqrt{d}} = \frac{\sqrt{d}}{(\log d)^2}$ , meaning that  $\log F_2 \leq \log d$  and  $\frac{1}{\log F_2} \geq \delta$ .
- By Assumption 3.2,  $\delta = \frac{1}{\log d} \leq \frac{\gamma_2^2}{16}$ , since  $d \geq c_3$ .
- The assumption that  $\mathbb{P}(|\iota| \geq \iota_{\mathrm{U}}) \leq \frac{\gamma_2^2}{16}$  holds by a standard bound on the tail of  $\mu_d$ .
Using the notation of Lemma D.4,
$$
F _ {2} = \frac {w _ {T _ {1}} \left(\iota_ {\mathrm {U}}\right)}{\iota_ {\mathrm {U}}} = \frac {w _ {\max }}{(\log d) / \sqrt {d}} = \frac {\sqrt {d}}{(\log d) ^ {2}} \tag {D.64}
$$
where  $w_{\mathrm{max}}$  is as defined in Definition 4.5. By Eq. (D.46), we therefore know that
$$
\frac {\sqrt {d}}{(\log d) ^ {2}} \gtrsim \frac {w _ {T _ {1}} (\iota)}{\iota} \gtrsim \frac {\sqrt {d}}{(\log d) ^ {2}} \tag {D.65}
$$
For the bound on the running time, observe that by Eq. (D.48),  $T_{1} \lesssim \frac{1}{\delta_{2,d}^{2}\gamma_{2}} \log \frac{\sqrt{d}}{(\log d)^{2}}$ . The final statement holds because  $F_{2} = \frac{\sqrt{d}}{(\log d)^{2}}$  and from the second statement of Lemma D.4. This completes the proof.
D.4 Analysis of Phase 2
Recall the formal definition of Phase 2 from Definition 4.6. Our analysis in this section has two main goals: (1) to show that the running time of Phase 2 is small, and (2) to show that a majority of the particles become at least  $\frac{1}{\log \log d}$  in magnitude, up to a constant factor. We will achieve goal (2) using Lemma D.4 — then, we show that goal (1) holds using goal (2). Additionally, goal (2) will be useful for the subsequent phases as well.
First, the following proposition defines a range of particles which will be relevant during Phase 2 and Phase 3. This range of particles is a strict subset of the interval  $(0,\iota_{\mathrm{U}})$  which we considered during Phase 1 - we will show that this new range of particles grows uniformly during the beginning of Phase 2, until the largest of them becomes about  $\frac{1}{\log\log d}$  in magnitude. Specifically, we define  $\iota_{\mathrm{L}}\triangleq \frac{\kappa}{\sqrt{d}}$  and  $\iota_{\mathrm{R}}\triangleq \frac{1}{\kappa\sqrt{d}}$  where  $\kappa \asymp \frac{1}{\log\log d}$ . The following proposition states that at initialization, a very large fraction of the  $\iota$ 's have magnitude between  $\iota_{\mathrm{L}}$  and  $\iota_{\mathrm{R}}$ .
Proposition D.6. Let  $\kappa \in (0,1)$ , and suppose  $\iota$  is drawn uniformly at random from  $\mu_d$ . Define  $\iota_{\mathrm{L}} = \frac{\kappa}{\sqrt{d}}$  and  $\iota_{\mathrm{R}} = \frac{1}{\kappa\sqrt{d}}$ . Then,  $\mathbb{P}(|\iota| \notin [\iota_{\mathrm{L}}, \iota_{\mathrm{R}}]) \lesssim \kappa$ .
Proof of Proposition D.6. First, we upper bound the probability that  $|\iota| \geq \iota_{\mathbb{R}}$ . By Markov's inequality,
$$
\mathbb {P} \left(\left| \iota \right| \geq \iota_ {\mathrm {R}}\right) \leq \frac {\mathbb {E} \left[ \iota^ {2} \right]}{\iota_ {\mathrm {R}} ^ {2}} = \frac {1 / d}{1 / \left(\kappa^ {2} d\right)} = \kappa^ {2} \tag {D.66}
$$
Now, we upper bound the probability that  $|\iota| \leq \iota_{\mathrm{L}}$ . By Eqs. (1.16) and (1.18) of Atkinson and Han [12], the probability density function of  $\iota$  is  $p(\iota) = \frac{\Gamma(d/2)}{\sqrt{\pi} \Gamma((d-1)/2)} \left(1 - \iota^2\right)^{\frac{d-3}{2}}$ . Thus, we can simply upper bound the density by its maximum value:
$$
\begin{array}{l} \mathbb {P} (| \iota | \leq \iota_ {\mathrm {L}}) \lesssim \frac {\Gamma (d / 2)}{\sqrt {\pi} \Gamma ((d - 1) / 2)} \int_ {- \iota_ {\mathrm {L}}} ^ {\iota_ {\mathrm {L}}} (1 - t ^ {2}) ^ {\frac {d - 3}{2}} d t (D.67) \\ \lesssim \frac {\Gamma (d / 2)}{\sqrt {\pi} \Gamma ((d - 1) / 2)} \cdot 2 \iota_ {\mathrm {L}} (D.68) \\ \lesssim \sqrt {d} \cdot \iota_ {\mathrm {L}} \quad (\text {B y S t i r l i n g ' s F o r m u l a}) \\ \lesssim \sqrt {d} \cdot \frac {\kappa}{\sqrt {d}} (D.69) \\ \lesssim \kappa (D.70) \\ \end{array}
$$
Next, as a corollary of Lemma D.4, we show that for a portion of Phase 2, the particles initialized at  $\iota \in [\iota_{\mathrm{L}},\iota_{\mathrm{R}}]$  grow by a similar factor. Specifically, this is true until the largest of these particles is  $\xi = \frac{1}{2\log\log d}$  in magnitude.
Lemma D.7 (Corollary of Lemma D.4). Suppose we are in the setting of Theorem 3.3. Let  $T_{\mathrm{R}} > T_{1}$  be the first time such that  $w_{T_{\mathrm{R}}}(\iota_{\mathrm{R}}) = \xi = \frac{1}{2\log\log d}$ . Then, for all  $\iota$  with  $|\iota| \in [\iota_{\mathrm{L}}, \iota_{\mathrm{R}}]$ , we have  $|w_{T_{\mathrm{R}}}(\iota)| \gtrsim \xi \kappa^{2}$ , and additionally,  $\frac{w_{T_{\mathrm{R}}}(\iota)}{w_{T_1}(\iota)} \asymp \xi \kappa (\log d)^2$ . Furthermore,  $T_{\mathrm{R}} - T_{1} \lesssim \frac{1}{\hat{\sigma}_{2,d}^{2}\gamma_{2}} \log \log d$ .
Proof of Lemma D.7. By Lemma D.5,  $w_{T_1}(\iota_{\mathrm{R}}) \asymp \frac{\sqrt{d}}{(\log d)^2}\iota_{\mathrm{R}} = \frac{1}{\kappa(\log d)^2}$ . Let  $\iota_{\mathrm{L}} = \frac{\kappa}{\sqrt{d}}$  and  $\iota_{\mathrm{R}} = \frac{1}{\kappa\sqrt{d}}$ , where  $\kappa = \frac{1}{\log\log d}$ . Now, we apply Lemma D.4, with  $t_0 = T_1$  and  $t_1 = T_{\mathrm{R}}$ , and with  $\iota_{1} = \iota$  for some  $\iota \in [\iota_{\mathrm{L}},\iota_{\mathrm{R}}]$  and  $\iota_{2} = \iota_{\mathrm{R}}$ , and with  $\delta = \frac{1}{2\log\log d}$ . Note that we assume without loss of generality that  $\iota > 0$  — the statement for  $\iota < 0$  follows by the symmetry of  $\rho_t$ . Let us verify that the assumptions hold:
- First,  $\delta = \frac{1}{2\log\log d} \gtrsim \frac{1}{\sqrt{d}}$ . Additionally, observe that by Lemma D.5,  $\frac{w_{T_1}(\iota_{\mathbb{R}})}{\iota_{\mathbb{R}}} \asymp \frac{\sqrt{d}}{(\log d)^2}$ , meaning that  $w_{T_1}(\iota_{\mathbb{R}}) \asymp \frac{1}{\kappa(\log d)^2}$ . Thus, using the notation of Lemma D.4, we have  $F_2 \lesssim \frac{1 / \log\log d}{1 / (\kappa(\log d)^2)} = \frac{\kappa(\log d)^2}{\log\log d}$ , and  $\log F_2 \leq 2\log\log d - 2\log\log\log d + C < \frac{1}{\delta}$ , as desired. (Here  $C$  is a universal constant, and the last inequality holds by Assumption 3.2 since  $d \geq c_3$ .)
- By Assumption 3.2,  $\delta = \frac{1}{2\log\log d} \lesssim \frac{\gamma_2^2}{16}$  since  $d \geq c_3$ .
- Additionally, the probability that  $|\iota| \geq |\iota_{\mathrm{R}}|$  is at most  $\kappa = \frac{1}{\log \log d}$  up to constant factors by Proposition D.6, and this is clearly at most  $\frac{\gamma_2^2}{16}$  by Assumption 3.2.
Using Lemma D.4, we obtain
$$
\frac {w _ {T _ {\mathrm {R}}} (\iota)}{w _ {T _ {1}} (\iota)} \asymp \frac {w _ {T _ {\mathrm {R}}} \left(\iota_ {\mathrm {R}}\right)}{w _ {T _ {1}} \left(\iota_ {\mathrm {R}}\right)} \asymp \frac {\xi}{w _ {T _ {1}} \left(\iota_ {\mathrm {R}}\right)} \asymp \frac {\xi}{\left(\frac {1}{\kappa (\log d) ^ {2}}\right)} \asymp \xi \kappa (\log d) ^ {2} \tag {D.71}
$$
Thus, using the conclusion of Lemma D.5 that  $\frac{w_{T_1}(\iota)}{\iota} \gtrsim \frac{\sqrt{d}}{(\log d)^2}$ , we obtain
$$
w _ {T _ {\mathrm {R}}} (\iota) \gtrsim \frac {w _ {T _ {\mathrm {R}}} (\iota)}{w _ {T _ {1}} (\iota)} \cdot \frac {w _ {T _ {1}} (\iota)}{\iota} \cdot \iota \gtrsim \xi \kappa (\log d) ^ {2} \cdot \frac {\sqrt {d}}{(\log d) ^ {2}} \cdot \frac {\kappa}{\sqrt {d}} \gtrsim \xi \kappa^ {2} \tag {D.72}
$$
as desired. Thus, we obtain the first statement of this lemma. In addition, by Eq. (D.48), and since  $w_{T_{\mathbb{R}}}(\iota_{\mathbb{R}}) = \xi$  and  $w_{T_1}(\iota_{\mathbb{R}}) \gtrsim \frac{1}{\kappa(\log d)^2}$ , the running time bound we obtain for this part of Phase 2 is  $T_{\mathrm{R}} - T_{1} \lesssim \frac{1}{\hat{\sigma}_{2,d}^2\tau^2}\log \frac{\xi}{1 / (\kappa(\log d)^2)} \lesssim \frac{1}{\hat{\sigma}_{2,d}^2\tau^2} (\log \log d)$ , as desired.
We now prove some auxiliary lemmas. First, we show an upper bound on  $L(\rho_0)$ , which is needed in order to later show an upper bound on the time required to have  $L(\rho_t) \leq (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2$ . In the process, we also show a reasonably tight bound on the magnitude of  $D_{2,t}$  and  $D_{4,t}$  during the first part of Phase 2.
Lemma D.8. Suppose we are in the setting of Theorem 3.3 and Lemma D.7, and suppose  $0 \leq t \leq T_{\mathbb{R}}$ . Then,  $-\frac{3\gamma_2}{2} \leq D_{2,t} \leq -\frac{\gamma_2}{2}$  and  $-\frac{3\gamma_4}{2} \leq D_{4,t} \leq -\frac{\gamma_4}{2}$ . As a corollary, for all  $t \geq 0$ ,  $L(\rho_t) \lesssim (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\gamma_2^2$ .
Proof. Since  $t \leq T_{\mathrm{R}}$ ,  $w_{t}(\iota_{\mathrm{R}}) \leq \xi = \frac{1}{2 \log \log d}$  (because for  $w \gtrsim \frac{1}{\sqrt{d}}$ , we have  $\frac{dw}{dt} \geq 0$  by Lemma 4.4). By Proposition D.37, for all  $\iota$  with  $|\iota| \leq \iota_{\mathrm{R}}$ ,  $|w_{t}(\iota)| \leq \xi$ . Thus, by Proposition D.6,
$$
\begin{array}{l} \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ w ^ {2} \right] \leq \mathbb {P} _ {\iota \sim \rho_ {0}} \left(\left| \iota \right| \leq \iota_ {\mathrm {R}}\right) \cdot \mathbb {E} \left[ w ^ {2} \mid \left| \iota \right| \leq \iota_ {\mathrm {R}} \right] + \mathbb {P} _ {\iota \sim \rho_ {0}} \left(\left| \iota \right| \geq \iota_ {\mathrm {R}}\right) \cdot \mathbb {E} \left[ w ^ {2} \mid \left| \iota \right| \geq \iota_ {\mathrm {R}} \right] \tag {D.73} \\ \leq \xi^ {2} + O (\kappa) \quad \text {(B y P r o p o s i t i o n D . 6)} \\ \end{array}
$$
Thus,  $D_{2,t} = O(\xi^2 +\kappa) - \gamma_2$  , meaning that  $-\frac{3\gamma_2}{2}\leq D_2\leq -\frac{\gamma_2}{2}$  , and similarly,  $-\frac{3\gamma_4}{2}\leq D_4\leq -\frac{\gamma_4}{2}$  The corollary follows from the fact that  $\gamma_4\lesssim \gamma_2^2$  by Assumption 3.2.
Next, we show that the particles are in fact increasing in magnitude during Phases 1 and 2. This makes use of our bound on the magnitudes of  $D_{2,t}$  and  $D_{4,t}$  up to time  $T_{\mathrm{R}}$ .
Lemma D.9. Suppose we are in the setting of Theorem 3.3, and suppose  $[t_0,t_1]$  is a time interval with  $0\leq t_0 < t_1\leq T_2$ . Then, for all  $\iota$  with  $|\iota |\in [\iota_{\mathrm{L}},\iota_{\mathrm{R}}],|w_t(\iota)|$  is increasing on  $[t_0,t_1]$ .
Proof. Recall that by Lemma 4.4,
$$
v _ {t} (w) = - \left(1 - w ^ {2}\right) \left(P _ {t} (w) + Q _ {t} (w)\right) \tag {D.74}
$$
and for  $w\geq 0$  , it suffices to show that  $P_{t}(w) + Q_{t}(w)\leq 0$  . For  $t\le T_{\mathbb{R}}$
$$
\begin{array}{l} P _ {t} (w) + Q _ {t} (w) \leq 2 \hat {\sigma} _ {2, d} ^ {2} D _ {2, t} w + 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} w ^ {3} + O \left(\frac {\hat {\sigma} _ {2 , d} ^ {2} | D _ {2 , t} | + \hat {\sigma} _ {4 , d} ^ {2} | D _ {4 , t} |}{d}\right) | w | \tag {D.75} \\ \leq - \hat {\sigma} _ {2, d} ^ {2} \gamma_ {2} w + O \left(\frac {\hat {\sigma} _ {2 , d} ^ {2} | D _ {2 , t} | + \hat {\sigma} _ {4 , d} ^ {2} | D _ {4 , t} |}{d}\right) | w | \quad (\text {B y}) \\ \leq 0 \quad (\text {B . c .} \hat {\sigma} _ {4, d} ^ {2} \lesssim \hat {\sigma} _ {2, d} ^ {2} \text {a n d} \gamma_ {2} \gtrsim \frac {| D _ {2 , t} |}{d} \text {a n d} \gamma_ {4} \gtrsim \frac {| D _ {4 , t} |}{d} (\text {A s s u m p t i o n 3 . 2})) \\ \end{array}
$$
For  $t \geq T_{\mathrm{R}}$ , suppose  $w \gtrsim \xi \kappa^2$  (which holds at time  $T_{\mathrm{R}}$  for  $\iota \in [\iota_{\mathrm{L}}, \iota_{\mathrm{R}}]$  by Lemma D.7). Then,
$$
\begin{array}{l} P _ {t} (w) + Q _ {t} (w) \leq 2 \hat {\sigma} _ {2, d} ^ {2} D _ {2, t} w + 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} w ^ {3} + O \left(\frac {\hat {\sigma} _ {2 , d} ^ {2} | D _ {2 , t} | + \hat {\sigma} _ {4 , d} ^ {2} | D _ {4 , t} |}{d}\right) | w | (By Lemma 4.4) \\ \leq \hat {\sigma} _ {2, d} ^ {2} D _ {2, t} \left(2 w - \frac {C w}{d}\right) + \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} \left(4 w ^ {3} - \frac {C w}{d}\right) (D.76) \\ \end{array}
$$
for some universal constant  $C > 0$ . The final expression on the right-hand side is negative — to see this, note that  $2w - \frac{w}{d} > 0$  if  $w \gtrsim \xi \kappa^2$ , and  $4w^3 - \frac{w}{d} > 0$  if  $w \gtrsim \xi \kappa^2$  since  $d \geq c_3$  by Assumption 3.2. This completes the proof of the lemma when  $\iota \geq 0$ . For  $\iota < 0$ , the lemma holds by the symmetry of  $\rho_t$ .
D.4.1 Summary of Phase 2