SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma D.21 (Phase 3, Case 1 Invariant). Suppose we are in the setting of Theorem 3.3. Let  $T_{\mathbb{R}}$  be as defined in the statement of Lemma D.7, and assume that  $D_{2,T_2} = 0$  and  $D_{4,T_2} < 0$ . Then, for all  $t \geq T_{\mathbb{R}}$  (in particular, for all times  $t$  during Phase 3 Case 1), if  $\iota, \iota'$  are sampled independently from  $\rho_0$ ,
$$
\left| \Phi \left(w _ {t} (\iota)\right) - \Phi \left(w _ {t} \left(\iota^ {\prime}\right)\right) \right| \gtrsim \xi \kappa^ {3} \tag {D.163}
$$
with probability at least  $1 - O(\kappa)$
Proof of Lemma D.21. By Lemma D.19 and Lemma D.20, if  $D_{2,T_2} = 0$  and  $D_{4,T_2} < 0$ , then for all  $t \geq T_2$ , we will have  $D_{2,t} \geq 0$  and  $D_{4,t} \leq 0$ . Thus, the lemma follows from Lemma D.13 and Lemma D.16, as well as the fact that  $D_{4,t} \leq 0$  for  $t \in [T_{\mathrm{R}}, T_2]$  (meaning that the potential difference is increasing in the time interval  $[T_{\mathrm{R}}, T_2]$ ).
Finally, we show Lemma D.22 which gives a running time bound for Phase 3, Case 1.
Lemma D.22 (Phase 3, Case 1 Summary). In the setting of Theorem 3.3, suppose that  $D_{2,T_2} = 0$  and  $D_{4,T_2} < 0$  (i.e. Phase 3, Case 1 holds). Then, the total amount of time  $t \geq T_2$  such that  $L(\rho_t) \geq \frac{1}{2} (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2$  is at most  $O\left(\frac{1}{\hat{\sigma}_{4,d}^2\gamma_2^8\xi^4\kappa^6}\log \left(\frac{\gamma_2}{\epsilon}\right)\right)$ .
Proof of Lemma D.22. Suppose  $t \geq T_2$  and assume that  $D_{2,T_2} = 0$  and  $D_{4,t} < 0$  (i.e. Phase 3, Case 1 occurs). First, suppose  $|D_{4,t}| \leq \xi |D_{2,t}|$  where  $\xi = \frac{1}{\log\log d}$ . Then, by Lemma D.18, if  $|w| \in [\gamma_2^{3/4}, 1 - \gamma_2^{1/2}]$ , we have
$$
\left| v _ {t} (w) \right| \gtrsim \gamma_ {2} ^ {5 / 4} \hat {\sigma} _ {2, d} ^ {2} \left| D _ {2, t} \right| \tag {D.164}
$$
(note that the lemma is stated for  $w \in [\gamma_2^{3/4}, 1 - \gamma_2^{1/4}]$ , but holds for  $|w| \in [\gamma_2^{3/4}, 1 - \gamma_2^{1/4}]$  since  $v_t(w)$  is an odd function). Therefore,
$$
\begin{array}{l} \underset {w \sim \rho_ {t}} {\mathbb {E}} | v _ {t} (w) | ^ {2} \gtrsim \mathbb {P} _ {w \sim \rho_ {t}} \left(| w | \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right]\right) \cdot \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ | v _ {t} (w) | ^ {2} \mid | w | \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right] \right] (D.165) \\ \gtrsim \frac {\gamma_ {2} ^ {2}}{2} \cdot \underset {w \sim \rho_ {t}} {\mathbb {E}} [ | v _ {t} (w) | ^ {2} \mid | w | \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right] ] (By Lemma D.17) \\ \gtrsim \frac {\gamma_ {2}}{2} \cdot \gamma_ {2} ^ {5 / 2} \hat {\sigma} _ {2, d} ^ {4} | D _ {2, t} | ^ {2} \quad (\text {B y E q . (D . 1 6 4)}) \\ \gtrsim \gamma_ {2} ^ {7 / 2} \hat {\sigma} _ {2, d} ^ {4} \left| D _ {2, t} \right| ^ {2} (D.166) \\ \begin{array}{c} \gtrsim \gamma_ {2} ^ {7 / 2} \hat {\sigma} _ {2, d} ^ {2} \cdot (\hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} | ^ {2} + \hat {\sigma} _ {4, d} ^ {2} | D _ {4, t} | ^ {2}) \\ \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad (B. c.   \hat {\sigma} _ {4, d} ^ {2} \lesssim \hat {\sigma} _ {2, d} ^ {2} (A s s u m p t i o n 3. 2) a n d | D _ {4, t} | \leq \xi | D _ {2, t} |) \end{array} \\ \gtrsim \gamma_ {2} ^ {7 / 2} \hat {\sigma} _ {2, d} ^ {2} L (\rho_ {t}) (D.167) \\ \end{array}
$$
Thus, by Lemma D.39, for any time  $t$  during Phase 3, Case 1 such that  $|D_{4,t}| \leq \xi |D_{2,t}|$ , we have
$$
\frac {d L \left(\rho_ {t}\right)}{d t} \lesssim - \gamma_ {2} ^ {7 / 2} \hat {\sigma} _ {2, d} ^ {2} L \left(\rho_ {t}\right) \tag {D.168}
$$
Let  $T_{*,\epsilon}$  be as in Theorem 3.3, and define
$$
I _ {\left| D _ {4} \right| \leq \xi \mid D _ {2} \mid} = \{t \leq T _ {* , \epsilon} \text {a n d} t \text {i n P h a s e 3 , C a s e 1} \mid \left| D _ {4, t} \right| \leq \xi \left| D _ {2, t} \right| \} \tag {D.169}
$$
Then, by Eq. (D.168),
$$
\frac {d L \left(\rho_ {t}\right)}{d t} \lesssim - \gamma_ {2} ^ {7 / 2} \hat {\sigma} _ {2, d} ^ {2} L \left(\rho_ {t}\right) 1 (t \in I _ {| D _ {4} | \leq \xi | D _ {2} |}) \tag {D.170}
$$
Thus, by Gronwall's inequality (Fact I.13),
$$
\begin{array}{l} \frac {L \left(\rho_ {T _ {* , \epsilon}}\right)}{L \left(\rho_ {0}\right)} \leq \exp \left(- C \int_ {T _ {2}} ^ {T _ {* , \epsilon}} \gamma_ {2} ^ {7 / 2} \hat {\sigma} _ {2, d} ^ {2} 1 (t \in I _ {| D _ {4} | \leq \xi | D _ {2} |}) d t\right) (D.171) \\ = \exp \left(- C \gamma_ {2} ^ {7 / 2} \hat {\sigma} _ {2, d} ^ {2} \int_ {I _ {| D _ {4} | \leq \xi | D _ {2} |}} 1 d t\right) (D.172) \\ \end{array}
$$
Since  $L(\rho_0) \lesssim (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\gamma_2^2$  by Lemma D.8 and  $L(\rho_{T_{*,\epsilon}}) \gtrsim (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)\epsilon^2$ , rearranging gives
$$
\int_ {I _ {| D _ {4} | \leq \xi | D _ {2} |}} 1 d t \lesssim \frac {1}{\hat {\sigma} _ {2 , d} ^ {2} \gamma_ {2} ^ {7 / 2}} \log \left(\frac {\gamma_ {2} ^ {2}}{\epsilon^ {2}}\right) \tag {D.173}
$$
Now, suppose  $t$  is such that  $|D_{4,t}| \geq \xi |D_{2,t}|$ . Let us lower bound the average velocity for  $w \in [\gamma_2^{3/4}, 1 - \gamma_2^{1/4}]$  first (then, we can conclude the average velocity is large by Lemma D.17). By Lemma 4.4,
$$
\left| v _ {t} (w) \right| = \left(1 - w ^ {2}\right) \left| P _ {t} (w) + Q _ {t} (w) \right| \tag {D.174}
$$
and we can write
$$
P _ {t} (w) = 2 \hat {\sigma} _ {2, d} ^ {2} D _ {2, t} w + 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} w ^ {3} = 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} w (w - r) (w + r) \tag {D.175}
$$
where  $r = \sqrt{-\frac{\hat{\sigma}_{2,d}^2D_{2,t}}{2\hat{\sigma}_{4,d}^2D_{4,t}}}$  is the positive root of  $P_{t}$ . For  $|w|\in [\gamma_2^{3 / 4},1 - \gamma_2^{1 / 2}]$ , we have
$$
1 - w ^ {2} \geq 1 - \left(1 - \gamma_ {2} ^ {1 / 2}\right) ^ {2} = 2 \gamma_ {2} ^ {1 / 2} - \gamma_ {2} \geq \gamma_ {2} ^ {1 / 2}. \tag {D.176}
$$
Additionally,  $|w| \geq \gamma_2^{3/4}$  and  $|w + r| \geq \gamma_2^{3/4}$ . Thus, for  $w \in [\gamma_2^{3/4}, 1 - \gamma_2^{1/2}]$ , we have
$$
\left| v _ {t} (w) \right| ^ {2} \gtrsim (1 - w ^ {2}) ^ {2} \left(P _ {t} (w) + Q _ {t} (w)\right) ^ {2} \tag {D.177}
$$
$$
\gtrsim \gamma_ {2} \left(P _ {t} (w) + Q _ {t} (w)\right) ^ {2} \quad (\mathrm {B . c .} 1 - w ^ {2} \geq \gamma_ {2} ^ {1 / 2})
$$
$$
\gtrsim \gamma_ {2} \cdot \left(P _ {t} (w) ^ {2} - 2 Q _ {t} (w) ^ {2}\right) \quad (\mathrm {B . c .} (a + b) ^ {2} \leq 2 a ^ {2} + 2 b ^ {2}, \text {s o} a ^ {2} - 2 b ^ {2} \leq 2 (a + b) ^ {2})
$$
$$
\gtrsim \gamma_ {2} \cdot \left(P _ {t} (w) ^ {2} - O \left(\frac {\hat {\sigma} _ {2 , d} ^ {4} D _ {2 , t} ^ {2} + \hat {\sigma} _ {4 , d} ^ {4} D _ {4 , t} ^ {2}}{d ^ {2}}\right) w ^ {2}\right) \tag {By Lemma 4.4}
$$
$$
\gtrsim \gamma_ {2} \cdot \left(P _ {t} (w) ^ {2} - O \left(\frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) L \left(\rho_ {t}\right)}{d ^ {2}}\right) w ^ {2}\right) \tag {By Lemma 4.2}
$$
$$
\gtrsim \gamma_ {2} \cdot \left(\hat {\sigma} _ {4, d} ^ {4} D _ {4, t} ^ {2} w ^ {2} (w + r) ^ {2} (w - r) ^ {2} - O \left(\frac {(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}) L (\rho_ {t})}{d ^ {2}}\right) w ^ {2}\right)
$$
(By factorization of  $P_{t}(w)$  above)
$$
\gtrsim \gamma_ {2} \cdot \left(\hat {\sigma} _ {4, d} ^ {4} D _ {4, t} ^ {2} \gamma_ {2} ^ {3} (w - r) ^ {2} - O \left(\frac {(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}) L (\rho_ {t})}{d ^ {2}}\right) w ^ {2}\right) \tag {D.178}
$$
where the last inequality is by the discussion above Eq. (D.177). Now, in order to take the expectation over  $w \sim \rho_t$  (conditioned on  $w \in [\gamma_2^{3/4}, 1 - \gamma_2^{1/2}]$ ), we first compute  $\mathbb{E}_{w \sim \rho_t}[(w - r)^2 \mid w \in [\gamma_2^{3/4}, 1 - \gamma_2^{1/2}]$ ):
$$
\underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ (w - r) ^ {2} \mid w \sim \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right] \right] \geq \operatorname {V a r} _ {w \sim \rho_ {t}} \left(w \mid w \sim \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right]\right) \tag {D.179}
$$
$$
\gtrsim \underset {w, w ^ {\prime} \sim \rho_ {t}} {\mathbb {E}} \left[ \left(w - w ^ {\prime}\right) ^ {2} \mid w, w ^ {\prime} \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right] \right] \tag {D.180}
$$
where the last inequality is by Proposition D.40. By Lemma D.21,
$$
\mathbb {P} _ {w, w ^ {\prime} \sim \rho_ {t}} \left(\left| \Phi (w) - \Phi \left(w ^ {\prime}\right) \right| \gtrsim \xi \kappa^ {3}\right) \geq 1 - O (\kappa) \tag {D.181}
$$
By Lemma D.14, if  $w, w' \in [\gamma_2^{3/4}, 1 - \gamma_2^{1/2}]$ , then
$$
| \Phi (w) - \Phi \left(w ^ {\prime}\right) | \leq \frac {1}{\gamma_ {2} ^ {3 / 2}} | w - w ^ {\prime} | \tag {D.182}
$$
Thus, combining the previous two equations gives
$$
\mathbb {P} _ {w, w ^ {\prime} \sim \rho_ {t}} \left(\left| w - w ^ {\prime} \right| <   \gamma_ {2} ^ {3 / 2} \xi \kappa^ {3} \mid w, w ^ {\prime} \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right]\right) \tag {D.183}
$$
$$
\lesssim \mathbb {P} _ {w, w ^ {\prime} \sim \rho_ {t}} \left(\left| \Phi (w) - \Phi \left(w ^ {\prime}\right) \right| <   \xi \kappa^ {3} \mid w, w ^ {\prime} \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right]\right) \tag {D.184}
$$
$$
\lesssim \frac {\mathbb {P} _ {w , w ^ {\prime} \sim \rho_ {t}} \left(\left| \Phi (w) - \Phi \left(w ^ {\prime}\right) \right| <   \xi \kappa^ {3} \text {a n d} w , w ^ {\prime} \in \left[ \gamma_ {2} ^ {3 / 4} , 1 - \gamma_ {2} ^ {1 / 2} \right]\right)}{\mathbb {P} _ {w , w ^ {\prime} \sim \rho_ {t}} \left(w , w ^ {\prime} \in \left[ \gamma_ {2} ^ {3 / 4} , 1 - \gamma_ {2} ^ {1 / 2} \right]\right)} \tag {D.185}
$$
$$
\lesssim \frac {\kappa}{\mathbb {P} _ {w , w ^ {\prime} \sim \rho_ {t}} \left(w , w ^ {\prime} \in \left[ \gamma_ {2} ^ {3 / 4} , 1 - \gamma_ {2} ^ {1 / 2} \right]\right)} \quad (\text {B y})
$$
$$
\lesssim \frac {\kappa}{\gamma_ {2}} \quad (\text {B y}
$$
and therefore, by Assumption 3.2, since  $\kappa / \gamma_2 \lesssim \frac{1}{\gamma_2 \log \log d}$ , this is at most  $\frac{1}{2}$ , since  $d$  is sufficiently large. Thus,
$$
\begin{array}{l} \underset {w, w ^ {\prime} \sim \rho_ {t}} {\mathbb {E}} \left[ \left(w - w ^ {\prime}\right) ^ {2} \mid w, w ^ {\prime} \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right] \right] (D.186) \\ \gtrsim \mathbb {P} _ {w, w ^ {\prime} \sim \rho_ {t}} \left(\left| w - w ^ {\prime} \right| \geq \gamma_ {2} ^ {3 / 2} \xi \kappa^ {3} \mid w, w ^ {\prime} \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right]\right) \cdot \gamma_ {2} ^ {3} \xi^ {2} \kappa^ {6} (D.187) \\ \gtrsim \gamma_ {2} ^ {3} \xi^ {2} \kappa^ {6} (D.188) \\ \end{array}
$$
where the last inequality is because  $\mathbb{P}_{w,w^{\prime}\sim \rho_t}(|w - w^{\prime}| <   \gamma_2^{3 / 2}\xi \kappa^3\mid w,w^{\prime}\in [\gamma_2^{3 / 4},1 - \gamma_2^{1 / 2}])\leq \frac{1}{2}$  By Eq. (D.178), we have
$$
\begin{array}{l} \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ \left| v _ {t} (w) \right| ^ {2} \mid w \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right] \right] \gtrsim \gamma_ {2} \cdot \left(\hat {\sigma} _ {4, d} ^ {4} D _ {4, t} ^ {2} \gamma_ {2} ^ {3} \underset {w \sim \rho_ {t}} {\mathbb {E}} \left[ (w - r) ^ {2} \mid w \in \left[ \gamma_ {2} ^ {3 / 4}, 1 - \gamma_ {2} ^ {1 / 2} \right] \right] \right. (D.189) \\ \left. - O \left(\frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) L \left(\rho_ {t}\right)}{d ^ {2}}\right)\right) (D.190) \\ \begin{array}{l} \gtrsim \gamma_ {2} \cdot \left(\hat {\sigma} _ {4, d} ^ {4} D _ {4, t} ^ {2} \gamma_ {2} ^ {3} \cdot \gamma_ {2} ^ {3} \xi^ {2} \kappa^ {6} - O \left(\frac {(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}) L (\rho_ {t})}{d}\right)\right) \\ \text {(B y E q . (D . 1 8 0) a n d E q . (D . 1 8 8))} \end{array} \\ \end{array}
$$
Since  $\mathbb{P}_{w\sim \rho_t}(w\in [\gamma_2^{3 / 4},1 - \gamma_2^{1 / 2}])\geq \frac{\gamma_2}{4}$  by Lemma D.17 and the symmetry of  $\rho_{t}$ , this implies that
$$
\begin{array}{l} \underset {w \sim \rho_ {t}} {\mathbb {E}} | v _ {t} (w) | ^ {2} \gtrsim \gamma_ {2} ^ {2} \cdot \left(\hat {\sigma} _ {4, d} ^ {4} D _ {4, t} ^ {2} \gamma_ {2} ^ {6} \xi^ {2} \kappa^ {6} - O \left(\frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) L \left(\rho_ {t}\right)}{d}\right)\right) \tag {D.191} \\ \gtrsim \gamma_ {2} ^ {2} \cdot \left(\hat {\sigma} _ {4, d} ^ {2} \xi^ {2} L (\rho_ {t}) \gamma_ {2} ^ {6} \xi^ {2} \kappa^ {6} - O \left(\frac {(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}) L (\rho_ {t})}{d}\right)\right) \\ \end{array}
$$
(B.c.  $\hat{\sigma}_{4,d}^2\gtrsim \hat{\sigma}_{2,d}^2$  (Assumption 3.2) and  $|D_{4,t}|\geq \xi |D_{2,t}|$  implies  $\hat{\sigma}_{4,d}^{2}D_{4,t}^{2}\gtrsim \xi^{2}L(\rho_{t})$
$$
\begin{array}{l} \gtrsim \gamma_ {2} ^ {2} \cdot \left(\hat {\sigma} _ {4, d} ^ {2} \gamma_ {2} ^ {6} \xi^ {4} \kappa^ {6} - O \left(\frac {\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}}{d}\right)\right) L \left(\rho_ {t}\right) \tag {D.192} \\ \gtrsim \hat {\sigma} _ {4, d} ^ {2} \gamma_ {2} ^ {8} \xi^ {4} \kappa^ {6} L (\rho_ {t}) \quad \text {(B e c a u s e} d \geq c _ {3} \text {b y A s s u m p t i o n 3 . 2)} \\ \end{array}
$$
Thus, by Lemma D.39,  $\frac{dL(\rho_t)}{dt} \lesssim -\hat{\sigma}_{4,d}^2 \gamma_2^8 \xi^4 \kappa^6 L(\rho_t)$ . By a similar argument as for the subcase where  $|D_{4,t}| \leq \xi |D_{2,t}|$ , since  $L(\rho_0) \lesssim \hat{\sigma}_{2,d}^2 \gamma_2^2$  (by Lemma D.8), if we define
$$
I _ {\left| D _ {4} \right| \geq \xi \mid D _ {2} \mid} = \{t \leq T _ {*, \epsilon} \text {a n d} t \text {i n P h a s e 3 , C a s e 1} \mid \left| D _ {4, t} \right| \geq \xi \left| D _ {2, t} \right| \} \tag {D.193}
$$
$$
\int_ {I _ {| D _ {4} | \geq \xi | D _ {2} |}} 1 d t \lesssim \frac {1}{\hat {\sigma} _ {4 , d} ^ {2} \gamma_ {2} ^ {8} \xi^ {4} \kappa^ {6}} \log \left(\frac {\gamma_ {2}}{\epsilon}\right) \tag {D.194}
$$
since  $L(\rho_{T_{*,\epsilon}}) \gtrsim \hat{\sigma}_{2,d}^{2}\epsilon^{2}$ . Combining Eq. (D.173) and Eq. (D.194), we find that the total time spent in Phase 3, Case 1 is at most  $O\left(\frac{1}{\hat{\sigma}_{4,d}^{2}\gamma_{2}^{8}\xi^{4}\kappa^{6}}\log \left(\frac{\gamma_{2}}{\epsilon}\right)\right)$ , as desired.
D.6 Phase 3, Case 2
Next, we analyze Phase 3, Case 2, where  $D_{4,t_2} = 0$  and  $D_{2,T_2} < 0$ . In this case, as we will show, for all  $t \geq T_2$ , we will have  $D_{4,t} \geq 0$  and  $D_{2,t} \leq 0$  (Lemma D.28). We will first give an outline of our analysis in this case — our goal is again to show that either  $|D_{2,t}| \leq \epsilon$  or  $|D_{4,t}| \leq \epsilon$  within  $\frac{1}{\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2} \cdot \mathrm{poly}\left(\frac{\log\log d}{\epsilon}\right)$  time after the start of Phase 3, Case 2.
Proof Strategy for Phase 3, Case 2 We again show that  $|v(w)|$  is large on average and then use Lemma D.39 to show that the loss decreases quickly. However, our argument to show that  $|v(w)|$  is large on average is significantly different from Phase 3, Case 1. We can approximately write
$$
\begin{array}{l} v _ {t} (w) = - \left(1 - w ^ {2}\right) \left(P _ {t} (w) + Q _ {t} (w)\right) (D.195) \\ \approx - \left(1 - w ^ {2}\right) P _ {t} (w) (D.196) \\ \approx - \left(1 - w ^ {2}\right) w \left(2 \hat {\sigma} _ {2, d} ^ {2} D _ {2, t} + 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} w ^ {2}\right) (D.197) \\ \approx (1 - w ^ {2}) w \left(2 \hat {\sigma} _ {2, d} ^ {2} \left| D _ {2, t} \right| - 4 \hat {\sigma} _ {4, d} ^ {2} \left| D _ {4, t} \right| w ^ {2}\right) \quad (\mathrm {B . c .} D _ {2, t} \leq 0 \text {a n d} D _ {4, t} \geq 0) \\ \approx - (1 - w ^ {2}) w \left(4 \hat {\sigma} _ {4, d} ^ {2} | D _ {4, t} | w ^ {2} - 2 \hat {\sigma} _ {2, d} ^ {2} | D _ {2, t} |\right) (D.198) \\ \end{array}
$$
Letting  $r = \sqrt{\frac{\hat{\sigma}_{2,d}^2|D_{2,t}|}{2\hat{\sigma}_{4,d}^2|D_{4,t}|}}$  be the positive root of  $P_{t}(w)$ , we obtain
$$
v _ {t} (w) \approx - 4 \hat {\sigma} _ {4, d} ^ {2} | D _ {4, t} | (w - r) (w + r) w \left(1 - w ^ {2}\right) \tag {D.199}
$$
This time,  $v_{t}(w)$  has a negative slope at  $r$ , meaning that the particles are attracted towards  $r$ , and we cannot use the same argument as in Phase 3, Case 1.
Instead, the key point of our argument in this case is the following. Recall that  $D_{4,t} \geq 0$  and  $D_{2,t} \leq 0$  for all  $t \geq T_2$  — these roughly imply that  $\mathbb{E}_{w \sim \rho_t}[w^4] \geq \gamma_4$  and  $\mathbb{E}_{w \sim \rho_t}[w^2] \leq \gamma_2$ . By Assumption 3.2, we have  $\gamma_4 \geq 1.1\gamma_2^2$  — in other words,  $\mathbb{E}_{w \sim \rho_t}[w^4] \geq 1.1(\mathbb{E}_{w \sim \rho_t}[w^2])^2$ . From this observation, we know that a significant fraction of the particles must be far from  $r$  — otherwise, if all of the particles are concentrated at  $r$ , then we would have  $\mathbb{E}_{w \sim \rho_t}[w^4] \approx (\mathbb{E}_{w \sim \rho_t}[w^2])^2$ , which would give a contradiction.
One complication in showing that the velocity is large is that, a priori, nearly all of the particles could be close to 0 or 1 — their velocities could be small even if they are far from  $r$ . We make use of the potential function  $\Phi$  defined in the analysis of Phase 3, Case 1 to avoid this issue. In this case, it turns out that for any two particles  $w, w'$ , their potential difference  $|\Phi(w) - \Phi(w')|$  is decreasing (Lemma D.23). We can use this to show that a  $1 - O(\gamma_4)$  fraction of particles  $w$  with initializations in  $[\iota_{\mathrm{L}}, \iota_{\mathrm{M}}]$  (where  $\iota_{\mathrm{M}}$  is defined later) cannot become too close to 0 or 1 in Lemma D.24 and Lemma D.25. Intuitively, this is because if  $|\Phi(w) - \Phi(w')|$  is bounded by  $\log B$  for two particles  $w, w' > 0$ , and  $C$  is a sufficiently large constant, then  $w \leq \left( \frac{1}{\log \log d} \right)^C$  roughly implies that  $w' \lesssim B \left( \frac{1}{\log \log d} \right)^C$ , simply by the definition  $\Phi(w) = \log \left( \frac{w}{\sqrt{1 - w^2}} \right)$  together with some algebraic manipulations. Moreover, if all of the particles  $w'$  in this group are at most  $B \left( \frac{1}{\log \log d} \right)^C$ , then this implies that  $D_2$  has a very large negative value, and this implies that all particles  $w$  have a positive velocity (as argued in Lemma D.25), meaning that the particles  $w$  in this group cannot become smaller than  $B \left( \frac{1}{\log \log d} \right)^C$ . We can show by a similar argument that the particles having initializations in  $[\iota_{\mathrm{L}}, \iota_{\mathrm{M}}]$  cannot become too close to 1.
As long as the positive root  $r$  of  $P_{t}(w)$  is at least  $\frac{1}{2}$ , this group of particles will have a large velocity as argued in Lemma D.29, since a significant fraction of these particles must be at most  $\frac{1}{3}$ . We encounter a potential issue when the positive root  $r$  is at most  $\frac{1}{2}$ : it is possible that the particles initialized in  $[\iota_{\mathrm{L}}, \iota_{\mathrm{M}}]$  get stuck at  $r$ . We mentioned above that by the constraints  $D_{2,t} < 0$  and  $D_{4,t} > 0$ , there must always be some mass away from  $r$ . However, this mass may only be a  $\gamma_4^{5/4}$  fraction of the total mass, while the particles initialized at  $[\iota_{\mathrm{L}}, \iota_{\mathrm{M}}]$  only have a  $1 - O(\gamma_4)$  fraction of the total mass. Thus, once  $r \leq \frac{1}{2}$ , it is possible for the mass located far away from  $r$  to be disjoint from the set of particles initialized at  $[\iota_{\mathrm{L}}, \iota_{\mathrm{M}}]$ . To resolve this, we argue in Lemma D.31 that the particles larger than  $w_{t}(\iota_{\mathrm{M}})$  (which may be close to 1) will move away from 1 at an exponentially fast rate while  $r \leq \frac{1}{2}$ . Once these particles, which are larger than  $w_{t}(\iota_{\mathrm{M}})$ , are sufficiently far away from 1, they will also contribute a significant amount to the velocity. Thus, we can argue as discussed above that there must be a significant fraction of particles away from  $r$  due to the constraints  $D_{2,t} \leq 0$  and  $D_{4,t} \geq 0$  (Lemma D.32).
We now begin the formal proof. We first show that as long as  $D_{4,t} \geq 0$ , the potential difference between any two particles  $w, w'$  is increasing. In particular, this holds during Phase 3, Case 2 (but note that this lemma is no longer applicable during Phase 1 or Phase 2).
Lemma D.23 ( $\Phi(w)$  and  $\Phi(w')$  Move Closer in Phase 3 Case 2). Suppose we are in the setting of Theorem 3.3, and suppose  $D_{4,t} \geq 0$  for all  $t$  in some time interval  $[t_0, t_1]$ . Then, for all  $\iota, \iota'$ ,
$$
\left| \Phi \left(w _ {t} (\iota)\right) - \Phi \left(w _ {t} \left(\iota^ {\prime}\right)\right) \right| \tag {D.200}
$$
is decreasing as a function of  $t$  on the interval  $[t_0, t_1]$ .
Proof of Lemma D.23. We mostly follow the proof of Lemma D.13. Suppose  $\iota, \iota' \in [0,1]$  such that  $\iota \leq \iota'$ . Then,  $w_{t}(\iota') \geq w_{t}(\iota)$  for all times  $t$  (by Proposition D.37), and since  $\Phi$  is an increasing function,  $\Phi(w_{t}(\iota')) \geq \Phi(w_{t}(\iota))$  for all times  $t$ . Thus, it suffices to show that  $\Phi(w_{t}(\iota')) - \Phi(w_{t}(\iota))$  is decreasing. For convenience, we write  $w_{t}' = w_{t}(\iota')$  and  $w_{t} = w_{t}(\iota)$ . We do a computation similar
to that in the proof of Lemma D.13:
$$
\frac {d}{d t} \left(\Phi \left(w _ {t} ^ {\prime}\right) - \Phi \left(w _ {t}\right)\right) = \left(- 2 \hat {\sigma} _ {2, d} ^ {2} D _ {2, t} - 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} \left(w _ {t} ^ {\prime}\right) ^ {2} - \lambda_ {d} ^ {(1)} - \lambda_ {d} ^ {(3)} \left(w _ {t} ^ {\prime}\right) ^ {2}\right) \tag {D.201}
$$
$$
- \left(- 2 \hat {\sigma} _ {2, d} ^ {2} D _ {2, t} - 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} w _ {t} ^ {2} - \lambda_ {d} ^ {(1)} - \lambda_ {d} ^ {(3)} w _ {t} ^ {2}\right)
$$
(By Lemma D.12)
$$
= - 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} \left(\left(w _ {t} ^ {\prime}\right) ^ {2} - w _ {t} ^ {2}\right) - \lambda_ {d} ^ {(3)} \left(\left(w _ {t} ^ {\prime}\right) ^ {2} - w _ {t} ^ {2}\right) \tag {D.202}
$$
According to the statement of Lemma 4.4,  $\lambda_d^{(3)} = 4\hat{\sigma}_{4,d}^2 D_{4,t}\cdot \frac{6d + 9}{d^2 - 1}$ , meaning that
$$
\frac {d}{d t} \left(\Phi \left(w _ {t} ^ {\prime}\right) - \Phi \left(w _ {t}\right)\right) = - 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} \left(\left(w _ {t} ^ {\prime}\right) ^ {2} - w _ {t} ^ {2}\right) - 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} \cdot \frac {6 d + 9}{d ^ {2} - 1} \left(\left(w _ {t} ^ {\prime}\right) ^ {2} - w _ {t} ^ {2}\right) \tag {D.203}
$$
$$
= - 4 \hat {\sigma} _ {4, d} ^ {2} D _ {4, t} \cdot \left(1 - \frac {6 d + 9}{d ^ {2} - 1}\right) \cdot \left(\left(w _ {t} ^ {\prime}\right) ^ {2} - w _ {t} ^ {2}\right) \tag {D.204}
$$
$$
<   0 \quad (\mathrm {B . c .} D _ {4, t} \geq 0 \text {a n d} w _ {t} ^ {\prime} > w _ {t} > 0, \text {a n d} d \geq c _ {3} (\text {A s s u m p t i o n} 3. 2))
$$
Thus, on any interval  $[t_0,t_1]$  where  $D_{4,t}\geq 0$ ,  $\Phi (w_t^{\prime}) - \Phi (w_t)$  is decreasing, as desired.
In the next lemma, we show an upper bound on  $|\Phi(w) - \Phi(w')|$ , at the beginning of Phase 3, Case 2, for particles  $w, w'$  which are respectively initialized at  $\iota, \iota' \in [\iota_{\mathrm{L}}, \iota_{\mathrm{M}}]$ . Here  $\iota_{\mathrm{M}}$  is defined in the statement/proof of the following lemma.
Lemma D.24. Suppose we are in the setting of Theorem 3.3. Assume that  $D_{4,T_2} = 0$  and  $D_{2,T_2} < 0$ , i.e. we are in Phase 3, Case 2. Then, there exists  $\iota_{\mathrm{M}} \in [\iota_{\mathrm{L}}, \iota_{\mathrm{R}}]$  such that  $\mathbb{P}_{\iota \sim \rho_0}(|\iota| > \iota_{\mathrm{M}}) \lesssim \gamma_4$  and for all  $\iota, \iota' \in [\iota_{\mathrm{L}}, \iota_{\mathrm{M}}]$ ,