SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finally, the claim (29) follows from (30) since  $\operatorname{sgn}(\bar{\pmb{g}}(\pmb{\theta}^k)) = \operatorname{sgn}(\underline{\pmb{g}}(\pmb{\theta}^k)) = \operatorname{sgn}(\pmb{g}(\pmb{\theta}^k))$  if we take  $\rho$  small enough.
First, we show that the weights must move significantly by time roughly  $T_{k + 1}$ . This is because of the contribution of coordinate  $i_k$ .
Lemma C.7 ( $\underline{t}_{k+1}$  is not much larger than  $T_{k+1}$ ). Suppose that  $T_{k+1} < \infty$ . Then there are  $\rho_{k+1}(\underline{\epsilon}_{k+1}) > 0$  and  $\underline{\alpha}_{k+1}(\rho, \underline{\epsilon}_{k+1}) > 0$  such that for all  $\rho < \rho_{k+1}$  and  $\alpha < \underline{\alpha}_{k+1}$ , the following holds.
$$
\underline {{t}} _ {k + 1} <   T _ {k + 1} + \underline {{\epsilon}} _ {k + 1}.
$$
Proof. Assume by contradiction that  $\underline{t}_{k+1} < T_{k+1} + \underline{\epsilon}_{k+1}$ . For all times  $t \in [\bar{t}_k, \min(\underline{t}_{k+1}, T_{k+1} + \underline{\epsilon}_{k+1})]$ , by Lemma C.6,
$$
\left| \log_ {\alpha} \left(w _ {\alpha , i _ {k}} (t)\right) - b _ {i _ {k}} ^ {t} + \left(t - \bar {t} _ {k}\right) g _ {i _ {k}} \left(\boldsymbol {\theta} ^ {k}\right) \right| <   O (\sqrt {\rho}).
$$
Since we know  $|\Delta_k(i_k) - (T_{k + 1} - \bar{t}_k)| <   \bar{\epsilon}_k$  and  $b_i^k -\Delta_k(i_k)g_{i_k}(\pmb {\theta}^k)\in \{0,2\}$  , it follows that
$$
\log_ {\alpha} \left(w _ {\alpha , i _ {k}} \left(T _ {k + 1} + \underline {{\epsilon}} _ {k + 1}\right)\right) \notin (- | g _ {i _ {k}} (\boldsymbol {\theta} ^ {k}) | (\underline {{\epsilon}} _ {k + 1} - \bar {\epsilon} _ {k + 1}), 2 + | g _ {i _ {k}} (\boldsymbol {\theta} ^ {k}) | (\underline {{\epsilon}} _ {k + 1} - \bar {\epsilon} _ {k + 1})) + O (\sqrt {\rho}).
$$
By taking  $\rho$  small enough, we see that  $|g_{i_k}(\pmb{\theta}^k)| (\underline{\epsilon}_{k+1} - \bar{\epsilon}_{k+1}) + O(\sqrt{\rho}) > \delta > 0$  for some  $\delta > 0$  that is independent of  $\alpha$ , so
$$
\log_ {\alpha} \left(w _ {\alpha , i _ {k}} \left(T _ {k + 1} + \underline {{\epsilon}} _ {k + 1}\right)\right) \notin (- \delta , 2 + \delta).
$$
So  $|u_{\alpha, i_k}(T_{k+1} + \underline{\epsilon}_{k+1})| > 1$  by Lemma D.2. But by the construction of  $\underline{t}_{k+1}$  this means that  $\underline{t}_{k+1} < T_{k+1} + \underline{\epsilon}_{k+1}$ .
Next, we show that until time  $\underline{t}_{k+1}$ , none of the coordinates in  $S_k^c$  move significantly, with the possible exception of coordinate  $i_k$ .
Lemma C.8 (No coordinates in  $S_{k}^{c} \setminus \{i_{k}\}$  move significantly during part (A)). Suppose  $T_{k+1} < \infty$ . Then there are  $\rho_{k+1}(\underline{\epsilon}_{k+1}) > 0$  and  $\underline{\alpha}_{k+1}(\rho, \underline{\epsilon}_{k+1}) > 0$  such that for all  $\rho < \rho_{k+1}$  and  $\alpha < \underline{\alpha}_{k+1}$ , the following holds. There is a constant  $c > 0$  depending on  $k$  such that for all  $i \in S_{k}^{c} \setminus \{i_{k}\}$  and  $t \in [\bar{t}_{k}, \underline{t}_{k+1}]$ ,
$$
\left| u _ {\alpha , i} (t) - u _ {\alpha , i} (\bar {t} _ {k}) \right|, \left| v _ {\alpha , i} (t) - v _ {\alpha , i} (\bar {t} _ {k}) \right| <   \alpha^ {c} + \bar {\epsilon} _ {k}.
$$
Proof. The previous lemma combined with the inductive hypothesis gives
$$
\underline {{t}} _ {k + 1} - \bar {t} _ {k} <   \Delta_ {k} (i _ {k}) + 2 \underline {{\epsilon}} _ {k + 1} \backslash \{i _ {k} \}.
$$
We analyze the movement of each coordinate  $i \in S_k^c \setminus \{i_k\}$  by breaking into two cases:
- Coordinate  $i \neq i_k$  such that  $b_i^k \in (0,2)$ . By Assumption 4.3, there is a unique winning coordinate so  $b_i^k - \tau g_i(\pmb{\theta}^k) \in (c,2 - c)$  for some constant  $c > 0$  for all  $\tau \in [0, t_{k+1} - \bar{t}_k] \subseteq [0, \Delta_k(i_k) + 2\epsilon_{k+1}]$ . By Lemma C.6,  $\log_\alpha(w_{\alpha,i}(t)) \in (-c/2, 2 - c/2)$  for all times  $t \in [\bar{t}_k, \underline{t}_{k+1}]$ . So by Lemma D.1,  $|u_{\alpha,i}(t)|, |v_{\alpha,i}(t)| \leq \alpha^{c/4}$ .
- Coordinate  $i \neq i_k$  such that  $b_i^k = 0$ . By Lemma D.4, we must be in the corner case where  $i \in S_{k-1} \cap S_k^c$  (i.e., the coordinate was active in the previous stage but was dropped from the support in this stage).
By Lemma D.4, since  $b_{i}^{k} = 0$  we have  $g_{i}(\pmb{\theta}^{k}) < 0$ . By Lemma C.6, this means  $\operatorname{sgn}(g_{i}(\pmb{\theta}_{\alpha}(t))) = \operatorname{sgn}(g_{i}(\pmb{\theta}^{k})) < 0$  for all  $t \in (\bar{t}_k, \underline{t}_{k+1})$ .
We break the analysis into two parts. Since  $b_{i}^{k} = 0$ , the sign is  $s_{i}^{k} = +1$ . The inductive hypothesis  $\| \theta_{\alpha}(\bar{t}_k) - \pmb{\theta}^k \| < \bar{\epsilon}_k$  implies that  $|u_{\alpha,i}(\bar{t}_k) - z_i^k| < \bar{\epsilon}_k$  and  $|v_{\alpha,i}(\bar{t}_k) - z_i^k| < \bar{\epsilon}_k$ . For small enough  $\bar{\epsilon}_k$  this means that  $\operatorname{sgn}(u_{\alpha,i}(\bar{t}_k)) = \operatorname{sgn}(v_{\alpha,i}(\bar{t}_k)) = +1$ . Now let  $t^* = \min(\underline{t}_{k+1}, \inf\{t > \bar{t}_k : v_{\alpha,i}(t) = 0\})$ . Since  $u_{\alpha,i}(t) > v_{\alpha,i}(t)$  without loss of generality (see Section C.3), we have  $\operatorname{sgn}(u_{\alpha,i}(t)) = \operatorname{sgn}(v_{\alpha,i}(t)) = +1$  for all  $t \in [\bar{t}_k, t^*]$ . So  $\frac{du_{\alpha,i}(t)}{dt}$ ,  $\frac{dv_{\alpha,i}(t)}{dt} < 0$  for all  $t \in [\bar{t}_k, t^*]$ . So, for any  $t \in [\bar{t}_k, t^*]$ ,
$$
\left| u _ {\alpha , i} (t) - u _ {\alpha , i} (\bar {t} _ {k}) \right|, \left| v _ {\alpha , i} (t) - v _ {\alpha , i} (\bar {t} _ {k}) \right| <   \bar {\epsilon} _ {k}
$$
Also, since  $\log_{\alpha}(w_{\alpha,i}(t^{*})) \approx 1$ , by Lemma C.6 we have  $t^{*} > c > 0$  for some constant  $c$  independent of  $\alpha$ . So for all  $t \in [t^{*}, \underline{t}_{k+1}]$  we have  $b_i^k - \tau g_i(\pmb{\theta}^k) \in (c, 2-c)$  for some constant  $c > 0$ . So  $|u_{\alpha,i}(t)|, |v_{\alpha,i}(t)| \leq \alpha^{c/4}$  for all  $t \in [t^{*}, \underline{t}_{k+1}]$ . The conclusion follows by triangle inequality.
- Coordinate  $i \neq i_k$  such that  $b_i^k = 2$ . The analysis is analogous to the case  $b_i^k = 0$ , except that we have  $s_i^k = -1$  instead and  $g_i(\pmb{\theta}^k) > 0$  by Lemma D.4.
Finally, we use this conclude that  $\underline{t}_{k + 1} \approx T_{k + 1}$  and that the weights at coordinate  $i_k$  are the only weights that change significantly, and by an amount approximately  $\rho$ .
Lemma C.9 (Coordinate  $i_k$  wins the part (A) race at time  $t_{k+1} \approx T_{k+1}$ ). Suppose that  $T_{k+1} < \infty$ . Then there are  $\rho_{k+1}(\underline{\epsilon}_{k+1}) > 0$  and  $\underline{\alpha}_{k+1}(\rho, \underline{\epsilon}_{k+1}) > 0$  such that for all  $\rho < \rho_{k+1}$  and  $\alpha < \underline{\alpha}_{k+1}$ , the following holds.
$$
\begin{array}{l} \left| \underline {{t}} _ {k + 1} - T _ {k + 1} \right| <   \underline {{\epsilon}} _ {k + 1}, \\ u _ {\alpha , i _ {k}} \left(\underline {{t}} _ {k + 1}\right) \in [ \rho , 3 \rho ], \\ \operatorname {s g n} (v _ {\alpha , i _ {k}} (\underline {{t}} _ {k + 1})) = s _ {i _ {k}} ^ {k + 1}. \\ \end{array}
$$
Proof. Let us analyze the case that  $b_{i_k}^k \in (0,2)$ . Notice that  $b_{i_k}^{k+1} = b_{i_k}^k - \Delta_k(i_k)g_{i_k}(\pmb{\theta}^k) \in \{0,2\}$  and that if  $b_i^{k+1} = 0$  then  $g_{i_k}(\pmb{\theta}^k) > 0$  and if it is 2 then  $b_{i_k}^{k+1} = g_{i_k}(\pmb{\theta}^k) < 0$ . So by Lemma C.6, for all times  $t \in [\bar{t}_k, \min(\underline{t}_{k+1}, T_{k+1} - \underline{\epsilon}_{k+1})]$ , we have  $w_{\alpha,i_k}(t) \in (c,2-c)$  for some  $c > 0$ . So for small enough  $\alpha$  by Lemma D.1,  $|u_{\alpha,i_k}(t)|, |v_{\alpha,i_k}(t)| \leq \alpha^{c/2}$ . Combining this with Lemma C.8, we see that for  $t \in [\bar{t}_k, \min(\underline{t}_{k+1}, T_{k+1} - \underline{\epsilon}_{k+1})]$  we have
$$
\left\| \boldsymbol {u} _ {\alpha} (t) - \boldsymbol {u} _ {\alpha} (\bar {t} _ {k}) \right\| + \left\| \boldsymbol {v} _ {\alpha} (t) - \boldsymbol {v} _ {\alpha} (\bar {t} _ {k}) \right\| <   2 \left(\alpha^ {c} + \bar {\epsilon} _ {k}\right) p <   \rho ,
$$
for small enough  $\alpha$ . So by definition of  $\underline{t}_{k+1}$  we must have  $\underline{t}_{k+1} > T_{k+1} - \underline{\epsilon}_{k+1}$ . Combined with Lemma C.7, we conclude that  $|T_{k+1} - \underline{t}_{k+1}| < \underline{\epsilon}_{k+1}$ , which is the first claim of the lemma. Furthermore, by Lemma C.8,
$$
\sum_ {i \in S _ {k} ^ {c} \backslash \{i _ {k} \}} | u _ {\alpha , i} (\underline {{t}} _ {k + 1}) - u _ {\alpha , i} (\bar {t} _ {k}) | + | v _ {\alpha , i} (\underline {{t}} _ {k + 1}) - v _ {\alpha , i} (\bar {t} _ {k}) | \leq 2 p (\alpha^ {c} + \bar {\epsilon} _ {k})) <   \rho / 2,
$$
so by definition of  $\underline{t}_{k+1}$  and triangle inequality we have  $|u_{\alpha, i_k}(\underline{t}_{k+1})| + |v_{\alpha, i_k}(\underline{t}_{k+1})| \geq 4\rho - \rho / 2 = 7\rho / 2$ . Also, since  $u_{\alpha, i_k}^2(\underline{t}_{k+1}) - v_{\alpha, i_k}^2(\underline{t}_{k+1}) = \Theta(\alpha^2)$  we have  $u_{\alpha, i_k}(\underline{t}_{k+1}) \in [\rho, 3\rho]$ . Finally, if  $b_{i_k}^{k+1} = 2$ , then  $s_{i_k}^{k+1} = -1$  and  $\log_\alpha(w_{\alpha, i_k}(\underline{t}_{k+1})) > 1.5$  so  $\operatorname{sgn}(v_{\alpha, i_k}(t)) < 0$  by Lemma D.3; analogously, if  $b_{i_k}^{k+1} = 0$ , we have  $s_{i_k}^{k+1} = 1$  and  $\log_\alpha(w_{\alpha, i_k}(\underline{t}_{k+1}) < 0.5$  so  $\operatorname{sgn}(v_{\alpha, i_k}(\underline{t}_{k+1}) > 0$ .
The case  $b_{i_k}^k \in \{0,2\}$  can be proved similarly to the analysis in Lemma C.8, where one shows that during the first period of time the magnitudes of  $|u_{i_k}(t)|$  and  $|v_{i_k}(t)|$  decrease, until the sign of  $v_{i_k}$  flips and they once again increase.
We have shown the claims (21), (22), (23) (24), and (25) for the time  $\underline{t}_{k+1}$ . In fact, if we let  $\underline{t}_{k+1}' \in [\bar{t}_k, \infty)$  be the first time  $t$  such that  $u_{\alpha, i_k}(t) = \rho$  we still have (21), (22), (23) and (25) by the same analysis as above, and (24) can be replaced with the slightly more convenient
$$
u _ {\alpha , i _ {k}} \left(\underline {{t}} _ {k + 1} ^ {\prime}\right) = \rho .
$$
C.6.2 Analysis in case where  $T_{k + 1} = \infty$
In this case that  $T_{k + 1}$ , we just have to show that the weights remain close to  $\theta^k$ . We show that for any  $\underline{\epsilon}_{k + 1} > 0$ , there is  $\underline{\alpha}_{k + 1}(\underline{\epsilon}_{k + 1}) > 0$  such that for all  $\alpha < \underline{\alpha}_{k + 1}$  and times  $t \in [T_k + \underline{\epsilon}_{k + 1}, T^*]$
$$
\left\| \boldsymbol {\theta} _ {\alpha} (t) - \boldsymbol {\theta} ^ {k} \right\| <   \underline {{\epsilon}} _ {k + 1}.
$$
We can use Lemmas C.5 and C.6, which were developed for the case of  $T_{k+1} < \infty$ , but still hold for  $T_{k+1} = \infty$ . Lemma C.5 guarantees that the weights do not move much until time  $\underline{t}_{k+1}$ , and so we only need to show that  $\underline{t}_{k+1} \geq T^*$  when we take  $\rho$  small enough. For this, observe that  $g_i(\pmb{\theta}^k) = 0$  for all  $i \notin S_k$ , because otherwise  $T_{k+1} < \infty$ . Therefore Lemma C.6 guarantees that until time  $\min(T_*, t_{k+1})$  all weights are close to the original on the logarithmic scale. Namely,
$$
\left\| \log_ {\alpha} \left(\boldsymbol {w} _ {\alpha} (t)\right) - \boldsymbol {b} ^ {k} \right\| <   \rho \underline {{\epsilon}} _ {k + 1} + C \rho^ {\prime} \left(T ^ {*} - \bar {t} _ {k}\right)
$$
Furthermore, by the non-degeneracy Assumption 4.3 we know that  $b_{i}^{k}\in (0,2)$  for all  $i\notin S_k$  by Lemma D.4. So if we take  $\rho$  small enough and  $\underline{\alpha}_{k + 1}$  small enough, we must have that  $t_{k + 1}\geq T^*$ .
C.7 Dynamics from time  $\underline{t}_k$  to time  $\bar{t}_k$  (Nonlinear evolution for  $O(1)$  unrescaled time)
Suppose that we know for some  $k \leq K$  that for any  $\underline{\epsilon}_k > 0$ , there is  $\rho_k(\underline{\epsilon}_k) > 0$  such that for all  $\rho < \rho_k$  there is  $\underline{\alpha}_k(\rho, \underline{\epsilon}_k) > 0$  such that for all  $\alpha < \underline{\alpha}_k$ , there is a time  $t_k = t_k(\alpha, \rho, \underline{\epsilon}_k)$  satisfying
$$
\left| T _ {k} - \underline {{t}} _ {k} \right| <   \underline {{\epsilon}} _ {k} \tag {31}
$$
$$
\left\| \boldsymbol {\theta} _ {\alpha} \left(\underline {{t}} _ {k}\right) - \boldsymbol {\theta} ^ {k - 1} \right\| <   \underline {{\epsilon}} _ {k} \tag {32}
$$
$$
\left\| \log_ {\alpha} \left(\boldsymbol {w} _ {\alpha} \left(\underline {{t}} _ {k}\right)\right) - \boldsymbol {b} ^ {k} \right\| <   \underline {{\epsilon}} _ {k} \tag {33}
$$
$$
u _ {\alpha , i _ {k - 1}} \left(\underline {{t}} _ {k}\right) = \rho , \tag {34}
$$
$$
\operatorname {s g n} \left(v _ {\alpha , i _ {k - 1}} \left(\underline {{t}} _ {k}\right)\right) = s _ {i _ {k - 1}} ^ {k}. \tag {35}
$$
Now we will show that for any  $\bar{\epsilon}_k > 0$ , there is  $\bar{\alpha}_k = \bar{\alpha}_k(\bar{\epsilon}_k) > 0$  such that for all  $0 < \alpha < \bar{\alpha}_k$ , there is a time  $\bar{t}_k = \bar{t}_k(\alpha, \bar{\epsilon}_k)$  satisfying
$$
\left| T _ {k} - \bar {t} _ {k} \right| <   \bar {\epsilon} _ {k} \tag {36}
$$
$$
\left\| \boldsymbol {\theta} _ {\alpha} \left(\bar {t} _ {k}\right) - \boldsymbol {\theta} ^ {k} \right\| <   \bar {\epsilon} _ {k} \tag {37}
$$
$$
\left\| \log_ {\alpha} \left(\boldsymbol {w} _ {\alpha} \left(\bar {t} _ {k}\right)\right) - \boldsymbol {b} ^ {k} \right\| <   \bar {\epsilon} _ {k} \tag {38}
$$
We give the construction for  $\bar{t}_k$ . For any desired accuracy  $\bar{\epsilon}_k > 0$  in this stage, we will construct an accuracy  $\underline{\epsilon}_k = \underline{\epsilon}_k(\bar{\epsilon}_k) = \bar{\epsilon}_k / 3 > 0$ . We will also construct a  $\rho = \rho (\underline{\epsilon}_k) > 0$  which is sufficiently small, and we will construct an cutoff for  $\alpha$  equal to  $\bar{\alpha}_k = \bar{\alpha}_{k + 1}(\bar{\epsilon}_k) > 0$  which satisfies  $\bar{\alpha}_k < \underline{\alpha}_k(\rho ,\underline{\epsilon}_k)$ . The values for these parameters  $\underline{\epsilon}_k$  and  $\rho$  and  $\bar{\alpha}_k$  will be chosen in the following lemma, and will depend only on  $\bar{\epsilon}_k$ .
Lemma C.10 (New local minimum reached in time  $O(1 / \log(1 / \alpha))$ ). For any  $\bar{\epsilon}_k > 0$ , we can choose  $\bar{\alpha}_k = \bar{\alpha}_k(\bar{\epsilon}_k) > 0$  small enough so that, for any  $0 < \alpha < \bar{\alpha}_k$ , there is  $\bar{t}_k = \bar{t}_k(\alpha, \bar{\epsilon}_k)$  for which conditions (36) to (38) hold.
Furthermore, there is a constant  $C''$  independent of  $\alpha$  such that  $|\pmb{\theta}_{\alpha}(t)| / |\pmb{\theta}_{\alpha}(\underline{t}_k)| \in [1 / C'', C'']^{2p}$  at all times  $t \in [\underline{t}_k, \overline{t}_k]$ .
Proof. Let  $\underline{t}_k = \underline{t}_k(\alpha, \rho, \underline{\epsilon}_k)$  be given by the induction. Let us compare the dynamics starting at  $\pmb{\theta}_{\alpha}(\underline{t}_k)$  with the dynamics starting at  $\tilde{\pmb{\theta}}(\underline{t}_k) = (\tilde{\pmb{u}}(\underline{t}_k), \tilde{\pmb{v}}(\underline{t}_k))$  which is given by
$$
\tilde {u} _ {i} (\underline {{t}} _ {k}) = \left\{ \begin{array}{l l} u _ {\alpha , i} (\underline {{t}} _ {k}), & i \in S _ {k - 1} \cup \{i _ {k - 1} \} \\ 0, & \text {o t h e r w i s e} \end{array} \right. \quad \text {a n d} \quad \tilde {v} _ {i} (\underline {{t}} _ {k}) = \left\{ \begin{array}{l l} v _ {\alpha , i} (\underline {{t}} _ {k}), & i \in S _ {k - 1} \cup \{i _ {k - 1} \} \\ 0, & \text {o t h e r w i s e} \end{array} \right.
$$
and run with
$$
\frac {d \tilde {\pmb {\theta}}}{d t} = - \log (1 / \alpha) \nabla_ {\pmb {w}} \mathcal {L} (\tilde {\pmb {\theta}}).
$$
By Assumption 4.5 we know there exists a unique solution  $\tilde{\pmb{\theta}}:[\underline{t}_k,\infty)\to \mathbb{R}^p$  as long as we take  $\underline{\epsilon}_k$  small enough because  $\mathrm{supp}(\tilde{\pmb{\theta}} (\underline{t}_k)) = S_{k - 1}\cup \{i_{k - 1}\}$  and  $\| \tilde{\pmb{\theta}}_i(\underline{t}_k) - \pmb{\theta}^{k - 1}\| < \underline{\epsilon}_k$ . Furthermore, by Assumption 4.5 if we take  $\underline{\epsilon}_k$  small enough there must be a time  $\tau \coloneqq \tau (\bar{\epsilon}_k,\rho) < \infty$  such that
$$
\left\| \tilde {\boldsymbol {\theta}} (t) - \boldsymbol {\theta} ^ {k} \right\| <   \bar {\epsilon} _ {k} / 2 \text {f o r} t \geq \underline {{t}} _ {k} + \tau / \log (1 / \alpha) \tag {39}
$$
$$
\bar {t} _ {k} = \underline {{t}} _ {k} + \tau / \log (1 / \alpha).
$$
So for  $\alpha$  small enough,  $|T_{k} - \bar{t}_{k}| < 2\epsilon_{k} < \bar{\epsilon}_{k}$ , proving (36).
We now compare  $\pmb{\theta}_{\alpha}(\bar{t}_k)$  with  $\tilde{\pmb{\theta}} (\bar{t}_k)$ , and show that if we take  $\alpha$  small enough, then the dynamics of  $\tilde{\pmb{\theta}}$  closely match the dynamics of  $\pmb{\theta}_{\alpha}(t)$  for times  $\underline{t}_k + O(1 / \log (1 / \alpha))$ . The argument uses Gronwall's inequality. Let  $t^* = \inf \{t > \underline{t}_k:\| \tilde{\pmb{\theta}} (t^*) - \pmb {\theta}_\alpha (t)\| >1 / 3\}$ . For times  $t\in [\underline{t}_k,t^*)$  by Lemma D.7 we have
$$
\| \frac {d}{d t} \tilde {\pmb {\theta}} (t) - \frac {d}{d t} \pmb {\theta} _ {\alpha} (t) \| = \log (1 / \alpha) \| \nabla_ {\pmb {\theta}} \mathcal {L} (\tilde {\pmb {\theta}} (t)) - \nabla_ {\pmb {\theta}} \mathcal {L} (\pmb {\theta} _ {\alpha} (t)) \| \leq K _ {\tilde {\pmb {\theta}} (t)} \log (1 / \alpha) \| \tilde {\pmb {\theta}} (t) - \pmb {\theta} _ {\alpha} (t) \|,
$$
where  $K_{\tilde{\boldsymbol{\theta}}(t)}$  is the smoothness constant from Lemma D.7. Note that since  $\| \tilde{\boldsymbol{\theta}}(t) \| < \infty$  for large enough  $t$  by (39), the trajectory of  $\tilde{\boldsymbol{\theta}}$  must lie in a compact set. Therefore, there must be a finite set of times  $s_1, \ldots, s_m \in [\underline{t}_k, t^*)$  such that  $\cup_{t \in [\underline{t}_k, t^*)} B(\tilde{\boldsymbol{\theta}}(t), 1/2) \subseteq \cup_{i=1}^{m} B(\tilde{\boldsymbol{\theta}}(s_i), 3/4)$ . So letting  $C = \max_{i=1}^{m} K_{\tilde{\boldsymbol{\theta}}(s_i)} < \infty$  for all times  $t \in [\underline{t}_k, t^*)$  we have
$$
\frac {d}{d t} \| \tilde {\boldsymbol {\theta}} (t) - \boldsymbol {\theta} _ {\alpha} (t) \| \leq C \log (1 / \alpha) \| \tilde {\boldsymbol {\theta}} (t) - \boldsymbol {\theta} _ {\alpha} (t) \|.
$$
By Gronwall's inequality, for all times  $t \in [\underline{t}_k, t^*)$ ,
$$
\left\| \tilde {\boldsymbol {\theta}} (t) - \boldsymbol {\theta} _ {\alpha} (t) \right\| \leq \left\| \tilde {\boldsymbol {\theta}} \left(\underline {{t}} _ {k}\right) - \boldsymbol {\theta} _ {\alpha} \left(\underline {{t}} _ {k}\right) \right\| \exp \left(C \log \left(1 / \alpha\right) \left(t - \underline {{t}} _ {k}\right)\right).
$$
We know from Lemma C.8 that there is a constant  $c > 0$  such that for any small enough  $0 < \alpha < \underline{\alpha}_k$ , such that
$$
\left\| \tilde {\boldsymbol {\theta}} \left(\underline {{t}} _ {k}\right) - \boldsymbol {\theta} _ {\alpha} \left(\underline {{t}} _ {k}\right) \right\| <   \alpha^ {c}
$$
If we take  $\alpha$  small enough that  $\alpha^c\exp (C\tau) < \bar{\epsilon}_k / 2 < 1 / 3$ , we must have  $t^{*} > \underline{t}_{k} + \tau /\log (1 / \alpha)$  and so we prove (37)
$$
\left\| \boldsymbol {\theta} ^ {k} - \boldsymbol {\theta} _ {\alpha} (\bar {t} _ {k}) \right\| \leq \bar {\epsilon} _ {k} / 2 + \left\| \tilde {\boldsymbol {\theta}} (\bar {t} _ {k}) - \boldsymbol {\theta} _ {\alpha} (\bar {t} _ {k}) \right\| <   \bar {\epsilon} _ {k}.
$$
It remains to show that (38) is satisfied. Since  $\| \tilde{\pmb{\theta}}(t) - \pmb{\theta}_{\alpha}(t) \| < 1/3$  for all  $t \in [\underline{t}_k, \bar{t}_k]$ , it holds that the trajectory of  $\pmb{\theta}_{\alpha}(t)$  lies in a compact set. So by Lemma D.7 we have  $\| \pmb{g}(\pmb{\theta}_{\alpha}(t)) \| < C'$  for some constant  $C'$  at all times  $t \in [\underline{t}_k, \bar{t}_k]$ . Since  $\frac{1}{\log(1 / \alpha)} |\frac{dw_{\alpha,i}}{dt}| = |w_{\alpha,i}(t)||g_i(\pmb{w}_{\alpha}(t))| < C'|w_{\alpha,i}(t)|$ , we must have  $|w_{\alpha,i}(t)| / |w_{\alpha,i}(\underline{t}_k)| \in [1 / C'', C'']$  for some constant  $C''$  independent of  $\alpha$  and all  $t \in [\underline{t}_k, \bar{t}_k]$ . Therefore, (38) follows from (33). A similar argument shows that  $|\pmb{\theta}_{\alpha}(t) / \pmb{\theta}_{\alpha}(\underline{t}_k)| \in [1 / C'', C'']^{2p}$ .
C.8 Concluding the proof of Theorem C.4
We have shown that Theorem 4.1 is true for solutions  $\pmb{\theta}_{\alpha}:[0,T^{*}]\to \mathbb{R}^{2p}$  to the gradient flow, where  $T_{*}\in (T_{K},T_{K + 1})$ . To establish Theorem C.4 it remains only to show that for any  $T_{*}\in (T_{K},T_{K + 1})$  and small enough  $\alpha$  such a solution to the gradient flow exists and is unique. To see this, note that in the inductive proof of the invariants we construct a sequence of times  $0 = \bar{t}_0\leq \underline{t}_1\leq \bar{t}_1\leq \dots \leq \bar{t}_K\leq \underline{t}_{K + 1} > T_*$ , where we guarantee that any gradient flow solution  $\pmb{\theta}_{\alpha}:[0,\underline{t}_{k + 1}]\rightarrow \mathbb{R}^p$  satisfies  $\pmb{\theta}_{\alpha}\in \cup_{k\in \{0,\ldots ,K\}}B(\pmb{\theta}^k,1)$  for all  $t\in \cup_{k\in \{0,\ldots ,K\}}[\bar{t}_k,\underline{t}_{k + 1}]$ . And also for  $t\in \cup_{k\in \{0,\ldots ,K - 1\}}[\underline{t}_k,\bar{t}_{k + 1}]$ , we have  $\pmb{\theta}_{\alpha}(t)\in B(0,C_k^{\prime \prime}\pmb{\theta}^k)$  for some constant  $C_k^{\prime \prime}$  independent of  $\alpha$  by Lemma C.10. So  $\pmb{\theta}_{\alpha}(t)\in B(0,C_K)$  for some constant  $C_K$  at all times  $t\in [0,T^{*}]$ . By Lemma D.7, the loss gradient  $\nabla_{\pmb{\theta}}\mathcal{L}(\pmb {\theta}) = (\pmb {v}\odot \pmb {g}(\pmb {\theta}),\pmb {u}\odot \pmb {g}(\pmb {\theta}))$  is Lipschitz-continuous on the compact set  $B(0,C_K)$ . So  $\pmb{\theta}_{\alpha}:[0,T^{*}]\to \mathbb{R}^{p}$  exists and is unique by the Cauchy-Lipschitz theorem.
D Technical lemmas
D.1 Relating the sum of the weights to the original weights using the conservation law
Lemma D.1. If for some constant  $0 < c < 1$  we have  $\log_{\alpha}(w_{\alpha,i}(t)) \in (c,2 - c)$ , then for small enough  $\alpha$
$$
\max  \left(\left| u _ {\alpha , i} (t) \right|, \left| v _ {\alpha , i} (t) \right|\right) \leq \alpha^ {c / 2}.
$$
Proof. Let  $\tilde{w}_{\alpha}(t) = \pmb{u}_{\alpha}(t) - \pmb{v}_{\alpha}(t)$ . By the conservation law (5),  $w_{\alpha,i}(t)\tilde{w}_{\alpha,i}(t) = w_{\alpha,i}(0)\tilde{w}_{\alpha,i}(0) = u_{\alpha,i}(0)^2 - v_{\alpha,i}(0)^2$ . By the non-degeneracy of initialization (Assumption 4.3), the right-hand-side is  $\Theta(\alpha^2)$ . So if  $\log_{\alpha}(w_{\alpha,i}(t)) \in (c,2-c)$  then for small enough  $\alpha$ , we have  $\log_{\alpha}(|\tilde{w}_{\alpha,i}(t)|) \in (3c/4,2-3c/4)$ . So  $|u_{\alpha,i}(t)| \leq |w_{\alpha,i}(t) + \tilde{w}_{\alpha,i}(t)| \leq \alpha^{c/2}$  and  $|v_{\alpha,i}(t)| \leq |w_{\alpha,i}(t) - \tilde{w}_{\alpha,i}(t)| \leq \alpha^{c/2}$ .
Lemma D.2. If for some constant  $0 < c$  we have  $\log_{\alpha}(w_{\alpha,i}(t)) \notin (-c, 2 + c)$ , then for small enough  $\alpha$ ,
$$
\left| u _ {\alpha , i} (t) \right| > 1.
$$
Proof. Define  $\tilde{\pmb{w}}_{\alpha} = \pmb{u}_{\alpha} - \pmb{v}_{\alpha}$  as in the proof of Lemma D.1. If  $\log_{\alpha}(w_{\alpha,i}(t)) < -c$  then  $\log_{\alpha}(|\tilde{w}_{\alpha,i}(t)|) > 2 - c/2$  for small enough  $\alpha$ , so  $u_{i}(t) > \alpha^{-c} - \alpha^{2-c/2} > 1$ . Similarly, if  $\log_{\alpha}(w_{\alpha,i}(t)) > 2 + c$  then  $\log_{\alpha}(|\tilde{w}_{\alpha,i}(t)|) < -c/2$  so  $|u_{i}(\alpha)| > \alpha^{-c/2} - \alpha^{2+c} > 1$ .
Lemma D.3. If for some constant  $c > 0$ , there is a small enough  $\alpha$  such that if we have  $\log_{\alpha}(w_{\alpha,i}(t)) > 1 + c$  then  $\operatorname{sgn}(v_{\alpha,i}(t)) < 0$ . Otherwise, if  $\log_{\alpha}(w_{\alpha,i}(t)) < 1 - c$  then  $\operatorname{sgn}(v_{\alpha,i}(t)) > 0$ .