SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| f _ {i} (t) \right| \leq \sqrt {\kappa_ {1} \kappa_ {2}},
$$
$$
\left| e ^ {- y _ {i} f _ {i} (t)} - 1 \right| \leq 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}}.
$$
Proof of Lemma C.2.
First, we define the hitting time
$$
T _ {\sqrt {\kappa_ {1} \kappa_ {2}}} := \inf  \left\{t > 0: \max  _ {i \in [ n ]} | f _ {i} (t) | > \sqrt {\kappa_ {1} \kappa_ {2}} \right\}.
$$
From  $|f_i(0)| \leq m \frac{\kappa_2}{\sqrt{m}} \frac{\kappa_1}{\sqrt{m}} = \kappa_1 \kappa_2 < \sqrt{\kappa_1 \kappa_2}$  and the continuity of  $f_i(\cdot)$ , we know  $T_{\sqrt{\kappa_1 \kappa_2}} > 0$ .
Then we will prove  $T_{\mathrm{I}} = 10\sqrt{\frac{\kappa_1}{\kappa_2}} \leq T_{\sqrt{\kappa_1\kappa_2}}$ .
For any  $k \in [m]$ ,  $i \in [n]$ , and  $t \leq T_{\sqrt{\kappa_1 \kappa_2}}$ , we have the following estimates.
Recalling the definition of  $F_{k}(t)$  (8), we have
$$
\begin{array}{l} \| \boldsymbol {F} _ {k} (t) \| = \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} e ^ {- y _ {i} f _ {i} (t)} \partial^ {\circ} \sigma \left(\langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {i} \rangle\right) y _ {i} \boldsymbol {x} _ {i} \right\lvert   \leq \max  _ {i \in [ n ]} \left| e ^ {- y _ {i} f _ {i} (t)} \right\| \\ \leq e ^ {\max  _ {i \in [ n ]} | f _ {i} (t) |} \leq e ^ {\sqrt {\kappa_ {1} \kappa_ {2}}} \leq 1. 1. \\ \end{array}
$$
Recalling the dynamics (7), for any  $t \leq T_{\sqrt{\kappa_1 \kappa_2}}$ ,
$$
\frac {\mathrm {d} \rho_ {k} (t)}{\mathrm {d} t} \leq \frac {\kappa_ {2}}{\sqrt {m}} \| \langle \boldsymbol {F} _ {k} (t), \boldsymbol {w} _ {k} (t) \rangle \| \leq \frac {\kappa_ {2}}{\sqrt {m}} \| \boldsymbol {F} _ {k} (t) \| \leq \frac {1 . 1 \kappa_ {2}}{\sqrt {m}}.
$$
Then combining  $\rho_{k}(0) = \frac{\kappa_{1}}{\sqrt{m}}$ , for any  $t \leq T_{\sqrt{\kappa_1\kappa_2}}$ ,
$$
\begin{array}{l} \rho_ {k} (t) \leq \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} t}{\sqrt {m}}, \\ \left| f _ {i} (t) \right| \leq \sum_ {k = 1} ^ {m} \left| a _ {k} \right| \rho_ {k} (t) \leq m \frac {\kappa_ {2}}{\sqrt {m}} \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} t}{\sqrt {m}} \leq \kappa_ {2} \left(\kappa_ {1} + 1. 1 \kappa_ {2} t\right). \tag {9} \\ \end{array}
$$
So for any  $t \leq T_1$ , we have
$$
\left| f _ {i} (t) \right| \leq \kappa_ {2} \left(\kappa_ {1} + 1. 1 \kappa_ {2} T _ {1}\right) \leq \kappa_ {2} \left(\kappa_ {1} + 1 1 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \leq 1 2 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \leq \sqrt {\kappa_ {1} \kappa_ {2}}. \tag {10}
$$
From the definition of  $T_{\sqrt{\kappa_1 \kappa_2}}$ , we have proved  $T_{\mathrm{I}} \leq T_{\sqrt{\kappa_1 \kappa_2}}$ .
Moreover, from this proof, we know (9) holds for any  $t \leq T_1$ . Moreover, by Mean Value Theorem,
$$
\left| e ^ {- y _ {i} f _ {i} (t)} - 1 \right| \leq \left(\max  _ {z \in [ 0, \sqrt {\kappa_ {1} \kappa_ {2}} ]} e ^ {z}\right) | - y _ {i} f _ {i} (t) | \leq 1. 1 | f _ {i} (t) | \leq 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}}.
$$
Now we delve into the optimization dynamics of all neurons in Phase I in the following Section C.1 and C.2.
C.1 The Dynamics of Positive Neurons
According to the initial direction, all positive neurons  $(s_k = 1)$  can be divided into the following four classes.
$$
\begin{array}{l} [ m / 2 ] = \left\{k \in [ m / 2 ]: \boldsymbol {w} _ {k} (0) \in \mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {+} \right\} \bigcup \left\{k \in [ m / 2 ]: \boldsymbol {w} _ {k} (0) \in \mathcal {M} _ {+} ^ {-} \cap \mathcal {M} _ {-} ^ {+} \right\} \\ \bigcup \left\{k \in [ m / 2 ]: \boldsymbol {w} _ {k} (0) \in \mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {-} \right\} \bigcup \left\{k \in [ m / 2 ]: \boldsymbol {w} _ {k} (0) \in \mathcal {M} _ {+} ^ {-} \cap \mathcal {M} _ {-} ^ {-} \right\}. \\ \end{array}
$$
In the following four lemmas, we will prove the dynamics for these four classes of positive neurons. In summary, in Phase I ( $t < T_{1}$ ), some of positive neurons align well with the direction  $\mu$ , and their norms experiment a small but significant increase, while other positive neurons go dead.
Lemma C.3 (Positive,  $\mathcal{M}_+^+\cap \mathcal{M}_-^+$
For positive neuron  $k \in \{k \in [m/2] : \mathbf{w}_k(0) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^+\}$ , at the end of Phase I, it holds that
$$
\begin{array}{l} \text {(D i r e c t i o n)}. I t \text {a l i g n e d w i t h} \mu : \langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {\mu} \rangle \geq \left(1 - \mathcal {O} \left(\sqrt {\kappa_ {1} \kappa_ {2}}\right)\right) \left(1 - \mathcal {O} \left(\left(\frac {\kappa_ {1}}{\kappa_ {2}}\right) ^ {0. 5 5}\right)\right); \\ (\mathbf {N o r m}). I t h a s \text {a s m a l l b u s i n g f i c i e n t n o r m}: \rho_ {k} (T _ {\mathrm {I}}) = \Theta \left(\frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}\right). \\ \end{array}
$$
We do the following analysis for any  $k \in \{k \in [m/2] : \pmb{w}_k(0) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^+\}$ , i.e.  $s_k = 1$ ,  $\langle \pmb{w}_k(0), \pmb{x}_+ \rangle > 0$ , and  $\langle \pmb{w}_k(0), \pmb{x}_- \rangle > 0$ .
Step I. The neuron stays in  $\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{+}$  for any  $t\leq T_{\mathrm{I}}$
$$
T _ {\text {h i t}} := \inf  \left\{t \in (0, T _ {\mathrm {I}} ]: \boldsymbol {w} _ {k} (t) \notin \partial \left(\mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {+}\right) \right\},
$$
and we aim to prove that  $T_{\mathrm{hit}}$  does not exist. From the definition of  $T_{\mathrm{hit}}$  and (4), the dynamics of the neuron is:
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}\right), t \leq T _ {\mathrm {h i t}}.
$$
From  $T_{\mathrm{hit}} \leq T_{\mathrm{I}}$  and Lemma C.2, we have  $\left|e^{-y_i f_i(t)} - 1\right| \leq 1.1 \sqrt{\kappa_1 \kappa_2}$  for any  $t \leq T_{\mathrm{hit}}$ . Then we have:
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \pmb {b} _ {k} (t) , \pmb {x} _ {+} \right\rangle}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {p e ^ {- f _ {+} (t)}}{1 + p} - \frac {e ^ {f _ {-} (t)}}{1 + p} \cos \Delta \Big) \\ \geq \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {p (1 - 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}})}{1 + p} - \frac {(1 + 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}) \cos \Delta}{1 + p} \Big) > 0, \\ \end{array}
$$
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {p e ^ {- f _ {+} (t)} \cos \Delta}{1 + p} - \frac {e ^ {f _ {-} (t)}}{1 + p} \Big) \\ \geq \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {p (1 - 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}) \cos \Delta}{1 + p} - \frac {(1 + 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}})}{1 + p} \Big) > 0. \\ \end{array}
$$
Hence, for any  $t \leq T_{\mathrm{hit}}$ ,  $\langle \pmb{b}_k(t), \pmb{x}_+ \rangle > \langle \pmb{b}_k(0), \pmb{x}_+ \rangle > 0$  and  $\langle \pmb{b}_k(t), \pmb{x}_- \rangle > \langle \pmb{b}_k(0), \pmb{x}_- \rangle > 0$ . According to the definition of  $T_{\mathrm{hit}}$ , we have proved that  $T_{\mathrm{hit}}$  does not exist, which means the neuron stays in  $\mathcal{M}_{+}^{+} \cap \mathcal{M}_{-}^{+}$  for any  $t \leq T_{\mathrm{I}}$ .
Step II. Estimate the evolution of  $\langle \pmb{w}_k(t),\pmb {\mu}\rangle$
With the help of Step I, we were able to determine the dynamics for  $t \leq T_{\mathrm{I}}$ . For any  $t \leq T_{\mathrm{I}}$ , we can do the following estimate.
From (7), the tangential dynamics of the neuron is
$$
\frac {\mathrm {d} \boldsymbol {w} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m} \rho_ {k} (t)} \left(\boldsymbol {F} _ {k} (t) - \langle \boldsymbol {F} _ {k} (t), \boldsymbol {w} _ {k} (t) \rangle \boldsymbol {w} _ {k} (t)\right),
$$
where  $\pmb{F}_k(t) := \frac{p}{1 + p} e^{-f_+(t)} \pmb{x}_+ - \frac{1}{1 + p} e^{f_- (t)} \pmb{x}_-$ .
Recalling the definitions of  $\pmb{\mu}$  and  $z$ , we can estimate the difference between  $\pmb{F}_k(t)$  and  $\pmb{z}$ :
$$
\begin{array}{l} \langle \boldsymbol {F} _ {k} (t), \boldsymbol {z} \rangle = \left\langle \boldsymbol {z} + \frac {p}{1 + p} \big (e ^ {- f _ {+} (t)} - 1 \big) \boldsymbol {x} _ {+} - \frac {1}{1 + p} \big (e ^ {f _ {-} (t)} - 1 \big) \boldsymbol {x} _ {-}, \boldsymbol {z} \right\rangle \\ = \| \boldsymbol {z} \| ^ {2} + \left\langle \frac {p}{1 + p} \left(e ^ {- f _ {+} (t)} - 1\right) \boldsymbol {x} _ {+} - \frac {1}{1 + p} \left(e ^ {f _ {-} (t)} - 1\right) \boldsymbol {x} _ {-}, \frac {p}{1 + p} \boldsymbol {x} _ {+} - \frac {1}{1 + p} \boldsymbol {x} _ {-} \right\rangle \\ \geq \| \boldsymbol {z} \| ^ {2} - \frac {p ^ {2}}{(1 + p) ^ {2}} \left| e ^ {- f _ {+} (t)} - 1 \right| - \frac {1}{(1 + p) ^ {2}} \left| e ^ {f _ {-} (t)} - 1 \right| - \frac {p \cos \Delta}{(1 + p) ^ {2}} \left(\left| e ^ {- f _ {+} (t)} - 1 \right| + \left| e ^ {f _ {-} (t)} - 1 \right|\right) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {\text {L e m m a C . 2}} {\geq} \| \boldsymbol {z} \| ^ {2} - 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}} \Big (\frac {p ^ {2}}{(1 + p) ^ {2}} + \frac {1}{(1 + p) ^ {2}} + \frac {2 p \cos \Delta}{(1 + p) ^ {2}} \Big) \geq \| \boldsymbol {z} \| ^ {2} - 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}}, \\ \left\| \boldsymbol {F} _ {k} (t) - \boldsymbol {z} \right\| = \left\| \frac {p}{1 + p} \left(e ^ {- f _ {+} (t)} - 1\right) \boldsymbol {x} _ {+} - \frac {1}{1 + p} \left(e ^ {f _ {-} (t)} - 1\right) \boldsymbol {x} _ {-} \right\| \\ \leq \left\| \frac {p}{1 + p} \left(e ^ {- f _ {+} (t)} - 1\right) \boldsymbol {x} _ {+} \right\| + \left\| \frac {1}{1 + p} \left(e ^ {f _ {-} (t)} - 1\right) \boldsymbol {x} _ {-} \right\| \\ \stackrel {\text {L e m m a C . 2}} {\leq} 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}} \frac {p}{1 + p} + 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}} \frac {1}{1 + p} = 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}}. \\ \end{array}
$$
The dynamics of  $\langle \pmb{w}_k(t),\pmb {\mu}\rangle$  is:
$$
\frac {\mathrm {d} \left\langle \boldsymbol {w} _ {k} (t) , \boldsymbol {\mu} \right\rangle}{\mathrm {d} t} = \frac {1}{\| \boldsymbol {z} \|} \frac {\mathrm {d} \left\langle \boldsymbol {w} _ {k} (t) , \boldsymbol {z} \right\rangle}{\mathrm {d} t} = \frac {\kappa_ {2}}{\| \boldsymbol {z} \| \sqrt {m} \rho_ {k} (t)} \Big (\left\langle \boldsymbol {F} _ {k} (t), \boldsymbol {z} \right\rangle - \left\langle \boldsymbol {F} _ {k} (t), \boldsymbol {w} _ {k} (t) \right\rangle \left\langle \boldsymbol {w} _ {k} (t), \boldsymbol {z} \right\rangle \Big).
$$
And it can be estimated by:
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \boldsymbol {w} _ {k} (t) , \boldsymbol {\mu} \right\rangle}{\mathrm {d} t} \\ = \frac {\kappa_ {2}}{\| \boldsymbol {z} \| \sqrt {m} \rho_ {k} (t)} \Big (\langle \boldsymbol {F} _ {k} (t), \boldsymbol {z} \rangle - \langle \boldsymbol {z}, \boldsymbol {w} _ {k} (t) \rangle \langle \boldsymbol {w} _ {k} (t), \boldsymbol {z} \rangle - \langle \boldsymbol {F} _ {k} (t) - \boldsymbol {z}, \boldsymbol {w} _ {k} (t) \rangle \langle \boldsymbol {w} _ {k} (t), \boldsymbol {z} \rangle \Big) \\ \geq \frac {\kappa_ {2}}{\| \boldsymbol {z} \| \sqrt {m} \rho_ {k} (t)} \Big (\left\langle \boldsymbol {F} _ {k} (t), \boldsymbol {z} \right\rangle - \left\langle \boldsymbol {w} _ {k} (t), \boldsymbol {z} \right\rangle^ {2} - \| \boldsymbol {F} _ {k} (t) - \boldsymbol {z} \| \| \boldsymbol {z} \| \Big) \\ \geq \frac {\kappa_ {2}}{\| \boldsymbol {z} \| \sqrt {m} \rho_ {k} (t)} \left(\| \boldsymbol {z} \| ^ {2} - 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}} - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {z} \rangle^ {2} - 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}} \| \boldsymbol {z} \|\right) \\ = \frac {\kappa_ {2}}{\| \boldsymbol {z} \| \sqrt {m} \rho_ {k} (t)} \left(\| \boldsymbol {z} \| ^ {2} - \| \boldsymbol {z} \| ^ {2} \left\langle \boldsymbol {w} _ {k} (t), \boldsymbol {\mu} \right\rangle^ {2} - 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}} \big (1 + \| \boldsymbol {z} \| \big)\right) \\ = \frac {\kappa_ {2} \| \boldsymbol {z} \|}{\sqrt {m} \rho_ {k} (t)} \left(1 - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {\mu} \rangle^ {2} - \frac {1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}} \left(1 + \| \boldsymbol {z} \|\right)}{\| \boldsymbol {z} \| ^ {2}}\right) \\ \geq \frac {2 \kappa_ {2}}{3 \sqrt {m} \rho_ {k} (t)} \left(1 - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {\mu} \rangle^ {2} - 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}} \Big (1. 5 + 1. 5 ^ {2} \Big)\right) > \frac {2 \kappa_ {2}}{3 \sqrt {m} \rho_ {k} (t)} \Big (1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}} - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {\mu} \rangle^ {2} \Big) \\ \end{array}
$$
$$
\geq \frac {2 \kappa_ {2}}{3 \sqrt {m} \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} t}{\sqrt {m}}} \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}} - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {\mu} \rangle^ {2}\right) = \frac {2}{3 \left(\frac {\kappa_ {1}}{\kappa_ {2}} + 1 . 1 t\right)} \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}} - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {\mu} \rangle^ {2}\right)
$$
Noticing  $\pmb{w}_k(0) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^+$ , we have:
$$
\langle \boldsymbol {w} _ {k} (0), \boldsymbol {\mu} \rangle = \left\langle \boldsymbol {w} _ {k} (0), \frac {p}{1 + p} \boldsymbol {x} _ {+} - \frac {1}{1 + p} \boldsymbol {x} _ {-} \right\rangle > - \left\langle \boldsymbol {w} _ {k} (0), \frac {1}{1 + p} \boldsymbol {x} _ {-} \right\rangle \geq - \frac {1}{1 + p}. \tag {12}
$$
Now we consider the following auxiliary ODE:
$$
\left\{ \begin{array}{l} \frac {\mathrm {d} U (t)}{\mathrm {d} t} = \frac {2}{3 \left(\frac {\kappa_ {1}}{\kappa_ {2}} + 1 . 1 t\right)} \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}} - U ^ {2} (t)\right) \\ U (0) = - \frac {1}{1 + p} \end{array} , \right. \tag {13}
$$
and let  $U(t)$  is the solution of (13). Due to (11) (12), we know that  $\langle \pmb{w}_k(t), \pmb{\mu} \rangle$  is an upper solution of ODE (13). From the Comparison Principle of ODEs, we know this means:
$$
\langle \boldsymbol {w} _ {k} (t), \boldsymbol {\mu} \rangle > U (t), \text {f o r a n y} t \leq T _ {1}.
$$
Hence, in order to estimate  $\langle \pmb{w}_k(t),\pmb {\mu}\rangle$  , we only need to study the solution of ODE (13). It is easy to verify that the solution of (13) satisfies
$$
\log \left(\frac {1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}} + U (t)}{1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}} - U (t)}\right) - \log \left(\frac {1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}} - \frac {1}{1 + p}}{1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}} + \frac {1}{1 + p}}\right) = \frac {4 (1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}})}{3 . 3} \log \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right).
$$
$$
\begin{array}{l} \log \left(\frac {1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}} + U (t)}{1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}} - U (t)}\right) \geq \log \left(\frac {1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}} - \frac {1}{6}}{1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}} + \frac {1}{6}}\right) + \frac {4 (1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}})}{3 . 3} \log \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) \\ \geq \log \left(0. 7 \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) ^ {1. 1 5}\right) \geq \log \left(0. 7 \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) ^ {1. 1 5}\right), \\ \end{array}
$$
$$
U (t) \geq \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \left(1 - \frac {2}{1 + 0 . 7 \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) ^ {1 . 1 5}}\right).
$$
Hence, we have the estimate of  $\langle \pmb{w}_k(t),\pmb {\mu}\rangle$  ..
$$
\left\langle \boldsymbol {w} _ {k} (t), \boldsymbol {\mu} \right\rangle \geq \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \left(1 - \frac {2}{1 + 0 . 7 \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) ^ {1 . 1 5}}\right). \tag {14}
$$
Specifically, we have:
$$
\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {\mu} \rangle \geq \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \left(1 - \frac {2}{1 + 0 . 7 \left(1 + 1 1 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right).
$$
Step III. A finer estimate of  $\rho_{k}(T_{1})$
In this step, we will estimate the lower bound and upper bound for  $\rho_{k}(T_{\mathrm{I}})$
First, lemma C.2 gives us the upper bound for  $\rho_{k}(T_{1})$ :
$$
\rho_ {k} (T _ {\mathrm {I}}) \leq \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} T _ {\mathrm {I}}}{\sqrt {m}} \leq \frac {\kappa_ {1} + 1 1 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \leq \frac {1 2 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}.
$$
Now we focus on the estimate of the lower bound. Recalling the dynamics of  $\rho_{k}(t)$  (7), for any  $t\leq T_{\mathrm{I}}$
$$
\begin{array}{l} \frac {\mathrm {d} \rho_ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m}} \left\langle \boldsymbol {F} _ {k} (t), \boldsymbol {w} _ {k} (t) \right\rangle = \frac {\kappa_ {2}}{\sqrt {m}} \Big (\left\langle \boldsymbol {z}, \boldsymbol {w} _ {k} (t) \right\rangle + \left\langle \boldsymbol {F} _ {k} (t) - \boldsymbol {z}, \boldsymbol {w} _ {k} (t) \right\rangle \Big) \\ \geq \frac {\kappa_ {2}}{\sqrt {m}} \Big (\left\langle \boldsymbol {z}, \boldsymbol {w} _ {k} (t) \right\rangle - \| \boldsymbol {F} _ {k} (t) - \boldsymbol {z} \| \Big) \geq \frac {\kappa_ {2}}{\sqrt {m}} \Big (\| \boldsymbol {z} \| \left\langle \boldsymbol {w} _ {k} (t), \boldsymbol {\mu} \right\rangle - 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}} \Big) \\ \end{array}
$$
$$
\begin{array}{l} \geq \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {2 \left(1 - 4 . 2 \sqrt {\kappa_ {1} \kappa_ {2}}\right)}{3} \left(1 - \frac {2}{1 + 0 . 7 \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) ^ {1 . 1 5}}\right) - 1. 1 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \\ \geq \frac {\kappa_ {2}}{\sqrt {m}} \left(0. 6 2 7 - \frac {1 . 2 7 8}{1 + 0 . 7 \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) ^ {1 . 1 5}}\right). \\ \end{array}
$$
We denote  $T_0 = \frac{10\kappa_1}{\kappa_2}$ , and it is easy to verify:
$$
0. 6 2 7 - \frac {1 . 2 7 8}{1 + 0 . 7 \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) ^ {1 . 1 5}} > 0. 5 3, \text {f o r a n y} t \in [ T _ {0}, T _ {\mathrm {I}} ].
$$
$$
\begin{array}{l} \rho_ {k} \left(T _ {1}\right) \geq \rho_ {k} \left(T _ {0}\right) + \int_ {T _ {0}} ^ {T _ {1}} \frac {\kappa_ {2}}{\sqrt {m}} \left(0. 6 2 7 - \frac {1 . 2 7 8}{1 + 0 . 7 \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) ^ {1 . 1 5}}\right) d t \\ > 0 + \int_ {\frac {1 0 \kappa_ {1}}{\kappa_ {2}}} ^ {T _ {\mathrm {I}}} \frac {0 . 5 3 \kappa_ {2}}{\sqrt {m}} \mathrm {d} t = \frac {0 . 5 3 (1 0 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} - \frac {1 0 \kappa_ {1}}{\kappa_ {2}}) \kappa_ {2}}{\sqrt {m}} \stackrel {(6)} {\geq} \frac {5 . 3 \cdot 0 . 9 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \geq \frac {4 . 7 7 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}. \\ \end{array}
$$
Lemma C.4 (Positive,  $\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{-}$
For positive neuron  $k\in \{k\in [m / 2]:\pmb {w}_k(0)\in \mathcal{M}_+^+\cap \mathcal{M}_-^-\}$  , at the end of Phase I, it holds that
$$
\begin{array}{l} \text {(D i r e c t i o n)}. I t \text {a l i g n e d w i t h} \mu : \langle \boldsymbol {w} _ {k} (T _ {1}), \boldsymbol {\mu} \rangle \geq \left(1 - \mathcal {O} \left(\sqrt {\kappa_ {1} \kappa_ {2}}\right)\right) \left(1 - \mathcal {O} \left(\left(\frac {\kappa_ {1}}{\kappa_ {2}}\right) ^ {0. 5 5}\right)\right); \\ \textbf {(N o r m)}. I t h a s \text {a s m a l l b u s i n g i c a n t n o r m}: \rho_ {k} (T _ {1}) = \Theta \left(\frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}\right). \\ \end{array}
$$
We do the following analysis for any  $k \in \{k \in [m/2] : \pmb{w}_k(0) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^-\}$ , i.e.  $s_k = 1$ ,  $\langle \pmb{w}_k(0), \pmb{x}_+ \rangle > 0$ , and  $\langle \pmb{w}_k(0), \pmb{x}_- \rangle \leq 0$ .
Step I. The neuron must arrives in  $\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{0}$  in  $\mathcal{O}\left(\frac{\kappa_1\Delta}{\kappa_2}\right)$  time.
The case  $\langle \pmb{w}_k(0),\pmb{x}_{-}\rangle = 0$  is trivial. Then we only need to consider the case  $\langle \pmb {w}_k(0),\pmb {x}_-\rangle < 0$
$$
T _ {\text {h i t}} := \inf  \left\{t \in (0, T _ {\mathrm {I}} ]: \boldsymbol {w} _ {k} (t) \notin \mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {-} \right\},
$$
and we aim to estimate  $T_{\mathrm{hit}}$  and prove  $\pmb{w}_k(T_{\mathrm{hit}})\in \mathcal{M}_+^+\cap \mathcal{M}_-^0$
We focus on the dynamics of  $\langle \pmb {b}_k(t),\pmb{x}_+\rangle$  and  $\langle \pmb {b}_k(t),\pmb {x}_{-}\rangle$
From the definition of  $T_{\mathrm{hit}}$  and (4), the dynamics of the neuron is:
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+}, t \leq T _ {\text {h i t}}.
$$
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \right\rangle}{\mathrm {d} t} = \left\langle \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+}, \boldsymbol {x} _ {+} \right\rangle = \frac {\kappa_ {2} p}{\sqrt {m} (1 + p)} e ^ {- f _ {+} (t)},
$$
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} = \left\langle \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+}, \boldsymbol {x} _ {-} \right\rangle = \frac {\kappa_ {2} p \cos \Delta}{\sqrt {m} (1 + p)} e ^ {- f _ {+} (t)}.
$$
It is clear  $\frac{\mathrm{d}\langle\pmb{b}_k(t),\pmb{x}_+\rangle}{\mathrm{d}t} >0$  so  $\langle \pmb {b}_k(t),\pmb {x}_+\rangle >\langle \pmb {b}_k(0),\pmb {x}_+\rangle >0$  for any  $t\leq T_{\mathrm{hit}}$ . If we denote
$$
T _ {\text {h i t}, -} := \inf  \left\{t \in (0, T _ {1} ]: \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \geq 0 \right\},
$$
then it holds:
$$
T _ {\text {h i t}} = T _ {\text {h i t , -}}.
$$
So we only need to estimate  $T_{\mathrm{hit}, -}$ . Due to  $T_{\mathrm{hit}} \leq T_{\mathrm{I}} \leq T_{\mathrm{init}}$  and Lemma C.2, for any  $t \leq T_{\mathrm{hit}}$ , we have  $\left|e^{-y_i f_i(t)} - 1\right| \leq 0.11$ . Then for any  $t \leq T_{\mathrm{hit}}$ , we have:
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} \geq \frac {0 . 8 9 \kappa_ {2} p \cos \Delta}{\sqrt {m} (1 + p)}.
$$
Recalling  $\langle \pmb{w}_k(0),\pmb{x}_+ \rangle > 0$  and  $\langle \pmb{w}_k(0),\pmb{x}_- \rangle < 0$ , with the help of Lemma I.2, we have  $\langle \pmb{w}_k(0),\pmb{x}_- \rangle > -\sin \Delta$ . Combining the two estimates, we have:
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \rangle \geq \langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {-} \rangle + \int_ {0} ^ {t} \frac {0 . 8 9 \kappa_ {2} p \cos \Delta}{\sqrt {m} (1 + p)} d t \\ > - \rho_ {k} (0) \sin \Delta + \frac {0 . 8 9 \kappa_ {2} p \cos \Delta}{\sqrt {m} (1 + p)} t = - \frac {\kappa_ {1} \sin \Delta}{\sqrt {m}} + \frac {0 . 8 9 \kappa_ {2} p \cos \Delta}{\sqrt {m} (1 + p)} t. \\ \end{array}
$$
$$
T _ {\mathrm {h i t}} = T _ {\mathrm {h i t , -}} \leq \frac {(1 + p) \tan \Delta}{0 . 8 9 p} \frac {\kappa_ {1}}{\kappa_ {2}} \leq 2 \Delta \frac {\kappa_ {1}}{\kappa_ {2}}.
$$
Step II. Dynamics after arriving in the manifold  $\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{0}$
In this step, we analyze the training dynamics after  $\pmb{w}_k(T_{\mathrm{hit}}) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^0$ .
First, we will prove  $\pmb{w}_k(t)$  passes immediately from one side of the surface  $\mathcal{M}_{+}^{+} \cap \mathcal{M}_{-}^{0}$  to the other, i.e.  $\pmb{w}_k(t)$  enters into  $\mathcal{M}_{+}^{+} \cap \mathcal{M}_{-}^{+}$  at time  $T_{\mathrm{hit}}$ . Equivalently, we only need to prove  $\pmb{b}_k(t)$  passes immediately from one side of the surface  $\mathcal{P}_{+}^{+} \cap \mathcal{P}_{-}^{0}$  to the other, i.e.  $\pmb{b}_k(t)$  enters into  $\mathcal{P}_{+}^{+} \cap \mathcal{P}_{-}^{+}$  at time  $T_{\mathrm{hit}}$ .