SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For any  $\tilde{\pmb{b}}\in \mathcal{P}_{+}^{+}\cap \mathcal{P}_{-}^{0}$  and  $0 <   \delta_0\ll 1$  , we know that  $\mathcal{P}_+^+\cap \mathcal{P}_-^0$  separates its neighborhood  $\mathcal{B}(\tilde{\pmb{b}},\delta_0)$  into two domains  $\mathcal{G}_{-} = \{\pmb {b}\in \mathcal{B}(\tilde{\pmb{b}},\delta_{0}):\langle \pmb {b},\pmb{x}_{-}\rangle <  0\}$  and  $\mathcal{G}_{+} = \{\pmb {b}\in \mathcal{B}(\tilde{\pmb{b}},\delta_{0}):\langle \pmb {b},\pmb{x}_{-}\rangle >0\}$  Following Definition H.1, we calculate the limited vector field on  $\tilde{\pmb{b}}$  from  $\mathcal{G}_{-}$  and  $\mathcal{G}_{+}$
(i) The limited vector field  $F^{-}$  on  $\tilde{b}$  (from  $\mathcal{G}_{-}$ ):
$$
\frac {\mathrm {d} \boldsymbol {b}}{\mathrm {d} t} = \boldsymbol {F} ^ {-}, \text {w h e r e} \boldsymbol {F} ^ {-} = \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+}\right).
$$
(ii) The limited vector field  $F^{+}$  on  $\tilde{b}$  (from  $\mathcal{G}_{+}$ ):
$$
\frac {\mathrm {d} \boldsymbol {b}}{\mathrm {d} t} = \boldsymbol {F} ^ {+}, \text {w h e r e} \boldsymbol {F} ^ {+} = \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} \boldsymbol {x} _ {+} - \frac {e ^ {f _ {-} (t)}}{1 + p} \boldsymbol {x} _ {-}\right).
$$
(iii) Then we calculate the projections of  $\pmb{F}^{-}$  and  $\pmb{F}^{+}$  onto  $\pmb{x}_{-}$  (the normal to the surface  $\mathcal{P}_{+}^{+} \cap \mathcal{P}_{-}^{0}$ ):
$$
F _ {N} ^ {-} = \left\langle \pmb {F} ^ {-}, \pmb {x} _ {-} \right\rangle = \frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)} \cos \Delta ,
$$
$$
F _ {N} ^ {+} = \left\langle \boldsymbol {F} ^ {+}, \boldsymbol {x} _ {-} \right\rangle = \frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)} \cos \Delta - \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)}.
$$
From  $T_{\mathrm{I}} < T_{\mathrm{init}}$  and Lemma C.2, we know  $|e^{-y_i f_i(t)} - 1| \leq 0.11$ , so  $p e^{-f_+(t)} \cos \Delta - e^{f_-(t)} \geq 0.89p \cos \Delta - 1.11 > 0$ , which means  $F_N^+ > 0$ . And it is clear that  $F_N^- > 0$ . Hence, the dynamics corresponds to Case (II) in Definition H.1 ( $F_N^- > 0$  and  $F_N^+ > 0$ ).
(iv) Hence,  $\pmb{b}_k(t)$  passes immediately from one side of the surface  $\mathcal{P}_{+}^{+} \cap \mathcal{P}_{-}^{0}$  to the other, i.e.  $\pmb{b}_k(t)$  enters into  $\mathcal{P}_{+}^{+} \cap \mathcal{P}_{-}^{+}$  at time  $T_{\mathrm{hit}}$ .
Second, proceeding as in the proof of Step I~III of the Proof of Theorem C.3, we have the results:
$$
\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {\mu} \rangle \geq \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \left(1 - \frac {2}{1 + 0 . 7 (1 + 1 . 1 (T _ {\mathrm {I}} - T _ {\mathrm {h i t}})) ^ {1 . 1 5}}\right)
$$
$$
\geq \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \left(1 - \frac {2}{1 + 0 . 7 \left(1 + 1 . 1 \frac {\kappa_ {2}}{\kappa_ {1}} \left(1 0 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} - 2 \Delta \frac {\kappa_ {1}}{\kappa_ {2}}\right)\right) ^ {1 . 1 5}}\right)
$$
$$
\geq \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \left(1 - \frac {2}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right).
$$
$$
\begin{array}{l} \rho_ {k} \left(T _ {\mathrm {I}}\right) \geq \rho_ {k} \left(T _ {0}\right) + \int_ {T _ {\mathrm {h i t}} + \frac {1 0 \kappa_ {1}}{\kappa_ {2}}} ^ {T _ {\mathrm {I}}} \frac {\kappa_ {2}}{\sqrt {m}} \left(0. 6 2 7 - \frac {1 . 2 7 8}{1 + 0 . 7 \left(1 + \frac {1 . 1 \kappa_ {2}}{\kappa_ {1}} t\right) ^ {1 . 1 5}}\right) \mathrm {d} t \\ > 0 + \int_ {T _ {\mathrm {h i t}} + \frac {1 0 \kappa_ {1}}{\kappa_ {2}}} ^ {T _ {\mathrm {I}}} \frac {0 . 5 3 \kappa_ {2}}{\sqrt {m}} \mathrm {d} t = \frac {0 . 5 3 (1 0 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} - \frac {(1 0 + 2 \Delta) \kappa_ {1}}{\kappa_ {2}}) \kappa_ {2}}{\sqrt {m}} \stackrel {(6)} {\geq} \frac {5 . 3 \cdot 0 . 8 8 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \geq \frac {4 . 6 6 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}. \\ \end{array}
$$
Lemma C.5 (Positive,  $\mathcal{M}_{+}^{-}\cap \mathcal{M}_{-}^{+})$
For positive neuron  $k \in \{k \in [m/2] : \pmb{w}_k(0) \in \mathcal{M}_+^- \cap \mathcal{M}_-^+\}$ , after  $\mathcal{O}\left(\frac{\kappa_1 p \Delta}{\kappa_2}\right)$  time, it goes dead:
$$
\boldsymbol {w} _ {k} (t) \in \mathcal {M} _ {+} ^ {-} \cap \mathcal {M} _ {-} ^ {-}, f o r a n y t \geq T _ {\mathrm {I}} > \mathcal {O} \left(\frac {\kappa_ {1} p \Delta}{\kappa_ {2}}\right).
$$
Proof of Lemma C.9.
We do the following analysis for any  $k \in \{k \in [m/2] : \pmb{w}_k(0) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^-\}$ , i.e.  $\mathrm{s}_k = 1$ ,  $\langle \pmb{w}_k(0), \pmb{x}_+ \rangle > 0$ , and  $\langle \pmb{w}_k(0), \pmb{x}_- \rangle \leq 0$ .
$$
T _ {\text {h i t}} := \inf  \left\{t \in (0, T _ {\mathrm {I}} ]: \boldsymbol {w} _ {k} (t) \notin \mathcal {M} _ {+} ^ {-} \cap \mathcal {M} _ {-} ^ {+} \right\},
$$
and we aim to estimate  $T_{\mathrm{hit}}$  and prove  $\pmb{w}_k(T_{\mathrm{hit}})\in \mathcal{M}_+^-\cap \mathcal{M}_-^-$
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}, t \leq T _ {\text {h i t}}.
$$
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \right\rangle}{\mathrm {d} t} = \left\langle - \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}, \boldsymbol {x} _ {+} \right\rangle = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{1 + p} e ^ {f _ {-} (t)} \cos \Delta ,
$$
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} = \left\langle - \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}, \boldsymbol {x} _ {-} \right\rangle = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{1 + p} e ^ {f _ {-} (t)}.
$$
It is clear  $\frac{\mathrm{d}\langle\pmb{b}_k(t),\pmb{x}_+\rangle}{\mathrm{d}t} < 0$ , so  $\langle \pmb {b}_k(t),\pmb {x}_+\rangle < \langle \pmb {b}_k(0),\pmb {x}_+\rangle \leq 0$  for any  $t\leq T_{\mathrm{hit}}$ . If we denote
$$
T _ {\text {h i t}, -} := \inf  \left\{t \in (0, T _ {\mathrm {I}} ]: \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq 0 \right\},
$$
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} \leq - \frac {0 . 8 9 \kappa_ {2}}{\sqrt {m} (1 + p)}.
$$
Recalling  $\langle \pmb{w}_k(0),\pmb{x}_+ \rangle \leq 0$  and  $\langle \pmb{w}_k(0),\pmb{x}_- \rangle > 0$ , with the help of Lemma I.2, we have  $\langle \pmb{w}_k(0),\pmb{x}_- \rangle \leq \sin \Delta$ . Combining the two estimates, we have:
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq \langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {-} \rangle - \int_ {0} ^ {t} \frac {0 . 8 9 \kappa_ {2}}{\sqrt {m} (1 + p)} d t \\ \leq \rho_ {k} (0) \sin \Delta - \frac {0 . 8 9 \kappa_ {2}}{\sqrt {m} (1 + p)} t = \frac {\kappa_ {1} \sin \Delta}{\sqrt {m}} - \frac {0 . 8 9 \kappa_ {2}}{\sqrt {m} (1 + p)} t. \\ \end{array}
$$
$$
T _ {\mathrm {h i t}} = T _ {\mathrm {h i t , +}} \leq \frac {(1 + p) \sin \Delta}{0 . 8 9} \frac {\kappa_ {1}}{\kappa_ {2}} <   T _ {\mathrm {I}} = 1 0 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}}.
$$
Moreover, the analysis gives us  $\pmb{w}_k(T_{\mathrm{hit}}) \in \mathcal{M}_+^- \cap \mathcal{M}_-^-$ . By Lemma B.2, we obtain:
$$
\boldsymbol {w} _ {k} (t) \in \mathcal {M} _ {+} ^ {-} \cap \mathcal {M} _ {-} ^ {-}, \text {f o r a n y} t \geq T _ {\mathrm {h i t}}.
$$
Lemma C.6 (Positive,  $\mathcal{M}_{+}^{-}\cap \mathcal{M}_{-}^{-}$
For positive neuron  $k\in \{k\in [m / 2]:\pmb {w}_k(0)\in \mathcal{M}_+^-\cap \mathcal{M}_-^-\}$  , it keeps dead forever.
Proof of Lemma C.6. Due to Lemma B.2, this lemma is trivial.
C.2 The Dynamics of Negative Neurons
According to the initial direction, all negative neurons  $(\mathrm{d}_k = -1)$  can be divided into the following four classes.
$$
\begin{array}{l} [ m ] - [ m / 2 ] \\ = \left\{k \in [ m ] - [ m / 2 ]: \boldsymbol {w} _ {k} (0) \in \mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {+} \right\} \bigcup \left\{k \in [ m ] - [ m / 2 ]: \boldsymbol {w} _ {k} (0) \in \mathcal {M} _ {+} ^ {-} \cap \mathcal {M} _ {-} ^ {+} \right\} \\ \bigcup \left\{k \in [ m ] - [ m / 2 ]: \boldsymbol {w} _ {k} (0) \in \mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {-} \right\} \bigcup \left\{k \in [ m ] - [ m / 2 ]: \boldsymbol {w} _ {k} (0) \in \mathcal {M} _ {+} ^ {-} \cap \mathcal {M} _ {-} ^ {-} \right\}. \\ \end{array}
$$
In the following four lemmas, we will prove the dynamics of these four classes of negative neurons. In summary, in Phase I ( $t < T_{\mathrm{I}}$ ), some of the negative neurons move to the manifold  $\mathcal{M}_{+}^{0} \cap \mathcal{M}_{-}^{+}$  in a shorter time and then remain on this manifold, and their norms grow slowly, while other negative neurons go dead.
Lemma C.7 (Negative,  $\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{+})$
For negative neuron  $k \in \{k \in [m] - [m/2] : \mathbf{w}_k(0) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^+\}$ , in Phase I ( $t \leq T_1$ ), it's dynamics must belong to one of the following two cases:
(i. Living). (S1).  $\pmb{w}_k(t) \in \mathcal{M}_+^0 \cap \mathcal{M}_-^+$  for any  $t \geq \mathcal{O}\left(\frac{\kappa_1}{\kappa_2}\right)$ ,
(S2). It has a small norm:  $\rho_{k}(T_{\mathrm{I}}) = \mathcal{O}\Big(\frac{\sqrt{\kappa_{1}\kappa_{2}}}{\sqrt{m}}\big(\sqrt{\frac{\kappa_{1}}{\kappa_{2}}} +\frac{\Delta}{p}\big)\Big),$
(S3). It is weakly aligned with  $\pmb{x}_{+}^{\perp}:\left\langle \pmb{w}_{k}(T_{\mathrm{I}}),\pmb{x}_{+}^{\perp}\right\rangle \geq 1 - \mathcal{O}\Bigl ((\sqrt{\frac{\kappa_{1}}{\kappa_{2}}}\frac{p}{\Delta})^{1.6}\Bigr);$
(ii. Dead).  $\pmb{w}_k(t) \in \mathcal{M}_+^- \cap \mathcal{M}_-^-$  for any  $t \geq \mathcal{O}\left(\frac{\kappa_1}{\kappa_2}\right)$ .
Moreover, if  $\langle \pmb{w}_k(0),\pmb{x}_{-}\rangle >\frac{(1 + \mathcal{O}(\kappa_1\kappa_2))p\cos\Delta - (1 - \mathcal{O}(\kappa_1\kappa_2))}{(1 - \mathcal{O}(\kappa_1\kappa_2))p - (1 + \mathcal{O}(\kappa_1\kappa_2))\cos\Delta}\langle \pmb{w}_k(0),\pmb{x}_{+}\rangle$  it must belongs to Case (i); if  $\langle \pmb{w}_k(0),\pmb{x}_{+}\rangle >\frac{(1 + \mathcal{O}(\kappa_1\kappa_2))p - (1 - \mathcal{O}(\kappa_1\kappa_2))\cos\Delta}{(1 - \mathcal{O}(\kappa_1\kappa_2))p\cos\Delta - (1 + \mathcal{O}(\kappa_1\kappa_2))}\langle \pmb{w}_k(0),\pmb{x}_{-}\rangle$  it must belongs to Case (ii).
Proof of Lemma C.7.
We do the following analysis for any  $k \in \{k \in [m] - [m/2] : \pmb{w}_k(0) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^+\}$ , i.e.  $\mathrm{s}_k = -1$ ,  $\langle \pmb{w}_k(0), \pmb{x}_+ \rangle > 0$ , and  $\langle \pmb{w}_k(0), \pmb{x}_- \rangle > 0$ .
Step I. Neuron must arrives in the border  $\partial (\mathcal{M}_+^+\cap \mathcal{M}_-^+)$  in  $\mathcal{O}\left(\frac{\kappa_1}{\kappa_2}\right)$  time.
$$
T _ {\mathrm {h i t}} := \inf  \left\{t \in (0, T _ {\mathrm {I}} ]: \boldsymbol {w} _ {k} (t) \in \partial (\mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {+}) \right\},
$$
and we aim to prove  $T_{\mathrm{hit}}$  exists and estimate  $T_{\mathrm{hit}}$ .
Recalling the decoupling  $\partial (\mathcal{M}_+^+\cap \mathcal{M}_-^+) = (\mathcal{M}_+^0\cap \mathcal{M}_-^+)\cup (\mathcal{M}_-^0\cap \mathcal{M}_+^+)\cup (\mathcal{M}_+^0\cap \mathcal{M}_-^0)$ , we only need to focus on the dynamics of  $\langle \pmb {b}_k(t),\pmb{x}_+\rangle$  and  $\langle \pmb {b}_k(t),\pmb{x}_{-}\rangle$
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{n} \sum_ {i = 1} ^ {n} e ^ {- y _ {i} f _ {i} (t)} y _ {i} \boldsymbol {x} _ {i} = - \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-} \Big), t \leq T _ {\mathrm {h i t}}.
$$
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \right\rangle}{\mathrm {d} t} = \left\langle - \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}\right), \boldsymbol {x} _ {+} \right\rangle \\ = - \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {p}{1 + p} e ^ {- f _ {+} (t)} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \cos \Delta \Big), \\ \end{array}
$$
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \pmb {b} _ {k} (t) , \pmb {x} _ {-} \right\rangle}{\mathrm {d} t} = \left\langle - \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {p}{1 + p} e ^ {- f _ {+} (t)} \pmb {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \pmb {x} _ {-} \Big), \pmb {x} _ {-} \right\rangle \\ = - \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {p}{1 + p} e ^ {- f _ {+} (t)} \cos \Delta - \frac {1}{1 + p} e ^ {f _ {-} (t)} \Big). \\ \end{array}
$$
Due to  $T_{\mathrm{hit}} \leq T_{\mathrm{I}} \leq T_{\mathrm{init}}$  and Lemma C.2, for any  $t \leq T_{\mathrm{hit}}$ , we have  $\left|e^{-y_i f_i(t)} - 1\right| \leq 1.1 \sqrt{\kappa_1 \kappa_2}$ . Then for any  $t \leq T_{\mathrm{hit}}$ , we have:
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \pmb {b} _ {k} (t) , \pmb {x} _ {+} \right\rangle}{\mathrm {d} t} \leq - \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {(1 - 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}) p}{1 + p} - \frac {1 + 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}}{1 + p} \cos \Delta \Big) \\ \leq - \frac {\kappa_ {2} \left((1 - 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}) p - (1 + 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}) \cos \Delta\right)}{\sqrt {m} (1 + p)} \leq - \frac {\kappa_ {2} (0 . 9 8 p - 1 . 0 2)}{\sqrt {m} (1 + p)}, \\ \end{array}
$$
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} \leq - \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {(1 - 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}) p}{1 + p} \cos \Delta - \frac {1 + 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}}{1 + p} \Big) \\ \leq - \frac {\kappa_ {2} \left(\left(1 - 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}\right) p \cos \Delta - \left(1 + 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}}\right)\right)}{\sqrt {m} (1 + p)} \leq - \frac {0 . 9 8 \kappa_ {2} (p \cos \Delta - 1)}{2 \sqrt {m} (1 + p)}. \\ \end{array}
$$
Now we consider the time
$$
T _ {\mathrm {t e s t}} := \frac {3 \kappa_ {1}}{\kappa_ {2}}.
$$
If we assume  $T_{\mathrm{test}} < T_{\mathrm{hit}}$ , then we have the estimate:
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k} (T _ {\text {t e s t}}), \boldsymbol {x} _ {+} \rangle \leq \langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {+} \rangle - \int_ {0} ^ {T _ {\text {t e s t}}} \frac {\kappa_ {2} (0 . 9 8 p - 1 . 0 2)}{\sqrt {m} (1 + p)} d t \\ \leq \frac {\kappa_ {1}}{\sqrt {m}} - \frac {\kappa_ {2} (0 . 9 8 p - 1 . 0 2)}{\sqrt {m} (1 + p)} \frac {3 \kappa_ {1}}{\kappa_ {2}} <   0, \\ \end{array}
$$
which is contradict to the definition of  $T_{\mathrm{hit}}$ . Hence, we have:
$$
T _ {\mathrm {h i t}} \leq T _ {\mathrm {t e s t}} \leq \frac {3 \kappa_ {1}}{\kappa_ {2}},
$$
which means neurons must arrive in the border  $\partial (\mathcal{M}_+^+\cap \mathcal{M}_-^+)$  in  $\mathcal{O}\bigl (\frac{\kappa_1}{\kappa_2}\bigr)$  time.
Because  $\partial (\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{+}) = \left(\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{+}\right)\cup \left(\mathcal{M}_{-}^{0}\cap \mathcal{M}_{+}^{+}\right)\cup \left(\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{0}\right)$ , the neuron must arrives in  $\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{+}$  or  $\mathcal{M}_{-}^{0}\cap \mathcal{M}_{+}^{+}$  or  $\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{0}$ . If the neuron arrives in  $\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{0}$ , it goes dead forever (Lemma B.2). We will analyze the training dynamics after arriving in  $\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{+}$  or  $\mathcal{M}_{-}^{0}\cap \mathcal{M}_{+}^{+}$  in the following Step II and Step III.
Step II. Dynamics after arriving in the manifold  $\mathcal{M}_+^0\cap \mathcal{M}_-^+$
In this step, we will analyze the training dynamics after  $\pmb{w}_k(T_{\mathrm{hit}}) \in \mathcal{M}_+^0 \cap \mathcal{M}_-^+$ , i.e. after  $\pmb{b}_k(T_{\mathrm{hit}}) \in \mathcal{P}_+^0 \cap \mathcal{P}_-^+$ .
We first analysis the vector field around the manifold  $\mathcal{P}_{+}^{0}\cap \mathcal{P}_{-}^{+}$  for  $T_{\mathrm{hit}}\leq t\leq T_{\mathrm{I}}$
For any  $\tilde{\pmb{b}}\in \mathcal{P}_{+}^{0}\cap \mathcal{P}_{-}^{+}$  and  $0 <   \delta_0\ll 1$  , we know that  $\mathcal{P}_+^0\cap \mathcal{P}_-^+$  separates its neighborhood  $\mathcal{B}(\tilde{\pmb{b}},\delta_0)$  into two domains  $\mathcal{G}_{-} = \{\pmb {b}\in \mathcal{B}(\tilde{\pmb{b}},\delta_{0}):\langle \pmb {b},\pmb{x}_{+}\rangle <  0\}$  and  $\mathcal{G}_{+} = \{\pmb {b}\in \mathcal{B}(\tilde{\pmb{b}},\delta_{0}):\langle \pmb {b},\pmb{x}_{+}\rangle >0\}$  Following Definition H.1, we calculate the limited vector field on  $\tilde{\pmb{b}}$  from  $\mathcal{G}_{-}$  and  $\mathcal{G}_{+}$
(i) The limited vector field  $\pmb{F}^{-}$  on  $\tilde{\pmb{b}}$  (from  $\mathcal{G}_{-}$ ):
$$
\frac {\mathrm {d} \boldsymbol {b}}{\mathrm {d} t} = \boldsymbol {F} ^ {-}, \text {w h e r e} \boldsymbol {F} ^ {-} = \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}.
$$
$$
\frac {\mathrm {d} \boldsymbol {b}}{\mathrm {d} t} = \boldsymbol {F} ^ {+}, \text {w h e r e} \boldsymbol {F} ^ {+} = - \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {p e ^ {- f _ {+} (t)}}{1 + p} \boldsymbol {x} _ {+} - \frac {e ^ {f _ {-} (t)}}{1 + p} \boldsymbol {x} _ {-}\right).
$$
(iii) Then we calculate the projections of  $\pmb{F}^{-}$  and  $\pmb{F}^{+}$  onto  $\pmb{x}_{+}$  (the normal to the surface  $\mathcal{P}_{+}^{0} \cap \mathcal{P}_{-}^{+}$ ):
$$
F _ {N} ^ {-} = \left\langle \boldsymbol {F} ^ {-}, \boldsymbol {x} _ {+} \right\rangle = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \cos \Delta ,
$$
$$
F _ {N} ^ {+} = \left\langle \boldsymbol {F} ^ {+}, \boldsymbol {x} _ {+} \right\rangle = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \cos \Delta - \frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)}.
$$
From  $T_{\mathrm{I}} < T_{\mathrm{init}}$  and Lemma C.2, we know  $|e^{-y_i f_i(t)} - 1| \leq 0.11$ , so  $p e^{-f_+(t)} \cos \Delta - e^{f_-(t)} \geq 0.89p \cos \Delta - 1.11 > 0$ , which means  $F_N^+ < 0$ . And it is clear that  $F_N^- > 0$ . Hence, the dynamics corresponds to Case (I) in Definition H.1 ( $F_N^- > 0$  and  $F_N^+ < 0$ ).
(iv) Hence,  $\pmb{b}_k(t)$  can not leave  $\mathcal{P}_{+}^{0} \cap \mathcal{P}_{-}^{+}$  for  $T_{\mathrm{hit}} \leq t \leq T_{\mathrm{I}}$ .
(v) Moreover, the dynamics of  $\pmb{b}_k$  on  $\mathcal{P}_{+}^{0} \cap \mathcal{P}_{-}^{+}$  satisfies:
$$
\frac {\mathrm {d} \boldsymbol {b}}{\mathrm {d} t} = \alpha \boldsymbol {F} ^ {+} + (1 - \alpha) \boldsymbol {F} ^ {-}, \quad \alpha = \frac {f _ {N} ^ {-}}{f _ {N} ^ {-} - f _ {N} ^ {+}},
$$
$$
\frac {\mathrm {d} \pmb {b} _ {k} (t)}{\mathrm {d} t} \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \Big (\pmb {x} _ {-} - \pmb {x} _ {+} \cos \Delta \Big).
$$
By Lemma C.1, we know that the dynamics of  $\pmb{w}_k(t)$  on  $\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{+}$  and the dynamics of  $\rho_{k}(t)$  are:
$$
\frac {\mathrm {d} \boldsymbol {w} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\rho_ {k} (t) \sqrt {m} (1 + p)} \left(\boldsymbol {x} _ {-} - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \boldsymbol {w} _ {k} - \boldsymbol {x} _ {+} \cos \Delta\right). \tag {15}
$$
$$
\frac {\mathrm {d} \rho_ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle . \tag {16}
$$
(vi) In this step, we aim to estimate  $\rho_{k}(T_{\mathrm{I}})$  and  $\langle \pmb {w}_k(t),\pmb{x}_+^{\perp}\rangle$
From  $\pmb{w}_k(T_{\mathrm{hit}}) \in \mathcal{M}_+^0 \cap \mathcal{M}_-^+$ , it holds that  $\langle \pmb{w}_k(T_{\mathrm{hit}}), \pmb{x}_+ \rangle = 0$  and  $\langle \pmb{w}_k(T_{\mathrm{hit}}), \pmb{x}_- \rangle > 0$ . Using lemma I.2, we have  $0 < \langle \pmb{w}_k(T_{\mathrm{hit}}), \pmb{x}_- \rangle \leq \sin \Delta$ .
Recalling Lemma C.2 and the estimate of  $T_{\mathrm{hit}}$  in Step I, we have:
$$
0 \leq \rho_ {k} (T _ {\mathrm {h i t}}) \leq \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} T _ {\mathrm {h i t}}}{\sqrt {m}},
$$
and we can estimate the dynamics for  $T_{\mathrm{hit}} \leq t \leq T_{\mathrm{I}}$  by (v)(vi):
$$
\begin{array}{l} 0 \leq \frac {\mathrm {d} \rho_ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \left\langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \right\rangle \leq \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \sin \Delta , \\ e ^ {f _ {-} (t)} \leq 1 + 0. 1 1 = 1. 1 1. \\ \end{array}
$$
Then we obtain the estimate of  $\rho_{k}(t)$  for any  $T_{\mathrm{hit}} < t \leq T_{1}$ :
$$
\begin{array}{l} \rho_ {k} (t) = \rho_ {k} \left(T _ {\mathrm {h i t}}\right) + \int_ {T _ {\mathrm {h i t}}} ^ {t} \frac {\mathrm {d} \rho_ {k} (s)}{\mathrm {d} t} \mathrm {d} s \leq \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} T _ {\mathrm {h i t}}}{\sqrt {m}} + \frac {\kappa_ {2} e ^ {f _ {-} (t)} \sin \Delta}{\sqrt {m} (1 + p)} \left(t - T _ {\mathrm {h i t}}\right) \\ \leq \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} T _ {\mathrm {h i t}}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} \left(T _ {\mathrm {I}} - T _ {\mathrm {h i t}}\right) \leq \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} \frac {3 \kappa_ {1}}{\kappa_ {2}}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} \left(T _ {\mathrm {I}} - \frac {3 \kappa_ {1}}{\kappa_ {2}}\right) \\ \leq \frac {4 . 3 \kappa_ {1}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} t. \\ \end{array}
$$
$$
\rho_ {k} (T _ {\mathrm {I}}) \leq \frac {4 . 3 \kappa_ {1}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} T _ {\mathrm {I}} \leq \frac {4 . 3 \kappa_ {1}}{\sqrt {m}} + \frac {1 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}} \sin \Delta}{\sqrt {m} (1 + p)} = \frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \Big (4. 3 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} + \frac {1 1 . 1 \sin \Delta}{1 + p} \Big).
$$
For any  $\pmb{w} \in \mathcal{M}_{+}^{0} \cap \mathcal{M}_{-}^{+}$ , we have  $\langle \pmb{w}, \pmb{x}_{+} \rangle = 0$ , so
$$
\left\langle \boldsymbol {w}, \boldsymbol {x} _ {+} ^ {\perp} \right\rangle = \left\langle \boldsymbol {w}, \frac {\boldsymbol {x} _ {-} - \boldsymbol {x} _ {+} \cos \Delta}{\| \boldsymbol {x} _ {-} - \boldsymbol {x} _ {+} \cos \Delta \|} \right\rangle = \frac {\left\langle \boldsymbol {w} , \boldsymbol {x} _ {-} \right\rangle}{\| \boldsymbol {x} _ {-} - \boldsymbol {x} _ {+} \cos \Delta \|} = \frac {1}{\sin \Delta} \left\langle \boldsymbol {w}, \boldsymbol {x} _ {-} \right\rangle .
$$
So we only need to focus on the dynamics of  $\langle \pmb{w}_k(t),\pmb{x}_{-}\rangle$  to derive the dynamics of  $\langle \pmb{w}_k(t),\pmb{x}_{+}^{\perp}\rangle$
By (15) and the estimate of  $\rho_{k}(t)$ , for any  $T_{\mathrm{hit}} \leq t \leq T_{\mathrm{I}}$  we have:
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \boldsymbol {w} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} = \left\langle \frac {e ^ {f _ {-} (t)}}{\rho_ {k} (t) \sqrt {m} (1 + p)} \left(\boldsymbol {x} _ {-} - \left\langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \right\rangle \boldsymbol {w} _ {k} (t) - \boldsymbol {x} _ {+} \cos \Delta\right), \boldsymbol {x} _ {-} \right\rangle \\ = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\rho_ {k} (t) \sqrt {m} (1 + p)} \Big (\sin^ {2} \Delta - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle^ {2} \Big) \geq \frac {(1 - 0 . 1 1) \kappa_ {2}}{(1 + p) \Big (4 . 3 \kappa_ {1} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{(1 + p)} t \Big)} \Big (\sin^ {2} \Delta - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle^ {2} \Big) \\ \geq \frac {0 . 8 9 \kappa_ {2}}{4 . 3 (1 + p) \kappa_ {1} + 1 . 1 1 \kappa_ {2} t \sin \Delta} \Big (\sin^ {2} \Delta - \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle^ {2} \Big). \\ \end{array}
$$