SlowGuess/NeurIPS_2023_OCR huggingface dataset:
And we have  $0 < \left\langle \pmb{w}_k\left(\frac{3\kappa_1}{\kappa_2}\right),\pmb{x}_{-}\right\rangle < \sin \Delta$
$$
\left\{ \begin{array}{l} \frac {\mathrm {d} U (t)}{\mathrm {d} t} = \frac {0 . 8 9 \kappa_ {2}}{4 . 3 (1 + p) \kappa_ {1} + 1 . 1 1 \kappa_ {2} t \sin \Delta} \left(\sin^ {2} \Delta - U ^ {2} (t)\right) \\ U (0) = 0 \end{array} , \right. \tag {17}
$$
and let  $U(t)$  is the solution of (13). We know that  $\langle \pmb{w}_k(t), \pmb{x}_{-} \rangle$  is an upper solution of ODE (17). From the Comparison Principle of ODEs, we know this means:
$$
\langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle > U (t), \text {f o r a n y} t \leq T _ {\mathrm {I}}.
$$
In order to estimate  $\langle \pmb{w}_k(t),\pmb{x}_{-}\rangle$  , we only need to study the solution of ODE (17). It is easy to verify that the solution of (17) satisfies
$$
\log \left(\frac {\sin \Delta + U (t)}{\sin \Delta - U (t)}\right) - \log \left(\frac {\sin \Delta}{\sin \Delta}\right) = \frac {1 . 7 8 \kappa_ {2} \Delta}{1 . 1 1 \kappa_ {2} \sin \Delta} \log \left(\frac {4 . 3 (1 + p) \kappa_ {1} + 1 . 1 1 \kappa_ {2} t \sin \Delta}{4 . 3 (1 + p) \kappa_ {1} + 3 . 3 3 \kappa_ {1} \sin \Delta}\right)
$$
$$
\begin{array}{l} \log \left(\frac {\sin \Delta + U (T _ {\mathrm {I}})}{\sin \Delta - U (T _ {\mathrm {I}})}\right) \geq \frac {1 . 7 8}{1 . 1 1} \log \left(\frac {4 . 3 (1 + p) \kappa_ {1} + 1 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}} \sin \Delta}{4 . 3 (1 + p) \kappa_ {1} + 3 . 3 3 \kappa_ {1} \sin \Delta}\right) \\ > 1. 6 \log \left(1 + \frac {\left(1 1 . 1 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}} - 3 . 3 3\right) \frac {\sin \Delta}{1 + p}}{4 . 3 + 3 . 3 3 \frac {\sin \Delta}{1 + p}}\right) > 1. 6 \log \left(1 + \frac {\left(1 1 . 1 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}} - 3 . 3 3\right) \frac {\sin \Delta}{1 + p}}{4 . 6}\right) \\ > 1. 6 \log \left(1 + \frac {1 0 . 7}{4 . 6} \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}} \frac {\sin \Delta}{1 + p}\right), \\ \end{array}
$$
$$
U (T _ {\mathrm {I}}) > \left(1 - \frac {2}{\left(1 + \frac {1 0 . 7}{4 . 6} \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}} \frac {\sin \Delta}{1 + p}\right) ^ {1 . 6} + 1}\right) \sin \Delta .
$$
Hence, we have the estimate of  $\langle \pmb{w}_k(t),\pmb{x}_+^\perp \rangle$
$$
\left\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {+} ^ {\perp} \right\rangle = \frac {1}{\sin \Delta} \left\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \right\rangle > \frac {1}{\sin \Delta} U (T _ {\mathrm {I}}) > 1 - \frac {2}{\left(1 + 2 . 3 2 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}} \frac {\sin \Delta}{1 + p}\right) ^ {1 . 6} + 1}.
$$
Step III. Dynamics after arriving in the manifold  $\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{0}$
In this step, we analyze the training dynamics after  $\pmb{w}_k(T_{\mathrm{hit}}) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^0$ , i.e.  $\pmb{b}_k(T_{\mathrm{hit}}) \in \mathcal{P}_+^+ \cap \mathcal{P}_-^0$ .
$$
\frac {\mathrm {d} \boldsymbol {b}}{\mathrm {d} t} = \boldsymbol {F} ^ {-}, \text {w h e r e} \boldsymbol {F} ^ {-} = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+}.
$$
$$
F _ {N} ^ {-} = \left\langle \boldsymbol {F} ^ {-}, \boldsymbol {x} _ {-} \right\rangle = - \frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)} \cos \Delta ,
$$
$$
F _ {N} ^ {+} = \left\langle \pmb {F} ^ {+}, \pmb {x} _ {-} \right\rangle = - \left(\frac {\kappa_ {2} p e ^ {- f _ {+} (t)}}{\sqrt {m} (1 + p)} \cos \Delta - \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)}\right).
$$
Then the dynamics of  $\pmb{b}_k$  in  $\mathcal{P}_{+}^{+} \cap \mathcal{P}_{-}^{-}$  satisfies:
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+}.
$$
(v) We define the following time, and our aim is to estimate  $T_{\mathrm{test},1}$ :
$$
T _ {\text {t e s t}, 1} := \inf  \left\{t \in \left(T _ {\text {h i t}}, T _ {\mathrm {I}} \right]: \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \text {o r} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \geq 0 \right\},
$$
$$
T _ {\text {t e s t}, 2} := \inf  \left\{t \in \left(T _ {\text {h i t}}, T _ {\mathrm {I}} \right]: \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \right\}
$$
It is clear  $T_{\mathrm{test},1} \leq T_{\mathrm{test},2}$ . Moreover, due to  $\frac{\mathrm{d}\langle \pmb{b}_k(t), \pmb{x}_- \rangle}{\mathrm{d}t} = -\frac{\kappa_2}{\sqrt{m}} \frac{p}{1 + p} e^{-f_+ (t)} \cos \Delta < 0$  and  $\langle \pmb{b}_k(T_{\mathrm{hit}}), \pmb{x}_- \rangle = 0$ , we know  $\langle \pmb{b}_k(t), \pmb{x}_- \rangle < 0$  holds for any  $t \leq T_{\mathrm{test},1}$ . Hence, we have
$$
T _ {\text {t e s t}, 1} = T _ {\text {t e s t}, 2}, \quad \langle \boldsymbol {w} _ {k} (T _ {\text {t e s t}, 1}), \boldsymbol {x} _ {+} \rangle = 0, \quad \langle \boldsymbol {w} _ {k} (T _ {\text {t e s t}, 1}), \boldsymbol {x} _ {-} \rangle <   0.
$$
And we only need to estimate  $T_{\mathrm{test},2}$ . For any  $T_{\mathrm{hit}} < t \leq T_{\mathrm{test},1} = T_{\mathrm{test},2}$ , we have
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \right\rangle}{\mathrm {d} t} = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \stackrel {\text {L e m m a C . 2}} {\leq} - \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} (1 - 0. 1 1) = - \frac {0 . 8 9 \kappa_ {2} p}{\sqrt {m} (1 + p)}.
$$
$$
\langle \boldsymbol {b} _ {k} (T _ {\mathrm {h i t}}), \boldsymbol {x} _ {+} \rangle \leq \| \rho_ {k} (t) \| \leq \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} T _ {\mathrm {h i t}}}{\sqrt {m}} \leq \| \rho_ {k} (t) \| \leq \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} \frac {3 \kappa_ {1}}{\kappa_ {2}}}{\sqrt {m}} = \frac {4 . 3 \kappa_ {1}}{\sqrt {m}}.
$$
Then for any  $T_{\mathrm{hit}} < t \leq T_{\mathrm{test},2}$ , we have:
$$
\langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq \langle \boldsymbol {b} _ {k} (T _ {\mathrm {h i t}}), \boldsymbol {x} _ {+} \rangle - \int_ {T _ {\mathrm {h i t}}} ^ {t} \frac {0 . 8 9 \kappa_ {2} p}{\sqrt {m} (1 + p)} \mathrm {d} s \leq \frac {4 . 3 \kappa_ {1}}{\sqrt {m}} - \frac {0 . 8 9 \kappa_ {2} p (t - T _ {\mathrm {h i t}})}{\sqrt {m} (1 + p)}.
$$
So we have the estimate
$$
T _ {\text {t e s t}, 1} = T _ {\text {t e s t}, 2} \leq T _ {\text {h i t}} + \frac {4 . 3 \kappa_ {1} (1 + p)}{0 . 8 9 \kappa_ {2} p} \leq \left(3 + \frac {4 . 3 \cdot 6}{0 . 8 9 \cdot 5}\right) \frac {\kappa_ {1}}{\kappa_ {2}} \leq \frac {9 \kappa_ {1}}{\kappa_ {2}} <   T _ {\mathrm {I}}.
$$
Recalling  $\pmb{w}_k(T_{\mathrm{test},1}) \in \mathcal{M}_+^-\cap \mathcal{M}_-^-$  and Lemma B.2, the neuron  $\pmb{b}_k(t)$  keeps dead for any  $t \geq T_1$ .
Step IV. Which subspace does the neuron select?
From Step II, we know that the neuron  $w_{k}(t)$  must arrives in  $\mathcal{M}_{+}^{0} \cap \mathcal{M}_{-}^{+}$  or  $\mathcal{M}_{-}^{0} \cap \mathcal{M}_{+}^{+}$  or  $\mathcal{M}_{+}^{0} \cap \mathcal{M}_{-}^{0}$ . In this step, we will analyze which subspace does the neuron select.
We only need to compare the following two times:
$$
T _ {\text {h i t}, +} := \inf  \left\{t \in (0, T _ {\mathrm {I}} ]: \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \right\},
$$
$$
T _ {\text {h i t}, -} := \inf  \left\{t \in (0, T _ {\mathrm {I}} ]: \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq 0 \right\}.
$$
From the definition of  $T_{\mathrm{hit}}$ , we know  $T_{\mathrm{hit}, + } = T_{\mathrm{hit}}$  or  $T_{\mathrm{hit}, - } = T_{\mathrm{hit}}$ .
Recalling the proof in Step II, we compare the following two dynamics for  $t < T_{\mathrm{hit}}$ :
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \right\rangle}{\mathrm {d} t} = - \frac {\kappa_ {2}}{\sqrt {m}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \cos \Delta\right),
$$
$$
\frac {\mathrm {d} \left\langle \pmb {b} _ {k} (t) , \pmb {x} _ {-} \right\rangle}{\mathrm {d} t} = - \frac {\kappa_ {2}}{\sqrt {m}} \Big (\frac {p}{1 + p} e ^ {- f _ {+} (t)} \cos \Delta - \frac {1}{1 + p} e ^ {f _ {-} (t)} \Big).
$$
With the help of Lemma C.2 and the estimate of  $T_{\mathrm{hit}}$ , for any  $t \leq T_{\mathrm{hit}}$
$$
\left| e ^ {- y _ {i} f _ {i} (t)} - 1 \right| \leq 1. 1 \kappa_ {2} \left(\kappa_ {1} + 1. 1 \kappa_ {2} T _ {\mathrm {h i t}}\right) \leq 1. 1 \kappa_ {2} \left(\kappa_ {1} + 3. 3 \kappa_ {1}\right) = 4. 7 3 \kappa_ {1} \kappa_ {2}.
$$
Hence, we have the estimate of the dynamics:
$$
- \frac {\kappa_ {2} \Big ((1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) p - (1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) \cos \Delta \Big)}{\sqrt {m} (1 + p)} \leq \frac {\mathrm {d} \langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \rangle}{\mathrm {d} t} \leq - \frac {\kappa_ {2} \Big ((1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) \cos \Delta \Big)}{\sqrt {m} (1 + p)},
$$
$$
- \frac {\kappa_ {2} \Big ((1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) p \cos \Delta - (1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) \Big)}{\sqrt {m} (1 + p)} \leq \frac {\mathrm {d} \langle \pmb {b} _ {k} (t) , \pmb {x} _ {-} \rangle}{\mathrm {d} t} \leq - \frac {\kappa_ {2} \Big ((1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p \cos \Delta - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) \Big)}{\sqrt {m} (1 + p)}.
$$
(i) If the initialization satisfies  $\langle \pmb{b}_k(0),\pmb{x}_{-}\rangle >\frac{(1 + 4.73\kappa_1\kappa_2)p\cos\Delta - (1 - 4.73\kappa_1\kappa_2)}{(1 - 4.73\kappa_1\kappa_2)p - (1 + 4.73\kappa_1\kappa_2)\cos\Delta}\langle \pmb{b}_k(0),\pmb{x}_{+}\rangle$  we will prove that the neuron selects  $\mathcal{M}_+^0\cap \mathcal{M}_-^+$  at  $T_{\mathrm{hit}}$
For any  $t < T_{\mathrm{hit}}$ , we have the estimate:
$$
\langle \pmb {b} _ {k} (t), \pmb {x} _ {+} \rangle \leq \langle \pmb {b} _ {k} (0), \pmb {x} _ {+} \rangle - \frac {\kappa_ {2} \Big ((1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) \cos \Delta \Big)}{\sqrt {m} (1 + p)} t,
$$
$$
\begin{array}{l} \langle \pmb {b} _ {k} (t), \pmb {x} _ {-} \rangle \geq \langle \pmb {b} _ {k} (0), \pmb {x} _ {-} \rangle - \frac {\kappa_ {2} \Big ((1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) p \cos \Delta - (1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) \Big)}{\sqrt {m} (1 + p)} t \\ > \frac {(1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) p \cos \Delta - (1 - 4 . 7 3 \kappa_ {1} \kappa_ {2})}{(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) \cos \Delta} \left(\langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {+} \rangle - \frac {\kappa_ {2} ((1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) \cos \Delta)}{\sqrt {m} (1 + p)} t\right), \\ \end{array}
$$
Comparing these two inequalities, we have:
$$
T _ {\text {h i t , +}} = T _ {\text {h i t}} <   T _ {\text {h i t , -}},
$$
$$
\langle \boldsymbol {b} _ {k} (T _ {\mathrm {h i t}}), \boldsymbol {x} _ {+} \rangle = 0, \quad \langle \boldsymbol {b} _ {k} (T _ {\mathrm {h i t}}), \boldsymbol {x} _ {-} \rangle > 0.
$$
So the neuron  $\pmb{w}_k(T_{\mathrm{hit}}) \in \mathcal{M}_+^0 \cap \mathcal{M}_-^+$ .
(ii) If the initialization satisfies  $\langle \pmb{b}_k(0),\pmb{x}_{+}\rangle >\frac{(1 + 4.73\kappa_1\kappa_2)p - (1 - 4.73\kappa_1\kappa_2)\cos\Delta}{(1 - 4.73\kappa_1\kappa_2)p\cos\Delta - (1 + 4.73\kappa_1\kappa_2)}\langle \pmb{b}_k(0),\pmb{x}_{-}\rangle$  we will prove that the neuron selects  $\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{-}$  at  $T_{\mathrm{hit}}$
$$
\langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {-} \rangle \leq \langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {-} \rangle - \frac {\kappa_ {2} \left((1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p \cos \Delta - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2})\right)}{\sqrt {m} (1 + p)} t,
$$
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \rangle \geq \langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {+} \rangle - \frac {\kappa_ {2} ((1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) p - (1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) \cos \Delta)}{\sqrt {m} (1 + p)} t \\ > \frac {(1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) p - (1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) \cos \Delta}{(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p \cos \Delta - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2})} \left(\langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {-} \rangle - \frac {\kappa_ {2} ((1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p \cos \Delta - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}))}{\sqrt {m} (1 + p)} t\right), \\ \end{array}
$$
$$
T _ {\text {h i t , -}} = T _ {\text {h i t}} <   T _ {\text {h i t , +}},
$$
$$
\langle \boldsymbol {b} _ {k} (T _ {\mathrm {h i t}}), \boldsymbol {x} _ {-} \rangle = 0, \quad \langle \boldsymbol {b} _ {k} (T _ {\mathrm {h i t}}), \boldsymbol {x} _ {+} \rangle > 0.
$$
So the neuron  $\pmb{w}_k(T_{\mathrm{hit}}) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^0$ .
Lemma C.8 (Negative,  $\mathcal{M}_{+}^{-}\cap \mathcal{M}_{-}^{+})$
For negative neuron  $k \in \{k \in [m] - [m/2] : \boldsymbol{w}_k(0) \in \mathcal{M}_+^- \cap \mathcal{M}_-^+\}$ , in Phase I ( $t \leq T_1$ ), we have:
(S1).  $\pmb{w}_k(t) \in \mathcal{M}_+^0 \cap \mathcal{M}_-^+$  for any  $t \leq \mathcal{O}\left(\frac{\kappa_1}{\kappa_2} p\Delta\right)$ ,
(S2). It has a small norm:  $\rho_{k}(T_{\mathrm{I}}) = \mathcal{O}\Big(\frac{\sqrt{\kappa_{1}\kappa_{2}}}{\sqrt{m}}\big(\sqrt{\frac{\kappa_{1}}{\kappa_{2}}} +\frac{\Delta}{p}\big)\Big),$
(S3). It is aligned with  $\pmb{x}_{+}^{\perp}:\left\langle \pmb{w}_{k}(T_{1}),\pmb{x}_{+}^{\perp}\right\rangle \geq 1 - \mathcal{O}\Bigl ((\sqrt{\frac{\kappa_{1}}{\kappa_{2}}}\frac{p}{\Delta})^{1.6}\Bigr)$
Proof of Lemma C.8.
We do the following analysis for any  $k \in \{k \in [m] - [m/2] : \pmb{w}_k(0) \in \mathcal{M}_+^- \cap \mathcal{M}_-^+\}$ , i.e.  $\mathrm{s}_k = -1$ ,  $\langle \pmb{w}_k(0), \pmb{x}_+ \rangle \leq 0$ , and  $\langle \pmb{w}_k(0), \pmb{x}_- \rangle > 0$ .
Step I. The neuron must arrives in  $\mathcal{M}_+^0\cap \mathcal{M}_-^+$  in  $\mathcal{O}\left(\frac{\kappa_1p\Delta}{\kappa_2}\right)$  time.
The case  $\langle \pmb{w}_k(0),\pmb{x}_{+}\rangle = 0$  is trivial. Then we only need to consider the case  $\langle \pmb{w}_k(0),\pmb{x}_{+}\rangle < 0$
and we aim to estimate  $T_{\mathrm{hit}}$  and prove  $\pmb{w}_k(T_{\mathrm{hit}})\in \mathcal{M}_+^0\cap \mathcal{M}_-^+$
We focus on the dynamics of  $\langle \pmb {b}_k(t),\pmb{x}_+\rangle$  and  $\langle \pmb {b}_k(t),\pmb {x}_-\rangle$
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}, t \leq T _ {\text {h i t}}.
$$
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \right\rangle}{\mathrm {d} t} = \left\langle \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}, \boldsymbol {x} _ {+} \right\rangle = \frac {\kappa_ {2} \cos \Delta}{\sqrt {m} (1 + p)} e ^ {f _ {-} (t)},
$$
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} = \left\langle \frac {\kappa_ {2}}{\sqrt {m}} \frac {1}{1 + p} e ^ {f _ {-} (t)} \boldsymbol {x} _ {-}, \boldsymbol {x} _ {-} \right\rangle = \frac {\kappa_ {2}}{\sqrt {m} (1 + p)} e ^ {f _ {-} (t)}.
$$
It is clear  $\frac{\mathrm{d}\langle\pmb{b}_k(t),\pmb{x}_{-}\rangle}{\mathrm{d}t} >0$  so  $\langle \pmb {b}_k(t),\pmb {x}_{-}\rangle >\langle \pmb {b}_k(0),\pmb {x}_{-}\rangle >0$  for any  $t\leq T_{\mathrm{hit}}$ . If we denote
$$
T _ {\mathrm {h i t}, +} := \inf  \left\{t \in (0, T _ {\mathrm {I}} ]: \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \right\},
$$
$$
T _ {\mathrm {h i t}} = T _ {\mathrm {h i t , +}}.
$$
So we only need to estimate  $T_{\mathrm{hit, + }}$ . Due to  $T_{\mathrm{hit}} \leq T_{\mathrm{I}} \leq T_{\mathrm{init}}$  and Lemma C.2, for any  $t \leq T_{\mathrm{hit}}$ , we have  $\left|e^{-y_i f_i(t)} - 1\right| \leq 0.11$ . Then for any  $t \leq T_{\mathrm{hit}}$ , we have:
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \right\rangle}{\mathrm {d} t} \geq \frac {0 . 8 9 \kappa_ {2} \cos \Delta}{\sqrt {m} (1 + p)}.
$$
Recalling  $\langle \pmb{w}_k(0),\pmb{x}_{+}\rangle < 0$  and  $\langle \pmb{w}_k(0),\pmb{x}_{-}\rangle >0$  , with the help of Lemma I.2, we have  $\langle \pmb {w}_k(0),\pmb{x}_+\rangle > - \sin \Delta$  and  $\langle \pmb {w}_k(0),\pmb{x}_{-}\rangle <  \sin \Delta$  . Combining the two estimate, we have:
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \rangle \geq \langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {+} \rangle + \int_ {0} ^ {t} \frac {0 . 8 9 \kappa_ {2} \cos \Delta}{\sqrt {m} (1 + p)} d t \\ > - \rho_ {k} (0) \sin \Delta + \frac {0 . 8 9 \kappa_ {2} \cos \Delta}{\sqrt {m} (1 + p)} t = - \frac {\kappa_ {1} \sin \Delta}{\sqrt {m}} + \frac {0 . 8 9 \kappa_ {2} \cos \Delta}{\sqrt {m} (1 + p)} t. \\ \end{array}
$$
$$
T _ {\mathrm {h i t}} = T _ {\mathrm {h i t , +}} \leq \frac {(1 + p) \tan \Delta}{0 . 8 9} \frac {\kappa_ {1}}{\kappa_ {2}} \leq 2 p \Delta \frac {\kappa_ {1}}{\kappa_ {2}} <   T _ {\mathrm {I}} = 1 0 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}}.
$$
Moreover, we can estimate of  $\rho_{k}(T_{\mathrm{hit}})$
Since  $\langle \pmb{w}_k(t),\pmb{x}_{+}\rangle < 0$  and  $\langle \pmb{w}_k(t),\pmb{x}_{-}\rangle >0$  hold for any  $t\leq T_{\mathrm{hit}}$ , with the help of Lemma I.2, we have  $\langle \pmb {w}_k(t),\pmb {x}_-\rangle < \sin \Delta$ . Combining (7), for any  $t\leq T_{\mathrm{hit}}$ , we have
$$
\begin{array}{l} \rho_ {k} (t) \leq \rho_ {k} (0) + \int_ {0} ^ {t} \frac {\kappa_ {2}}{\sqrt {m} (1 + p)} e ^ {f _ {-} (t)} \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \rangle \mathrm {d} t \\ \leq \frac {\kappa_ {1}}{\sqrt {m}} + \int_ {0} ^ {t} \frac {1 . 1 1 \kappa_ {2}}{\sqrt {m} (1 + p)} \sin \Delta \mathrm {d} t \leq \frac {\kappa_ {1}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} t. \\ \end{array}
$$
Proceeding as in the proof of Step II in the Proof of Theorem C.7, we have:
$\pmb{w}_k(t)$  can not leave  $\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{+}$  for  $T_{\mathrm{hit}}\leq t\leq T_{\mathrm{I}}$  . Moreover, the dynamics of  $\pmb{w}_k$  on  $\mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{+}$  satisfies:
$$
\frac {\mathrm {d} \boldsymbol {w} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\rho_ {k} (t) \sqrt {m} (1 + p)} \Big (\boldsymbol {x} _ {-} - \langle \boldsymbol {w} _ {k}, \boldsymbol {x} _ {-} \rangle \boldsymbol {w} _ {k} - \boldsymbol {x} _ {+} \cos \Delta \Big),
$$
$$
\frac {\mathrm {d} \rho_ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \left<   \pmb {w} _ {k} (t), \pmb {x} _ {-} \right > ,
$$
$$
\frac {\mathrm {d} \pmb {b} _ {k} (t)}{\mathrm {d} t} = \frac {\kappa_ {2} e ^ {f _ {-} (t)}}{\sqrt {m} (1 + p)} \Big (\pmb {x} _ {-} - \pmb {x} _ {+} \cos \Delta \Big).
$$
Recalling the estimate of  $T_{\mathrm{hit}}$  in Step I, we have
$$
\rho_ {k} (T _ {\mathrm {h i t}}) \leq \frac {\kappa_ {1}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} T _ {\mathrm {h i t}}.
$$
As the proof of Step II in the Proof of Theorem C.7, for any  $T_{\mathrm{hit}} < t \leq T_{\mathrm{I}}$ , we have
$$
\begin{array}{l} \rho_ {k} (t) = \rho_ {k} (T _ {\mathrm {h i t}}) + \int_ {T _ {\mathrm {h i t}}} ^ {t} \frac {\mathrm {d} \rho_ {k} (s)}{\mathrm {d} s} \mathrm {d} s \leq \frac {\kappa_ {1}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} T _ {\mathrm {h i t}} + \frac {\kappa_ {2} e ^ {f _ {-} (t)} \sin \Delta}{\sqrt {m} (1 + p)} (t - T _ {\mathrm {h i t}}) \\ \leq \frac {\kappa_ {1}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} T _ {\mathrm {h i t}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} (t - T _ {\mathrm {h i t}}) = \frac {\kappa_ {1}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} t. \\ \end{array}
$$
Combining the estimate in Step I, for any  $0 < t \leq T_1$ , we have:
$$
\rho_ {k} (t) \leq \frac {\kappa_ {1}}{\sqrt {m}} + \frac {1 . 1 1 \kappa_ {2} \sin \Delta}{\sqrt {m} (1 + p)} t.
$$
$$
\rho_ {k} (T _ {1}) \leq \frac {\kappa_ {1}}{\sqrt {m}} + \frac {1 1 . 1 \sqrt {\kappa_ {1} \kappa_ {2}} \sin \Delta}{\sqrt {m} (1 + p)} \leq \frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left(\sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} + 1 1. 1 \frac {\Delta}{1 + p}\right).
$$
Similar to the proof of Step II in the Proof of Theorem C.7, we have the estimate of the dynamics of  $\langle \pmb{w}_k(t),\pmb{x}_{-}\rangle$
$$
\frac {\mathrm {d} \left\langle \boldsymbol {w} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} \geq \frac {0 . 8 9 \kappa_ {2}}{(1 + p) \kappa_ {1} + 1 . 1 1 \kappa_ {2} t \sin \Delta} \Big (\sin^ {2} \Delta - \left\langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {-} \right\rangle^ {2} \Big), \quad 0 <   t \leq T _ {\mathrm {I}},
$$
$$
0 <   \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle <   \sin \Delta .
$$
In the same way, we can derive
$$
\langle \pmb {w} _ {k} (T _ {\mathrm {I}}), \pmb {x} _ {-} \rangle > \left(1 - \frac {2}{\left(1 + 1 1 . 1 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}} \frac {\sin \Delta}{1 + p}\right) ^ {1 . 6} + 1}\right) \sin \Delta
$$
$$
\left\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {+} ^ {\perp} \right\rangle = \frac {1}{\sin \Delta} \left\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {-} \right\rangle > 1 - \frac {2}{\left(1 + 1 1 . 1 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}} \frac {\sin \Delta}{1 + p}\right) ^ {1 . 6} + 1}.
$$
Lemma C.9 (Negative,  $\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{-}$
For negative neuron  $k \in \{k \in [m] - [m/2] : \pmb{w}_k(0) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^-\}$ , it keeps dead:
$$
\boldsymbol {w} _ {k} (t) \in \mathcal {M} _ {+} ^ {-} \cap \mathcal {M} _ {-} ^ {-}, f o r a n y t \geq T _ {1} > \mathcal {O} \left(\frac {\kappa_ {1} \Delta}{\kappa_ {2}}\right).
$$
We do the following analysis for any  $k \in \{k \in [m] - [m/2] : \pmb{w}_k(0) \in \mathcal{M}_+^+ \cap \mathcal{M}_-^-\}$ , i.e.  $\mathrm{s}_k = -1$ ,  $\langle \pmb{w}_k(0), \pmb{x}_+ \rangle > 0$ , and  $\langle \pmb{w}_k(0), \pmb{x}_- \rangle \leq 0$ .
and we aim to estimate  $T_{\mathrm{hit}}$  and prove  $\pmb {w}_k(T_{\mathrm{hit}})\in \mathcal{M}_+^-\cap \mathcal{M}_-^-$
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+}, t \leq T _ {\text {h i t}}.
$$
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \right\rangle}{\mathrm {d} t} = \left\langle - \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+}, \boldsymbol {x} _ {+} \right\rangle = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)}, \\ \frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {-} \right\rangle}{\mathrm {d} t} = \left\langle - \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \boldsymbol {x} _ {+}, \boldsymbol {x} _ {-} \right\rangle = - \frac {\kappa_ {2}}{\sqrt {m}} \frac {p}{1 + p} e ^ {- f _ {+} (t)} \cos \Delta . \\ \end{array}
$$
It is clear  $\frac{\mathrm{d}\langle\pmb{b}_k(t),\pmb{x}_{-}\rangle}{\mathrm{d}t} < 0$ , so  $\langle \pmb {b}_k(t),\pmb {x}_{-}\rangle < \langle \pmb {b}_k(0),\pmb {x}_{-}\rangle \leq 0$  for any  $t\leq T_{\mathrm{hit}}$ . If we denote
$$
T _ {\text {h i t}, +} := \inf  \left\{t \in (0, T _ {1} ]: \langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq 0 \right\},
$$
$$
T _ {\text {h i t}} = T _ {\text {h i t , +}}.
$$
So we only need to estimate  $T_{\mathrm{hit, + }}$ . Due to  $T_{\mathrm{hit}} \leq T_{\mathrm{I}} \leq T_{\mathrm{init}}$  and Lemma C.2, for any  $t \leq T_{\mathrm{hit}}$ , we have  $\left|e^{-y_if_i(t)} - 1\right| \leq 0.11$ . Then for any  $t \leq T_{\mathrm{hit}}$ , we have:
$$
\frac {\mathrm {d} \left\langle \boldsymbol {b} _ {k} (t) , \boldsymbol {x} _ {+} \right\rangle}{\mathrm {d} t} \leq - \frac {0 . 8 9 \kappa_ {2} p}{\sqrt {m} (1 + p)}.
$$