SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recalling  $\langle \pmb{w}_k(0),\pmb{x}_+ \rangle > 0$  and  $\langle \pmb{w}_k(0),\pmb{x}_- \rangle \leq 0$ , with the help of Lemma I.2, we have  $\langle \pmb{w}_k(0),\pmb{x}_+ \rangle \leq \sin \Delta$ . Combining the two estimates, we have:
$$
\begin{array}{l} \langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {+} \rangle \leq \langle \boldsymbol {b} _ {k} (0), \boldsymbol {x} _ {+} \rangle - \int_ {0} ^ {t} \frac {0 . 8 9 \kappa_ {2} p}{\sqrt {m} (1 + p)} d t \\ \leq \rho_ {k} (0) \sin \Delta - \frac {0 . 8 9 \kappa_ {2} p}{\sqrt {m} (1 + p)} t = \frac {\kappa_ {1} \sin \Delta}{\sqrt {m}} - \frac {0 . 8 9 \kappa_ {2} p}{\sqrt {m} (1 + p)} t. \\ \end{array}
$$
$$
T _ {\mathrm {h i t}} = T _ {\mathrm {h i t}, +} \leq \frac {(1 + p) \sin \Delta}{0 . 8 9 p} \frac {\kappa_ {1}}{\kappa_ {2}} <   T _ {\mathrm {I}} = 1 0 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}}.
$$
Lemma C.10 (Negative,  $\mathcal{M}_{+}^{-}\cap \mathcal{M}_{-}^{-}$
For negative neuron  $k \in \{k \in [m] - [m/2] : \pmb{w}_k(0) \in \mathcal{M}_+^- \cap \mathcal{M}_-^-\}$ , it keeps dead:  $\pmb{w}_k(t) \in \mathcal{M}_+^- \cap \mathcal{M}_-^+$  for any  $t \geq 0$ .
Proof of Lemma C.10. Due to Lemma B.2, this lemma is trivial.
C.3 Initialization Estimation and Proof of Theorem 4.1
To get the number of neurons in the eight classes in the subsection above, we also need to estimate the initial positions of these neurons under the random initialization.
Lemma C.11 (Initialization Estimation).
If  $m = \Omega\left(\log(1/\delta)\right)$ , then with probability at least  $1 - \delta$ , we have:
$$
\begin{array}{l} \left| \# \left\{k \in [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle > 0 \right\} - \frac {m}{4} \right| \leq 0. 0 4 m, \\ \# \left\{k \in [ m ] - [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > 0, \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > A \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle \right\} \geq 0. 0 7 5 m, \\ \# \left\{k \in [ m ] - [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > 0, \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle \leq B \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle \right\} \leq 0. 2 0 5 m. \\ \end{array}
$$
where  $A = \frac{(1 + 4.73\kappa_1\kappa_2)p\cos\Delta - (1 - 4.73\kappa_1\kappa_2)}{(1 - 4.73\kappa_1\kappa_2)p - (1 + 4.73\kappa_1\kappa_2)\cos\Delta}$  and  $B = \frac{(1 + 4.73\kappa_1\kappa_2)p - (1 - 4.73\kappa_1\kappa_2)\cos\Delta}{(1 - 4.73\kappa_1\kappa_2)p\cos\Delta - (1 + 4.73\kappa_1\kappa_2)}$  (mentioned in Lemma C.7).
Proof of Lemma C.11.
(i) By Hoeffding's Inequality (Lemma I.1), for any  $\epsilon > 0$  we have:
$$
\begin{array}{l} \mathbb {P} \left(\left| \# \left\{k \in [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle > 0 \right\} - \frac {m}{4} \right| \geq \frac {m \epsilon}{2}\right) = \mathbb {P} \left(\left| \frac {2}{m} \sum_ {k \in [ m / 2 ]} \mathbb {I} \left\{\langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle > 0 \right\} - \frac {1}{2} \right| \geq \epsilon\right) \\ = \mathbb {P} \left(\left| \frac {2}{m} \sum_ {k \in [ m / 2 ]} \mathbb {I} \left\{\langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle > 0 \right\} - \mathbb {E} \left[ \mathbb {I} \left\{\langle \boldsymbol {w} _ {1} (0), \boldsymbol {x} _ {+} \rangle > 0 \right\} \right] \right| \geq \epsilon\right) \\ \leq 2 \exp \Big (- \frac {2 (\frac {m}{2}) ^ {2} \epsilon^ {2}}{\frac {m}{2}} \Big) = 2 \exp (- m \epsilon^ {2}). \\ \end{array}
$$
(ii) From  $\boldsymbol{w}_k(0) \sim \mathbb{U}(\mathbb{S}^{d-1})$ , without loss of generality, we can let  $\boldsymbol{x}_- = \boldsymbol{e}_1$  and  $\boldsymbol{x}_+ = \boldsymbol{e}_1 \cos \Delta + \boldsymbol{e}_2 \sin \Delta$ .
$$
\begin{array}{l} \left\{k \in [ m ] - [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > 0, \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > A \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle \right\} \\ = \left\{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, w _ {k, 1} (0) > A \left(w _ {k, 1} (0) \cos \Delta + w _ {k, 2} (0) \sin \Delta\right) \right\} \\ = \Bigl \{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, (1 - A \cos \Delta) w _ {k, 1} (0) > A w _ {k, 2} (0) \sin \Delta \Bigr \}. \\ \end{array}
$$
From (6), we have  $A > 0$  and
$$
\begin{array}{l} A = 1 + \frac {\left(\left(1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}\right) \cos \Delta - \left(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}\right)\right) (p + 1)}{\left(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}\right) p - \left(1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}\right) \cos \Delta} \\ \leq 1 + \frac {4 . 7 3 \kappa_ {1} \kappa_ {2} (1 + \cos \Delta)}{1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}} \frac {p + 1}{p - \frac {1 0}{9}} \leq 1 + \frac {9 . 4 6 \kappa_ {1} \kappa_ {2}}{1 - \frac {1}{1 9}} \frac {9 0}{7 1} \leq 1 + 1 2. 6 6 \kappa_ {1} \kappa_ {2}, \\ A = 1 + \frac {\left(\left(1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}\right) \cos \Delta - \left(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}\right)\right) (p + 1)}{\left(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}\right) p - \left(1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}\right) \cos \Delta} \\ \geq 1 - \frac {4 . 7 3 \kappa_ {1} \kappa_ {2} (1 + \cos \Delta) + (1 - \cos \Delta)}{1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}} \frac {p + 1}{p - \frac {1 0}{9}} \geq 1 - \frac {\frac {2}{1 9} + \frac {\Delta^ {2}}{2}}{1 - \frac {1}{1 9}} \frac {9 0}{7 1} \\ \geq 1 - \frac {\frac {2}{1 9} + \frac {1}{1 9}}{\frac {1 8}{1 9}} \frac {9 0}{7 1} \geq 0. 7 8, \\ \end{array}
$$
$$
\frac {1 - A \cos \Delta}{A \sin \Delta} \geq \frac {1 - A}{A \sin \Delta} \geq - \frac {1 2 . 6 6 \kappa_ {1} \kappa_ {2}}{A \sin \Delta} \geq - \frac {1 2 . 6 6 \kappa_ {1} \kappa_ {2}}{0 . 7 8 \sin \Delta} \geq - \frac {1 2 . 6 6 \kappa_ {1} \kappa_ {2}}{0 . 7 8 \frac {2}{\pi} \Delta} \geq - \frac {2 5 . 5 \kappa_ {1} \kappa_ {2}}{\Delta} \geq - \frac {1}{1 0 0}.
$$
For simplicity, we denote the event
$$
A _ {k} := \left\{w _ {k, 1} (0) > 0, - \frac {1}{1 0 0} w _ {k, 1} (0) > w _ {k, 2} (0) \right\}, k \in [ m ] - [ m / 2 ].
$$
Then we have the estimate:
$$
\begin{array}{l} \# \left\{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, (1 - A \cos \Delta) w _ {k, 1} (0) > A w _ {k, 2} (0) \sin \Delta \right\} \\ \geq \# \left\{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, - \frac {1}{1 0 0} w _ {k, 1} (0) > w _ {k, 2} (0) \right\} = \sum_ {k \in [ m ] - [ m / 2 ]} \mathbb {I} \{A _ {k} \}. \\ \end{array}
$$
We first estimate the lower bound for  $\mathbb{E}[\mathbb{I}\{A_m\} ]$
$$
\mathbb {E} \left[ \mathbb {I} \left\{A _ {m} \right\} \right] = \mathbb {P} (A _ {m}) = \mathbb {P} \left(w _ {m, 1} (0) > 0, - \frac {1}{1 0 0} w _ {m, 1} (0) > w _ {k, 2} (0)\right)
$$
$$
\begin{array}{l} \mathbf {g} \sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {d}) \mathbb {P} \left(\frac {g _ {1}}{\| \boldsymbol {g} \|} > 0, - \frac {1}{1 0 0} \frac {g _ {1}}{\| \boldsymbol {g} \|} > \frac {g _ {2}}{\| \boldsymbol {g} \|}\right) = \mathbb {P} \left(g _ {1} > 0, g _ {1} <   - 1 0 0 g _ {2}\right) = \mathbb {P} \left(g _ {1} > 0, g _ {1} <   1 0 0 g _ {2}\right) \\ = \mathbb {P} \left(1 0 0 g _ {2} > g _ {1} > 0\right) \geq \sup  _ {t > 0} \mathbb {P} \left(g _ {2} > t, 1 0 0 t > g _ {1} > 0\right) ^ {g \sim \mathcal {N} (0, 1)} \sup  _ {t > 0} \mathbb {P} \left(g > t\right) \mathbb {P} \left(1 0 0 t > g > 0\right) \\ \geq \mathbb {P} \left(g > \frac {1}{1 0}\right) \mathbb {P} \left(1 0 > g > 0\right) \geq 0. 2 3. \\ \end{array}
$$
Secondly, by Hoeffding's inequality (Lemma I.1), for any  $\epsilon > 0$ , we have
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {k \in [ m ] - [ m / 2 ]} \mathbb {I} \{A _ {k} \} - 0. 1 1 5 m \leq - \frac {m}{2} \epsilon\right) \leq \mathbb {P} \left(\sum_ {k \in [ m ] - [ m / 2 ]} \mathbb {I} \{A _ {k} \} - \frac {m}{2} \mathbb {E} \left[ \mathbb {I} \{A _ {m} \} \right] \leq - \frac {m}{2} \epsilon\right) \\ = \mathbb {P} \left(\frac {2}{m} \sum_ {k \in [ m ] - [ m / 2 ]} \mathbb {I} \{A _ {k} \} - \mathbb {E} \left[ \mathbb {I} \{A _ {m} \} \right] \leq - \epsilon\right) \leq \exp \left(- \frac {2 (\frac {m}{2}) ^ {2} \epsilon^ {2}}{\frac {m}{2}}\right) = \exp (- m \epsilon^ {2}) \\ \end{array}
$$
(iii) This proof is similar to (ii). From  $\pmb{w}_k(0) \sim \mathbb{U}(\mathbb{S}^{d-1})$ , without loss of generality, we can let  $\pmb{x}_{-} = \pmb{e}_1$  and  $\pmb{x}_{+} = \pmb{e}_1 \cos \Delta + \pmb{e}_2 \sin \Delta$ .
$$
\begin{array}{l} \left\{k \in [ m ] - [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > 0, \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle \leq B \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle \right\} \\ = \left\{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, w _ {k, 1} (0) \cos \Delta + w _ {k, 2} (0) \sin \Delta \leq B w _ {k, 1} (0) \right\} \\ = \Big \{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, (B - \cos \Delta) w _ {k, 1} (0) > w _ {k, 2} (0) \sin \Delta \Big \}. \\ \end{array}
$$
From (6), we have  $B > 0$  and
$$
\begin{array}{l} B - \cos \Delta = \frac {(1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}) p - (1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) \cos \Delta}{(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p \cos \Delta - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2})} - \cos \Delta \\ = \frac {\left(1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}\right) (p + \cos \Delta) - \left(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}\right) (1 + p \cos \Delta) \cos \Delta}{\left(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}\right) p \cos \Delta - \left(1 + 4 . 7 3 \kappa_ {1} \kappa_ {2}\right)} \\ = \frac {p \sin^ {2} \Delta + 4 . 7 3 \kappa_ {1} \kappa_ {2} (p + 2 \cos \Delta + p \cos^ {2} \Delta)}{(1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}) p \cos \Delta - (1 + 4 . 7 3 \kappa_ {1} \kappa_ {2})} \leq \frac {\sin^ {2} \Delta + 9 . 4 6 \kappa_ {1} \kappa_ {2}}{1 - 4 . 7 3 \kappa_ {1} \kappa_ {2}} \frac {p + 1}{p \cos \Delta - \frac {1 0}{9}} \\ \leq \frac {\sin^ {2} \Delta + 9 . 4 6 \frac {\Delta}{2 5 5 0}}{1 - \frac {1}{1 9}} \frac {p + 1}{\frac {9}{1 0} p - \frac {1 0}{9}} \leq \frac {\sin^ {2} \Delta + 9 . 4 6 \frac {\pi \sin \Delta}{5 1 0 0}}{1 - \frac {1}{1 9}} \frac {p + 1}{\frac {9}{1 0} p - \frac {1 0}{9}} \\ \leq \frac {0 . 3 1 5 + \frac {9 . 4 6 \pi}{5 1 0 0}}{\frac {1 8}{1 9}} \frac {1 0}{\frac {8 1}{1 0} - \frac {1 0}{9}} \sin \Delta \leq \frac {\sin \Delta}{2}, \\ \end{array}
$$
$$
B _ {k} := \left\{w _ {k, 1} (0) > 0, \frac {1}{2} w _ {k, 1} (0) > w _ {k, 2} (0) \right\}, k \in [ m ] - [ m / 2 ].
$$
$$
\begin{array}{l} \# \left\{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, (B - \cos \Delta) w _ {k, 1} (0) > w _ {k, 2} (0) \sin \Delta \right\} \\ \leq \# \left\{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, \frac {1}{2} w _ {k, 1} (0) > w _ {k, 2} (0) \right\} = \sum_ {k \in [ m ] - [ m / 2 ]} \mathbb {I} \left\{B _ {k} \right\}. \\ \end{array}
$$
We first estimate the lower bound for  $\mathbb{E}[\mathbb{I}\{B_m\} ]$
$$
\mathbb {E} \left[ \mathbb {I} \left\{B _ {m} \right\} \right] = \mathbb {P} (B _ {m}) = \mathbb {P} \left(w _ {m, 1} (0) > 0, \frac {1}{2} w _ {m, 1} (0) > w _ {k, 2} (0)\right)
$$
$$
\begin{array}{l} \mathbf {g} ^ {\sim \mathcal {N} (\mathbf {0}, \mathbf {I} _ {d})} \mathbb {P} \left(\frac {g _ {1}}{\| \boldsymbol {g} \|} > 0, \frac {1}{2} \frac {g _ {1}}{\| \boldsymbol {g} \|} > \frac {g _ {2}}{\| \boldsymbol {g} \|}\right) = \mathbb {P} \left(g _ {1} > 0, g _ {1} > 2 g _ {2}\right) = \mathbb {P} \left(g _ {1} > 0, g _ {2} \leq 0\right) + \mathbb {P} \left(g _ {1} > 2 g _ {2} > 0\right) \\ = \frac {1}{4} + \mathbb {P} \left(g _ {1} > 2 g _ {2} > 0\right) \leq \frac {1}{4} + \frac {1}{2 \pi} \int_ {0} ^ {+ \infty} e ^ {- \frac {x ^ {2}}{2}} \int_ {0} ^ {\frac {x}{2}} e ^ {- \frac {y ^ {2}}{2}} d y d x \leq \frac {1}{4} + \frac {1}{2 \pi} \int_ {0} ^ {+ \infty} \frac {x}{2} e ^ {- \frac {x ^ {2}}{2}} d x \\ \leq \frac {1}{4} + \frac {1}{4 \pi}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {k \in [ m ] - [ m / 2 ]} \mathbb {I} \{B _ {k} \} - \left(\frac {1}{8} + \frac {1}{8 \pi}\right) m \geq \frac {m}{2} \epsilon\right) \leq \mathbb {P} \left(\sum_ {k \in [ m ] - [ m / 2 ]} \mathbb {I} \{B _ {k} \} - \frac {m}{2} \mathbb {E} \left[ \mathbb {I} \{B _ {m} \} \right] \geq \frac {m}{2} \epsilon\right) \\ = \mathbb {P} \left(\frac {2}{m} \sum_ {k \in [ m ] - [ m / 2 ]} \mathbb {I} \{B _ {k} \} - \mathbb {E} \left[ \mathbb {I} \{B _ {m} \} \right] \geq \epsilon\right) \leq \exp \left(- \frac {2 (\frac {m}{2}) ^ {2} \epsilon^ {2}}{\frac {m}{2}}\right) = \exp (- m \epsilon^ {2}). \\ \end{array}
$$
let  $\epsilon = 0.08$  and  $\delta = 4\exp (-m\epsilon^2 /2)$ . Combining the uniform bounds in (i)(ii)(iii), we obtain this theorem:
If  $m \geq \frac{2\log(4 / \delta)}{0.08^2}$ , then with probability at least  $1 - \delta$ , we have:
$$
\begin{array}{l} \left| \# \left\{k \in [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle > 0 \right\} - \frac {m}{4} \right| \leq 0. 0 4 m, \\ \# \left\{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, (1 - A \cos \Delta) w _ {k, 1} (0) > A w _ {k, 2} (0) \sin \Delta \right\} \geq 0. 0 7 5 m, \\ \# \left\{k \in [ m ] - [ m / 2 ]: w _ {k, 1} (0) > 0, (B - \cos \Delta) w _ {k, 1} (0) > w _ {k, 2} (0) \sin \Delta \right\} \leq 0. 2 0 5 m. \\ \end{array}
$$
So far, Lemma C.3, C.4, C.5, C.6, C.7, C.8, C.9, C.10 characterize the training dynamics of each neuron in Phase I, and Lemma C.11 estimate the initial positions of the neurons. Now we can prove our main theorem in Phase I.
Theorem C.12 (Restatement of Theorem 4.1).
Under the data Assumption 3.1, let the two-layer network trained by Gradient Flow (2) starting from random initialization. Let the width  $m = \Omega (\log (1 / \delta))$ , the initialization scales satisfy (6). Then with probability at least  $1 - \delta$ , the following results (S1)~(S5) hold at the end of Phase I ( $T_{\mathrm{I}} = 10\sqrt{\frac{\kappa_1}{\kappa_2}}$ ):
(S1). For positive neurons  $k \in [m/2]$  ( $s_k = 1$ ), let  $\mathcal{K}_+$  be the index set of living neurons, i.e.  $\mathcal{K}_+ := \{k \in [m/2] : \pmb{w}_k(T_1) \in \mathcal{M}_+^+ \cup \mathcal{M}_-^+\}$ . Then  $0.21m \leq |\mathcal{K}_| \leq 0.29m$ . Moreover, for any neuron  $k \in \mathcal{K}_+$ , it has the following properties (P1)(P2).
(P1). Its norm is small but significant:  $\frac{4.66\sqrt{\kappa_1\kappa_2}}{\sqrt{m}} \leq \rho_k(T_{\mathrm{I}}) \leq \frac{12\sqrt{\kappa_1\kappa_2}}{\sqrt{m}}$ .
(P2). Its direction is strongly aligned with  $\mu$  :
$$
\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {\mu} \rangle \geq \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \left(1 - \frac {2}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right).
$$
(S2). For negative neurons  $k \in [m] - [m/2]$  ( $s_k = -1$ ), let  $\mathcal{K}_{-}$  be the index set of living neurons, i.e.  $\mathcal{K}_{-} := \{k \in [m] - [m/2] : \pmb{w}_k(T_1) \in \mathcal{M}_+^+ \cup \mathcal{M}_-^+\}$ . Then  $0.075m \leq |\mathcal{K}_{-}| \leq 0.205m$ . Moreover, for any neuron  $k \in \mathcal{K}_{-}$ , it has the following properties (N1)(N2)(N3).
(N1). Its norm is tiny:  $\rho_{k}(T_{\mathrm{I}})\leq \frac{\sqrt{\kappa_{1}\kappa_{2}}}{\sqrt{m}}\Big(4.3\sqrt{\frac{\kappa_{1}}{\kappa_{2}}} +\frac{11.1\sin\Delta}{1 + p}\Big)$
(N2). It lies on a manifold perpendicular to  $\pmb{x}_{+}:\pmb{w}_{k}(t)\in \mathcal{M}_{+}^{0}\cap \mathcal{M}_{-}^{+}$
(N3). Its direction is weakly aligned with  $\pmb{x}_{+}^{\perp}:\left\langle \pmb{w}_{k}(T_{\mathrm{I}}),\pmb{x}_{+}^{\perp}\right\rangle >1 - \frac{2}{\left(1 + 2.32\sqrt{\frac{\kappa_{2}}{\kappa_{1}}}\frac{\sin\Delta}{1 + p}\right)^{1.6} + 1}$
(S3). For other neurons  $k \notin \mathcal{K}_{+} \cup \mathcal{K}_{-}$ , it will remain dead forever:
$$
\boldsymbol {w} _ {k} \left(T _ {\mathrm {I}}\right) \in \mathcal {M} _ {+} ^ {-} \cap \mathcal {M} _ {-} ^ {-}, \quad \boldsymbol {b} _ {k} (t) \equiv \boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right), \quad \forall t \in [ T _ {\mathrm {I}}, + \infty)
$$
(S4). The predictions for  $\pmb{x}_{+}$  and  $\pmb{x}_{-}$  have the estimate:
$$
0. 9 7 8 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \left(\left(\frac {p - 1}{p + 1}\right) ^ {2} - 0. 1 1\right) \leq f _ {+} (T _ {\mathrm {I}}) \leq 3. 8 5 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}},
$$
$$
0. 9 4 7 \left(\left(\frac {p - 1}{p + 1}\right) ^ {2} \cos \Delta - 0. 2\right) \leq f _ {-} \left(T _ {\mathrm {I}}\right) \leq 3. 8 5 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}},
$$
and the training accuracy is  $\mathrm{Acc}(T_{\mathrm{I}}) = \frac{p}{1 + p}$ .
(S5).  $0 < 0.258 \leq \frac{0.075}{0.29} \leq \frac{|\mathcal{K}_{-}|}{|\mathcal{K}_{+}|} \leq \frac{0.205}{0.21} \leq 0.977 < 1$ .
Proof of Theorem C.12.
This theorem is a corollary of Lemma C.3, Lemma C.4, Lemma C.5, Lemma C.6, Lemma C.7, Lemma C.8, Lemma C.9, Lemma C.10, and Lemma C.11. We focus on the end of Phase I:  $T_{\mathrm{I}} = 10\sqrt{\frac{\kappa_1}{\kappa_2}}$ .
Proof of (S1)(S2). From Lemma C.11, we know that: if  $m = \Omega$  ( $\log(1/\delta)$ ), then with probability at least  $1 - \delta$ , we have:
$$
\left| \# \left\{k \in [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle > 0 \right\} - \frac {m}{4} \right| \leq 0. 0 4 m,
$$
$$
\# \left\{k \in [ m ] - [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > 0, \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > A \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle \right\} \geq 0. 0 7 5 m,
$$
$$
\# \left\{k \in [ m ] - [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > 0, \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle \leq B \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle \right\} \leq 0. 2 0 5 m.
$$
where  $A = \frac{(1 + 4.73\kappa_1\kappa_2)p\cos\Delta - (1 - 4.73\kappa_1\kappa_2)}{(1 - 4.73\kappa_1\kappa_2)p - (1 + 4.73\kappa_1\kappa_2)\cos\Delta}$  and  $B = \frac{(1 + 4.73\kappa_1\kappa_2)p - (1 - 4.73\kappa_1\kappa_2)\cos\Delta}{(1 - 4.73\kappa_1\kappa_2)p\cos\Delta - (1 + 4.73\kappa_1\kappa_2)}$
Recalling the dynamics analysis in Lemma C.3, C.4, C.5, and C.6, we have:
$$
0. 2 1 m \leq | \mathcal {K} _ {+} | = \# \left\{k \in [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle > 0 \right\} \leq 0. 2 9 m.
$$
Recalling the dynamics analysis in Lemma C.7, C.8 C.9, and C.10, we have:
$$
\left| \mathcal {K} _ {-} \right| \geq \# \left\{k \in [ m ] - [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > 0, \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > A \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle \right\} \geq 0. 0 7 5 m,
$$
$$
\left| \mathcal {K} _ {-} \right| \leq \# \left\{k \in [ m ] - [ m / 2 ]: \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle > 0, \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {+} \rangle \leq B \langle \boldsymbol {w} _ {k} (0), \boldsymbol {x} _ {-} \rangle \right\} \leq 0. 2 0 5 m.
$$
Moreover, the estimates in Lemma C.3, C.4, C.5, and C.6 ensure that for any  $k \in \mathcal{K}_+$ , the following results hold:
$$
\begin{array}{l} \langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {\mu} \rangle \geq \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}}\right) \left(1 - \frac {2}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right); \\ \frac {4 . 6 6 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \leq \rho_ {k} (T _ {\mathrm {I}}) \leq \frac {1 2 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}}. \\ \end{array}
$$
Similarly, the estimates in Lemma C.7, C.8 C.9, and C.10 ensure that for any  $k \in \mathcal{K}_{-}$ , the following results hold:
$$
\begin{array}{l} \rho_ {k} \left(T _ {\mathrm {I}}\right) \leq \frac {\sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left(4. 3 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}} + \frac {1 1 . 1 \sin \Delta}{1 + p}\right); \\ \boldsymbol {w} _ {k} \left(T _ {\mathrm {I}}\right) \in \mathcal {M} _ {+} ^ {0} \cap \mathcal {M} _ {-} ^ {+}; \\ \left\langle \boldsymbol {w} _ {k} (T _ {1}), \boldsymbol {x} _ {+} ^ {\perp} \right\rangle > 1 - \frac {2}{\left(1 + 2 . 3 2 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}} \frac {\sin \Delta}{1 + p}\right) ^ {1 . 6} + 1}. \\ \end{array}
$$
Proof of (S3). A direct corollary of Lemma C.3, C.4, C.5, C.6, C.7, C.8, C.9, C.10.
Proof of (S4). (S4) are direct corollaries of (S1)(S2).
For  $f_{+}(T_{\mathrm{I}})$ , we have the following estimate:
$$
\begin{array}{l} f _ {+} \left(T _ {\mathrm {I}}\right) = \sum_ {k \in \mathcal {K} _ {+}} a _ {k} \sigma \left(\boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right) ^ {\top} \boldsymbol {x} _ {+}\right) + \sum_ {k \in \mathcal {K} _ {-}} a _ {k} \sigma \left(\boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right) ^ {\top} \boldsymbol {x} _ {+}\right) \\ = \sum_ {k \in \mathcal {K} _ {+}} a _ {k} \sigma \left(\boldsymbol {b} _ {k} \left(T _ {\mathrm {I}}\right) ^ {\top} \boldsymbol {x} _ {+}\right) + 0 = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \rho_ {k} \left(T _ {\mathrm {I}}\right) \sigma \left(\boldsymbol {w} _ {k} \left(T _ {\mathrm {I}}\right) ^ {\top} \boldsymbol {x} _ {+}\right) \\ \geq \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {4 . 6 6 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {+} \right\rangle \geq \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {4 . 6 6 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \Big (\left\langle \boldsymbol {\mu}, \boldsymbol {x} _ {+} \right\rangle - \| \boldsymbol {w} _ {k} (T _ {\mathrm {I}}) - \boldsymbol {\mu} \| \Big) \\ = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {4 . 6 6 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \Big (\| z \| \langle z, x _ {+} \rangle - 2 + 2 \langle w _ {k} (T _ {\mathrm {I}}), \boldsymbol {\mu} \rangle \Big) \\ \geq \left| \mathcal {K} _ {+} \right| \frac {4 . 6 6 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}}}{m} \left(\frac {p - 1}{p + 1} \frac {p - \cos \Delta}{p + 1} - 2 + 2 \left(1 - 4. 2 \sqrt {\kappa_ {1} \kappa_ {2}} - \frac {2}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right)\right) \\ \geq 0. 2 1 \cdot 4. 6 6 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \left(\frac {p - 1}{p + 1} \frac {p - \cos \Delta}{p + 1} - 8. 4 \sqrt {\kappa_ {1} \kappa_ {2}} - \frac {4}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right) \\ \geq 0. 9 7 8 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \left(\left(\frac {p - 1}{p + 1}\right) ^ {2} - 8. 4 \sqrt {\kappa_ {1} \kappa_ {2}} - \frac {4}{1 + 0 . 7 \left(1 + 9 . 9 \sqrt {\frac {\kappa_ {2}}{\kappa_ {1}}}\right) ^ {1 . 1 5}}\right) \\ \stackrel {(6)} {\geq} 0. 9 7 8 \kappa_ {2} \sqrt {\kappa_ {1} \kappa_ {2}} \Bigl (\bigl (\frac {p - 1}{p + 1} \bigr) ^ {2} - 0. 1 1 \Bigr); \\ \end{array}
$$
$$
\begin{array}{l} f _ {+} \left(T _ {\mathrm {I}}\right) = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \rho_ {k} \left(T _ {\mathrm {I}}\right) \sigma \left(\boldsymbol {w} _ {k} \left(T _ {\mathrm {I}}\right) ^ {\top} \boldsymbol {x} _ {+}\right) \\ \leq \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {1 2 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left\langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {x} _ {+} \right\rangle \leq \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {1 2 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \Big (\left\langle \boldsymbol {\mu}, \boldsymbol {x} _ {+} \right\rangle + \| \boldsymbol {w} _ {k} (T _ {\mathrm {I}}) - \boldsymbol {\mu} \| \Big) \\ = \sum_ {k \in \mathcal {K} _ {+}} \frac {\kappa_ {2}}{\sqrt {m}} \frac {1 2 \sqrt {\kappa_ {1} \kappa_ {2}}}{\sqrt {m}} \left(\| \boldsymbol {z} \| \langle \boldsymbol {z}, \boldsymbol {x} _ {+} \rangle + 2 - 2 \langle \boldsymbol {w} _ {k} (T _ {\mathrm {I}}), \boldsymbol {\mu} \rangle\right) \\ \end{array}
$$