SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In summary, our whole four-phase optimization dynamics capture more nonlinear behaviors than these works. Furthermore, we conduct a more thorough and detailed theoretical analysis of these nonlinear behaviors, providing a more systematic and comprehensive understanding.
In this work, we study the optimization dynamics of ReLU neural networks trained by GF on a linearly separable data. Our analysis captures the whole optimization process starting from random initialization to final convergence. Throughout the whole training process, we reveal four different phases and identify rich nonlinear behaviors theoretically. However, theoretical understanding of the training of NNs still has a long way to go. For instance, although we conduct a fine-grained analysis of GF, the dynamics of GD are more complex and exhibit other nonlinear behaviors such as progressive sharpening and edge of stability (Wu et al., 2018; Jastrzewski et al., 2019; Cohen et al., 2021; Ma et al., 2022; Li et al., 2022; Damian et al., 2022; Zhu et al., 2022; Ahn et al., 2022a,b). Additionally, unlike GD, SGD uses only mini-batches of data and injects noise (Zhu et al., 2019; Thomas et al., 2020; Feng and Tu, 2021; Liu et al., 2021; Ziyin et al., 2022; Wu et al., 2022; Wojtowitsch, 2023; Wang and Wu, 2023) in each iteration, which can have a pronounced impact on the optimization dynamics and nonlinear behaviors. Better understanding of the nonlinear behaviors during GD or SGD training is an important direction of future work.
We thank Prof. Weinan E, Prof. Lei Wu, Prof. Zhi-Qin John Xu and anonymous reviewers for helpful suggestions. Mingze Wang is supported in part by the National Key Basic Research Program of China: 2015CB856000.
Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pages 4782-4887. PMLR, 2022a.
Emmanuel Abbe, Elisabetta Cornacchia, Jan Hazla, and Christopher Marquis. An initial alignment between neural network and target is needed for gradient descent to learn. In International Conference on Machine Learning, pages 33-52. PMLR, 2022b.
Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. arXiv preprint arXiv:2302.11055, 2023.
Kwangjun Ahn, Sébastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via the" edge of stability". arXiv preprint arXiv:2212.07469, 2022a.
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. In International Conference on Machine Learning, pages 247-257. PMLR, 2022b.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322-332. PMLR, 2019.
Devansh Arpit, Stanisław Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242. PMLR, 2017.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. International Conference on Learning Representations, 2022.
Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an Ornstein-uhlenbeck like process. In Conference on learning theory, pages 483–513. PMLR, 2020.
Jérôme Bolte, Aris Daniilidis, Olivier Ley, and Laurent Mazet. Characterizations of lojasiewicz inequalities: subgradient flows, talweg, convexity. Transactions of the American Mathematical Society, 362(6):3319-3363, 2010.
Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. Advances in Neural Information Processing Systems, 2022.
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. Sgd learns overparameterized networks that provably generalize on linearly separable data. arXiv preprint arXiv:1710.10174, 2017.
Niladri S Chatterji, Philip M Long, and Peter L Bartlett. When does gradient descent with logistic loss find interpolating two-layer networks? Journal of Machine Learning Research, 22(159):1-48, 2021.
Zhengan Chen, Yuqing Li, Tao Luo, Zhangchen Zhou, and Zhi-Qin John Xu. Phase diagram of initial condensation for two-layer neural networks. arXiv preprint arXiv:2303.06561, 2023.
Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, and Rong Ge. Understanding edge-of-stability training dynamics with a minimalist example. arXiv preprint arXiv:2210.03294, 2022.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference on Machine Learning, pages 7654-7663. PMLR, 2019.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820-29834, 2021.
Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of minibatch noise in SGD. In International Conference on Learning Representations, 2022.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks. arXiv preprint arXiv:1811.08888, 2018.
A Experimental Details 17
B Proof Preparation 20
C Proofs of Optimization Dynamics in Phase I 22
D Proofs of Optimization Dynamics in Phase II 47
E Proofs of Optimization Dynamics in Phase III 66
F Proofs of Optimization Dynamics in Phase IV 79
G Clarke Subdifferential and KKT Conditions for Non-smooth Optimization 91
H Solution of Discontinuous System 92
I Some Basic Inequalities 93
All experiments are conducted on a MacBook pro 13 (M2) only using CPU. See the code at https://github.com/wmz9/Understanding_Multi-phase_Optimization_NeurIPS2023.
A.1 Experiments on standard Dataset
We train the two-layer network on the dataset that satisfies Assumption 3.1 with  $d = 20$ ,  $p = 4$  and  $\Delta = \pi / 15$ . Specifically, we choose the network width  $m = 100$ ; the initialization scale  $\kappa_1 = 0.1$ ,  $\kappa_2 = 1$ ; the small learning rate  $\eta = 0.01$ .
In Figure 3, we show some key data directions in this dataset, as well as the training accuracy, which contains a long plateau. Furthermore, in Figure 4, we provide the evolution of all neurons during training from  $t = 0$  to  $t = 150000$ , which is a more complete version of Figure 1.
Figure 3: (left) Some key data directions: the directions of  $x_{+}, x_{-}, x_{+}^{\perp}, x_{-}^{\perp}, \mu$  are plotted in blue, orange, green, red and pink colors, respectively; (right) The training accuracy.
(c)  $t = 2000$
(e)  $t = 50000$
(f)  $t = 60000$
(d)  $t = 30000$
(g)  $t = 80000$
Figure 4: (A more complete version of Figure 1) These figures visualize (in polar coordinates) the projections of all neurons  $\{b_k(t)\}_{k\in [m]}$  onto the 2d subspace span  $\{\pmb{x}_+, \pmb{x}_-\}$  during training (from  $t = 0$  to  $t = 150000$ ). Each purple star represents a positive neuron ( $k \in [m/2]$ ), while each brown star represents a negative neuron ( $k \in [m] - [m/2]$ ).
(h)  $t = 120000$
(i)  $t = 150000$
A.2 Numerical validation on our theoretical bounds
We conduct experiments to validate our theoretical bounds on  $T_{\mathrm{plat}}$  and  $T_{\mathrm{III}}$  under different  $p$  and  $\Delta$ , and the results are shown in 2.
In the first experiment (1st and 2nd subtable), we fix  $p = 4$  for change  $\Delta$ ; in the second experiment (3rd and 4th subtable), we fix  $\Delta = \pi / 15$  and change  $p$ . As for other hyperparameters (such as  $\kappa_1, \kappa_2, d, m$ ), we keep the same scales as our setups in Appendix A.1.
We have two main conclusions: (1) four training phases in our theory persistently exist; (the same as Fig 1 in Appendix A, and be omitted due to the limited space) (2) our theoretical estimates on  $T_{\mathrm{plat}}$  and  $T_{\mathrm{III}}$  are relatively tight, basically in the same magnitude as the realistic time.
Table 2: The change of our theoretical bounds on  $T_{\mathrm{plat}}$  and  $T_{\mathrm{III}}$  under different  $p$  and  $\Delta$ .
Δ	4π/45	4π/45·3/4 = π/15	4π/45·(3/4)2	4π/45·(3/4)3
Realistic Tplat	1.96 × 10^4	3.68 × 10^4	7.25 × 10^4	12.87 × 10^4
Our estimate Θ(1/Δ²): 950/Δ² + 1520/Δ - 9943	1.90 × 10^4	3.82 × 10^4	7.14 × 10^4	12.89 × 10^4
Δ	4π/45	4π/45·3/4 = π/15	4π/45·(3/4)2	4π/45·(3/4)3
Realistic TIII	4.98 × 10^4	6.14 × 10^4	9.18 × 10^4	15.63 × 10^4
Our estimate Θ(1/Δ²): 1772/Δ² - 12218/Δ + 67621	4.97 × 10^4	6.17 × 10^4	9.16 × 10^4	15.63 × 10^4
p	6	8	10	12
Realistic Tplat	6.14 × 104	9.57 × 104	13.96 × 104	17.61 × 104
Our estimate Θ(p): 19400p - 56400	6.00 × 104	9.88 × 104	13.76 × 104	17.64 × 104
p	6	8	10	12
Realistic TIII	8.92 × 104	13.40 × 104	19.72 × 104	27.68 × 104
Our estimate Θ(p1/1-α cosΔ): 6912p1.5 - 15897	8.59 × 104	14.05 × 104	20.27 × 104	27.14 × 104
A.3 Experiments on Noisy Dataset
We conduct numerical experiments on the setting of adding small stochastic noise on top of  $\pmb{x}_{+}$  and  $\pmb{x}_{-}$ , a little bit more realistic setting. Specifically, in span  $\{\pmb{x}_{+},\pmb{x}_{-}\}$ , we perturb the angles of  $n_{+} - 1$  instances of  $\pmb{x}_{+}$  and  $n_{-} - 1$  instances of  $\pmb{x}_{-}$  using stochastic noise  $\xi \sim \mathrm{Unif}([0,\Delta /4])$ .
In Figure 5, we visualize (i) the evolution of each neuron throughout the training process; (ii) some key data directions; (iii) the evolution of training accuracy.
From the numerical results in Figure 5, we have two main conclusions: (1) we ascertain that the same four-phase optimization dynamics and nonlinear behaviors persist, even for our dataset with small stochastic noise; (2) a slight difference is that there is more than one plateau of training accuracy in Phase II. The reason is that for noisy data, GF needs to learn negative data one by one in Phase II. For example, three distinct negative data are employed in this experiment, so three plateaus of training accuracy emerge (12/15, 13/15, and 14/15).
(a) key data directions
(b) training acc
(c)  $t = 0$
(d)  $t = 200$
(e)  $t = 2000$
(f)  $t = 30000$
(g)  $t = 65000$
(h)  $t = 90000$
(i)  $t = 200000$
(j)  $t = 400000$
Figure 5: The experimental results for noisy data with  $\Delta = \pi /15$ ,  $n_{+} = 12$ ,  $n_{-} = 3$ . Fig (a): Some key data directions, including  $\pmb{x}_{+}$ ,  $\pmb{x}_{+}$  +noise,  $\pmb{x}_{-}$ ,  $\pmb{x}_{-}$  +noise,  $\pmb{\mu}$ ,  $\pmb{x}_{+}^{\perp}$ , and  $\pmb{x}_{-}^{\perp}$ . Fig (b): The evolution of training accuracy. Figs (c)~(j): the evolution of the projections of all neurons  $\{b_k(t)\}_{k\in [m]}$  onto the 2d subspace span  $\{\pmb{x}_{+},\pmb{x}_{-}\}$  during training (from  $t = 0$  to  $t = 400000$ ). Each purple star represents a positive neuron ( $k\in [m / 2]$ ), while each brown star represents a negative neuron ( $k\in [m] - [m / 2]$ ). Four-phase dynamics: from Fig (c) to (d) is Phase I; from Fig (d) to (g) is Phase II; from Fig (g) to (h) is Phase III; from Fig (h) to (j) is Phase IV. To compare these results with noiseless data, please refer to Figures 3 and 4.
B Proof Preparation
Selection of initialization parameters.
For the data satisfying Assumption 3.1, we consider the regime that  $\Delta \ll 1$  with  $p\cos \Delta >1$ . During the entire proof, we select the initialization scale  $\kappa_{1},\kappa_{2}$  as follows:
$$
\kappa_ {2} = \mathcal {O} (1), \quad \frac {\kappa_ {1}}{\kappa_ {2}} = \mathcal {O} (\Delta^ {8}). \tag {3}
$$
Gradient Flow. In general, for any  $k \in [m]$ , the GF dynamics of  $b_{k}(t)$  can be written as
$$
\begin{array}{l} \frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} \in \frac {\partial^ {\circ} \mathcal {L} (\boldsymbol {\theta})}{\partial \boldsymbol {b} _ {k}} = \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m}} \frac {1}{n} \sum_ {i = 1} ^ {n} e ^ {- y _ {i} f _ {i} (t)} \partial^ {\circ} \sigma \left(\left\langle \boldsymbol {b} _ {k} (t), \boldsymbol {x} _ {i} \right\rangle\right) y _ {i} \boldsymbol {x} _ {i} \tag {4} \\ = \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m}} \left(\frac {p}{1 + p} e ^ {- f _ {+} (t)} \partial^ {\circ} \sigma (\langle \pmb {b} _ {k} (t), \pmb {x} _ {+} \rangle) \pmb {x} _ {+} - \frac {1}{1 + p} e ^ {f _ {-} (t)} \partial^ {\circ} \sigma (\langle \pmb {b} _ {k} (t), \pmb {x} _ {-} \rangle) \pmb {x} _ {-}\right), \\ \end{array}
$$
where  $\partial^{\circ}$  is Clarke's subdifferential defined in Definition G.1. Notice that if  $\mathcal{L}$  is continuously differentiable at  $\pmb{\theta}$ , then  $\partial^{\circ}\mathcal{L}(\pmb{\theta}) = \{\nabla \mathcal{L}(\pmb{\theta})\}$  is unique.
However, for discontinuous differentiable points of  $\mathcal{L}$ , the differential inclusion flow  $\frac{\mathrm{d}\pmb{\theta}}{\mathrm{d}t} \in \partial^\circ \mathcal{L}(\pmb{\theta})$  may not be unique. To study a more specific dynamics, we also utilize Definition H.1 to determine GF at some of such points, which overcomes non-uniqueness of GF trajectories to some extent. It is worth noting that Definition H.1 and Definition G.1 are compatible and specifically, the dynamics defined in Definition H.1(Case I, III) lie in the convex hull defined in Definition G.1.
Remark B.1. In (Lyu et al., 2021), the non-branching starting point Assumption is employed to address the technical challenge of non-uniqueness in GF trajectories. By comparison, in this work, we do not need this assumption. We adopt Definition H.1 to uniquely determine the Gradient Flow trajectories theoretically near some discontinuous differential regions, such as "Ridge", "Valley", and "Refraction edge" discussed in Section I.2 in (Lyu et al., 2021).
Additionally, in the following sections, we may rewrite this dynamics accordingly, such as specific forms and dynamics decomposition.
Additional Notations. As a similar description to  $\mathrm{sgn}_k^+(\cdot)$  and  $\mathrm{sgn}_k^-(\cdot)$ , we also employ the following six manifolds to characterize activation patterns (by judging which manifold the neuron  $w_k$  belongs to):
$$
\mathcal {M} _ {+} ^ {+} := \left\{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {+} \rangle > 0 \right\}, \quad \mathcal {M} _ {-} ^ {+} := \left\{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle > 0 \right\},
$$
$$
\mathcal {M} _ {+} ^ {0} := \left\{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {+} \rangle = 0 \right\}, \quad \mathcal {M} _ {-} ^ {0} := \left\{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle = 0 \right\},
$$
$$
\mathcal {M} _ {+} ^ {-} := \{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {+} \rangle \leq 0 \}, \quad \mathcal {M} _ {-} ^ {-} := \{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle \leq 0 \}.
$$
As one of our interested manifolds, the border  $\partial (\mathcal{M}_{+}^{+}\cap \mathcal{M}_{-}^{+})$  can be divided into
$$
\begin{array}{l} \partial \left(\mathcal {M} _ {+} ^ {+} \cap \mathcal {M} _ {-} ^ {+}\right) = \left\{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {+} \rangle = 0 \text {o r} \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle = 0 \right\} \\ = \{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {+} \rangle = 0, \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle > 0 \} \bigcup \{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {+} \rangle > 0, \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle = 0 \} \\ \bigcup \left\{\boldsymbol {w} \in \mathbb {S} ^ {d - 1}: \langle \boldsymbol {w}, \boldsymbol {x} _ {+} \rangle = 0, \langle \boldsymbol {w}, \boldsymbol {x} _ {-} \rangle = 0 \right\} \\ = \left(\mathcal {M} _ {+} ^ {0} \cap \mathcal {M} _ {-} ^ {+}\right) \cup \left(\mathcal {M} _ {-} ^ {0} \cap \mathcal {M} _ {+} ^ {+}\right) \cup \left(\mathcal {M} _ {+} ^ {0} \cap \mathcal {M} _ {-} ^ {0}\right). \\ \end{array}
$$
Furthermore, we also utilize the following notations:
$$
\mathcal {P} _ {+} ^ {+} := \left\{\boldsymbol {b} \in \mathbb {R} ^ {d}: \langle \boldsymbol {b}, \boldsymbol {x} _ {+} \rangle > 0 \right\}, \quad \mathcal {P} _ {-} ^ {+} := \left\{\boldsymbol {b} \in \mathbb {R} ^ {d}: \langle \boldsymbol {b}, \boldsymbol {x} _ {-} \rangle > 0 \right\},
$$
$$
\mathcal {P} _ {+} ^ {0} := \left\{\boldsymbol {b} \in \mathbb {R} ^ {d}: \langle \boldsymbol {b}, \boldsymbol {x} _ {+} \rangle = 0 \right\}, \quad \mathcal {P} _ {-} ^ {0} := \left\{\boldsymbol {b} \in \mathbb {R} ^ {d - 1}: \langle \boldsymbol {b}, \boldsymbol {x} _ {-} \rangle = 0 \right\},
$$
$$
\mathcal {P} _ {+} ^ {-} := \left\{\boldsymbol {b} \in \mathbb {R} ^ {d}: \langle \boldsymbol {b}, \boldsymbol {x} _ {+} \rangle \leq 0 \right\}, \quad \mathcal {P} _ {-} ^ {-} := \left\{\boldsymbol {b} \in \mathbb {R} ^ {d}: \langle \boldsymbol {b}, \boldsymbol {x} _ {-} \rangle \leq 0 \right\}.
$$
Notice that for the direction of a neuron  $\pmb{b}_k \neq \mathbf{0}$ , using  $\mathcal{P}_{+}^{+}, \mathcal{P}_{+}^{0}, \mathcal{P}_{+}^{-}, \mathcal{P}_{-}^{+}, \mathcal{P}_{-}^{0}, \mathcal{P}_{-}^{-}$  to describe  $\pmb{b}_k$  is equivalent to using  $\mathcal{M}_{+}^{+}, \mathcal{M}_{+}^{0}, \mathcal{M}_{+}^{-}, \mathcal{M}_{-}^{+}, \mathcal{M}_{-}^{0}, \mathcal{M}_{-}^{-}$  to describe  $\pmb{w}_k$ .
Lemma B.2 (Dead neurons keep dead). For the  $k$ -th neuron, if there exists a  $t_0 \geq 0$ , s.t.  $\pmb{w}_k(t_0) \in \mathcal{M}_+^-\cap \mathcal{M}_-^-$ , then it dies and remains unchanged during the remaining training:  $\pmb{w}_k(t) \in \mathcal{M}_+^-\cap \mathcal{M}_-^-$  and  $\pmb{b}_k(t) \equiv \pmb{b}_k(t_0)$  for any  $t \geq t_0$ .
Proof of Lemma B.2. A straightforward calculation.
The above fact is a basic fact in our setting, which illustrates that if the neuron  $\pmb{b}_k$  is deactivated for both data  $x_+$  and  $x_-$  at some time, then it remains "dead" forever.
It is worth mentioning that if the neuron  $b_{k}$  is deactivated on  $x_{+}$  but activated on  $x_{-}$  at some time, it can still reactivate on  $x_{+}$  later, which is one of the important reasons why our ReLU optimization dynamics are complicated.
C Proofs of Optimization Dynamics in Phase I
In this section, we conduct a detailed analysis of the training dynamics of each neuron in Phase I. The main proof idea is to decompose neurons' dynamics into tangential and radial dynamics. For small initialization, in Phase I, the radial increasing of neurons is much slower than their tangential velocity, which can result in condensation. However, the main challenges arise from the initial direction's randomness and ReLU's discontinuous derivative, leading to eight categories of neuron dynamics. Moreover, it is also nontrivial and requires meticulous analysis to estimate the number of two classes of living neurons at the end of Phase I.
We define the time
$$
T _ {1} := 1 0 \sqrt {\frac {\kappa_ {1}}{\kappa_ {2}}}, \tag {5}
$$
and call  $t\in [0,T_{\mathrm{I}}]$  "Phase  $\mathbf{I}$
Recall that the selection (3) about initialization parameters can guarantee the whole four-phase optimization dynamics. Nevertheless, when we focus on Phase I,  $\kappa_{2} = \mathcal{O}(1)$  and  $\kappa_{1} / \kappa_{2} = \mathcal{O}(1)$  suffice to ensure the dynamics in Phase I. Specifically, during the proof of Phase I, we can use the following selection (6) on  $\kappa_{1},\kappa_{2}$ , which is much weaker than (3):
$$
\kappa_ {2} = \mathcal {O} (1), \quad \frac {\kappa_ {1}}{\kappa_ {2}} = \mathcal {O} (1). \tag {6}
$$
For simplicity, we assume  $\Delta \leq \frac{1}{5}$ . And for convenience, we assume  $p \geq 5$ . It is worth mentioning that our proof approach also applies for  $p = 2, 3, 4$ , with at most one absolute constant difference.
Prepared for the analysis in Phase I, we decompose the dynamics of  $\pmb{b}_k(t)$  into the radial movement  $\rho_k(t) \in \mathbb{R}$  and the tangential movement  $\pmb{w}_k(t) \in \mathbb{S}^{d-1}$  satisfied to  $\pmb{b}_k(t) = \rho_k(t)\pmb{w}_k(t)$ .
Lemma C.1 (Dynamics decomposition). For any  $k \in [m]$ , the dynamics of  $\pmb{b}_k(t)$  can be decomposed into the radial movement  $\rho_k(t) \in \mathbb{R}$  and the tangential movement  $\pmb{w}_k(t) \in \mathbb{S}^{d-1}$ :
$$
\frac {\mathrm {d} \boldsymbol {w} _ {k} (t)}{\mathrm {d} t} \in \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m} \rho_ {k} (t)} \left(\boldsymbol {F} _ {k} (t) - \langle \boldsymbol {F} _ {k} (t), \boldsymbol {w} _ {k} (t) \rangle \boldsymbol {w} _ {k} (t)\right), \tag {7}
$$
$$
\frac {\mathrm {d} \rho_ {k} (t)}{\mathrm {d} t} \in \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m}} \left\langle \boldsymbol {F} _ {k} (t), \boldsymbol {w} _ {k} (t) \right\rangle ,
$$
where  $\rho_{k}(t) = \| \pmb{b}_{k}(t)\|$ ,  $\pmb{w}_{k}(t) = \pmb{b}_{k}(t) / \| \pmb{b}_{k}(t)\|$  and
$$
\boldsymbol {F} _ {k} (t) = \frac {1}{n} \sum_ {i = 1} ^ {n} e ^ {- y _ {i} f _ {i} (t)} \partial^ {\circ} \sigma \left(\left\langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {i} \right\rangle\right) y _ {i} \boldsymbol {x} _ {i}. \tag {8}
$$
From the dynamics of  $\pmb{b}_k(t)$ :
$$
\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} \in \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m}} \boldsymbol {F} _ {k} (t) = \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m}} \frac {1}{n} \sum_ {i = 1} ^ {n} e ^ {- y _ {i} f _ {i} (t)} \partial^ {\circ} \sigma (\langle \boldsymbol {w} _ {k} (t), \boldsymbol {x} _ {i} \rangle) y _ {i} \boldsymbol {x} _ {i},
$$
$$
\frac {\mathrm {d} \rho_ {k} (t)}{\mathrm {d} t} = \frac {1}{2 \| \boldsymbol {b} _ {k} (t) \|} \frac {\mathrm {d} \| \boldsymbol {b} _ {k} (t) \| ^ {2}}{\mathrm {d} t} = \frac {1}{\| \boldsymbol {b} _ {k} (t) \|} \left\langle \boldsymbol {b} _ {k} (t), \frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} \right\rangle
$$
$$
\in \frac {1}{\| \boldsymbol {b} _ {k} (t) \|} \left\langle \boldsymbol {b} _ {k} (t), \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m}} \boldsymbol {F} _ {k} (t) \right\rangle = \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m}} \left\langle \boldsymbol {F} _ {k} (t), \boldsymbol {w} _ {k} (t) \right\rangle ,
$$
$$
\begin{array}{l} \frac {\mathrm {d} \boldsymbol {w} _ {k} (t)}{\mathrm {d} t} = \frac {\left\| \boldsymbol {b} _ {k} (t) \right\| \frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} - \frac {\mathrm {d} \left\| \boldsymbol {b} _ {k} (t) \right\|}{\mathrm {d} t} \boldsymbol {b} _ {k} (t)}{\left\| \boldsymbol {b} _ {k} (t) \right\| ^ {2}} = \frac {1}{\rho_ {k} (t)} \left(\frac {\mathrm {d} \boldsymbol {b} _ {k} (t)}{\mathrm {d} t} - \frac {\mathrm {d} \rho_ {k} (t)}{\mathrm {d} t} \boldsymbol {w} _ {k} (t)\right) \\ \in \frac {1}{\rho_ {k} (t)} \Big (\frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m}} \pmb {F} _ {k} (t) - \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m}} \left\langle \pmb {F} _ {k} (t), \pmb {w} _ {k} (t) \right\rangle \pmb {w} _ {k} (t) \Big) = \frac {\mathrm {s} _ {k} \kappa_ {2}}{\sqrt {m} \rho_ {k} (t)} \Big (\pmb {F} _ {k} (t) - \left\langle \pmb {F} _ {k} (t), \pmb {w} _ {k} (t) \right\rangle \pmb {w} _ {k} (t) \Big). \\ \end{array}
$$
Prepared for the analysis of the neurons' dynamics, we establish a rough estimate about the norm and prediction growth of each neuron in Phase I, and we will improve it later.
Lemma C.2 (A Rough Estimate of Norm and Prediction in Phase I). For any  $t \leq T_{\mathrm{I}}$ ,  $k \in [m]$ , and  $i \in [n]$ , we have the following estimates.
$$
\rho_ {k} (t) \leq \frac {\kappa_ {1} + 1 . 1 \kappa_ {2} t}{\sqrt {m}},
$$