SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(S1) Let  $\mathcal{K}_{+}$  be the index set of living positive neurons at  $T_{\mathrm{I}}$ , i.e.  $\mathcal{K}_{+} := \{k \in [m/2] : \operatorname{sgn}_{k}^{+}(T_{\mathrm{I}}) = 1$  or  $\operatorname{sgn}_{k}^{-}(T_{\mathrm{I}}) = 1\}$ . Then, (i)  $0.21m \leq |\mathcal{K}_{+}(T_{\mathrm{I}})| \leq 0.29m$ . Moreover, for any  $k \in \mathcal{K}_{+}$ , (ii) its norm is small but significant:  $\rho_{k}(T_{\mathrm{I}}) = \Theta \left( \sqrt{\frac{\kappa_{1}\kappa_{2}}{m}} \right)$ ; (iii) Its direction is strongly aligned with  $\pmb{\mu}$ :  $\langle \pmb{w}_{k}(T_{\mathrm{I}}), \pmb{\mu} \rangle \geq 1 - \mathcal{O}\left( \sqrt{\kappa_{1}\kappa_{2}} \right) - \mathcal{O}\left( \left( \kappa_{1} / \kappa_{2} \right)^{0.55} \right)$ ; (iv)  $\operatorname{sgn}_{k}^{+}(T_{\mathrm{I}}) = \operatorname{sgn}_{k}^{-}(T_{\mathrm{I}}) = 1$ .
(S2) Let  $\mathcal{K}_{-}$  be the index set of living negative neurons at  $T_{\mathrm{I}}$ , i.e.  $\mathcal{K}_{-} := \{k \in [m] - [m/2] : \operatorname{sgn}_k^+(T_{\mathrm{I}}) = 1 \text{ or } \operatorname{sgn}_k^-(T_{\mathrm{I}}) = 1\}$ . Then, (i)  $0.075m \leq |\mathcal{K}_{-}| \leq 0.205m$ . Moreover, for any  $k \in \mathcal{K}_{-}$ , (ii) its norm is tiny:  $\rho_k(T_{\mathrm{I}}) = \mathcal{O}\left(\frac{\sqrt{\kappa_1\kappa_2}}{\sqrt{m}}\left(\sqrt{\frac{\kappa_1}{\kappa_2}} + \frac{\Delta}{p}\right)\right)$ ; (iii) its direction is aligned with  $\boldsymbol{x}_{+}^{\perp} : \left\langle \boldsymbol{w}_k(T_{\mathrm{I}}), \boldsymbol{x}_{+}^{\perp} \right\rangle \geq 1 - \mathcal{O}\left((\sqrt{\frac{\kappa_1}{\kappa_2}}\frac{p}{\Delta})^{1.6}\right)$ ; (iv)  $\operatorname{sgn}_k^-(T_{\mathrm{I}}) = 1$ , but  $\operatorname{sgn}_k^+(T_{\mathrm{I}}) = 0$ .
(S3) For other neuron  $k \notin \mathcal{K}_{+} \cup \mathcal{K}_{-}$ , it dies and remains unchanged during the remaining training process:  $\mathbf{sgn}_k^+(t) \leq 0$ ,  $\mathbf{sgn}_k^-(t) \leq 0$ ,  $\pmb{b}_k(t) \equiv \pmb{b}_k(T_1)$ ,  $\forall t \geq T_1$ .
(S4).  $f_{+}(T_{\mathrm{I}}) = \Theta \left(\kappa_{2}\sqrt{\kappa_{1}\kappa_{2}}\right), f_{-}(T_{\mathrm{I}}) = \Theta \left(\kappa_{2}\sqrt{\kappa_{1}\kappa_{2}}\right), \text{and} \operatorname{Acc}(T_{\mathrm{I}}) = \frac{p}{1 + p}.$
Initial condensation and simplification. Theorem 4.1 (S1)(S2)(S3) show that, after a short time  $T_{\mathrm{I}} = \Theta (\sqrt{\kappa_1 / \kappa_2})$ , all neurons are implicitly simplified to three categories:  $\mathcal{K}_{+}$ ,  $\mathcal{K}_{-}$  and others. The living positive neurons  $k \in \mathcal{K}_{+}$  align strongly with  $\mu$ , and the living negative neurons  $k \in \mathcal{K}_{-}$  align with  $x_{+}^{\perp}$  and lie on the manifold orthogonal to  $x_{+}$ . Other neurons die and remain unchanged during the remaining training process. Moreover, we also estimate tight bounds for  $|\mathcal{K}_{+}|$  and  $|\mathcal{K}_{+}|$ . Actually,  $\kappa_{1} / \kappa_{2} = \mathcal{O}(1)$  can ensure Theorem 4.1 and initial condensation hold (please refer to Appendix C), and we write  $\kappa_{1} / \kappa_{2} = \mathcal{O}(\Delta^{8})$  here to ensure that the dynamics of later phases hold. In Phase I, the dynamics exhibit a fast condensation phenomenon, i.e., in addition to dead neurons, living positive and negative neurons condense in one direction each.
Saddle-to-Plateau. The network is initially close to the saddle point at 0 (where the loss gradient vanishes). However, Theorem 4.1 (S1) reveals that despite being small, there is a significant growth in the norm of living positive neuron  $k \in \mathcal{K}_{+}$  from  $\Theta(\kappa_{1}/\sqrt{m})$  to  $\Theta(\sqrt{\kappa_{1}\kappa_{2}}/\sqrt{m})$  and the predictions also experience substantial growth (S4). This means that GF escapes from this saddle rapidly. Furthermore, it is worth noting that initial training accuracy can randomly be 0,  $\frac{1}{1+p}$ ,  $\frac{p}{1+p}$ , or 1. However, after Phase I, the training accuracy reaches  $\mathrm{Acc}(T_{\mathrm{I}}) = \frac{p}{1+p}$  which we will prove as a plateau in the next subsection. Therefore, Phase I exhibits saddle-to-plateau dynamics.
Remark 4.2. Throughout the following subsections, we call the neuron  $k \in \mathcal{K}_{+}$  the "living positive neuron", the neuron  $k \in \mathcal{K}_{-}$  the "living negative neuron", and the neuron  $k \notin \mathcal{K}_{+} \cup \mathcal{K}_{-}$  the
"dead neuron". Moreover, we denote  $m_{+} \coloneqq |\mathcal{K}_{+}|$ ,  $m_{-} \coloneqq |\mathcal{K}_{-}|$ , and  $\alpha \coloneqq \frac{m_{-}}{m_{+}}$ . Notice that Theorem 4.1(S1)(S2) guarantee that  $0 < \frac{0.075}{0.29} \leq \alpha \leq \frac{0.205}{0.21} < 1$ .
Remark 4.3. The results in the following subsections are all based on the occurrence of the events in Theorem 4.1 and with the same settings as Theorem 4.1. So they all hold with probability at least  $1 - \delta$ .
Please refer to Appendix C for the proof of Phase I.
4.3 Phase II. Getting Stuck in and Escaping from Plateau
In this phase, we study the dynamics before the patterns of living neurons change again after Phase I. Specifically, we define
$$
T _ {\mathrm {I I}} := \inf  \left\{t > T _ {\mathrm {I}}: \exists k \in \mathcal {K} _ {+} \cup \mathcal {K} _ {-}, \operatorname {s g n} _ {k} ^ {+} (t) \neq \operatorname {s g n} _ {k} ^ {+} \left(T _ {\mathrm {I}}\right) \text {o r} \operatorname {s g n} _ {k} ^ {-} (t) \neq \operatorname {s g n} _ {k} ^ {-} \left(T _ {\mathrm {I}}\right) \right\},
$$
and call  $t\in (T_{\mathrm{I}},T_{\mathrm{II}}]$  Phase II.
Theorem 4.4 (End of Phase II). (S1)  $T_{\mathrm{II}} = \Theta \left(\frac{\frac{1}{p^{1 - \alpha}\cos\Delta}}{\kappa_2^2\Delta^2}\right)$ . (S2)  $\mathcal{L}(\theta(T_{\mathrm{II}})) = \Theta \left(p^{-\frac{1}{1 - \alpha}\cos\Delta}\right)$ . (S3) One of living positive neuron  $k_0 \in \mathcal{K}_+$  precisely changes its pattern on  $x_-$  at  $T_{\mathrm{II}}$ :  $\lim_{t \to T_{\mathrm{II}}^-} \operatorname{sgn}_{k_0}^- (t) = 1$  and  $\lim_{t \to T_{\mathrm{II}}^+} \operatorname{sgn}_{k_0}^- (t) = 0$ , while all other activation patterns remain unchanged.
Recalling the results in Theorem 4.1, during Phase II, the activation patterns do not change with  $\mathrm{sgn}_k^+(t) = \mathrm{sgn}_k^- (t) = 1$  for  $k\in \mathcal{K}_{+}$  and  $\mathrm{sgn}_k^+(t) = 0$ ,  $\mathrm{sgn}_k^- (t) = 1$  for  $k\in \mathcal{K}_{-}$ . Theorem 4.4 demonstrates that at the end of Phase II, except for one of living positive neuron  $k_{0}\in \mathcal{K}_{+}$  precisely changes its pattern on  $\pmb{x}_{-}$ , all other activation patterns remain unchanged.
Theorem 4.5 (Plateau). We define the hitting time  $T_{\mathrm{plat}} \coloneqq \inf \{t \in [T_{\mathrm{I}}, T_{\mathrm{II}}] : \mathrm{Acc}(t) = 1\}$ . Then, (S1)  $T_{\mathrm{plat}} = \Theta \left( \frac{p}{\kappa_2^2 \Delta^2} \right)$ ; (S2)  $\forall t \in [T_{\mathrm{I}}, T_{\mathrm{plat}}]$ ,  $\mathrm{Acc}(t) \equiv \frac{p}{1 + p}$ ; (S3)  $\forall t \in (T_{\mathrm{plat}}, T_{\mathrm{II}}]$ ,  $\mathrm{Acc}(t) \equiv 1$ .
Plateau of training accuracy. According to Theorem 4.5, during Phase II, the training accuracy gets stuck in a long plateau  $\frac{p}{1 + p}$ , which lasts for  $\Theta\left(\frac{p}{\kappa_2^2\Delta^2}\right)$  time. However, once escaping from this plateau, the training accuracy rises to  $100\%$ . It is worth noting that this plateau is essentially induced by the dataset. All that's required is only mild imbalance ( $p$  is slightly greater than 1 such that  $p\cos \Delta > 1$ ) and a small margin  $\sin (\Delta / 2)$  of two data classes. Notably, if the dataset has an extremely tiny margin ( $\Delta \to 0$ ), then the length of this plateau will be significantly prolonged ( $T_{\mathrm{plat}} \to +\infty$ ), which implies how the data separation can affect the training dynamics. Additionally, using a smaller initialization scale  $\kappa_1$  of the input layers cannot avoid this plateau.
Please refer to Appendix D for the proof of Phase II.
4.4 Phase III. Simplifying by Neuron Deactivation, and Trying to Learn by Simplest Network
Building upon Phase II, we demonstrate that within a short time, all the living positive neurons  $\mathcal{K}_{+}$  change their activation patterns, corresponding to a "phase transition". Specifically, we define
$$
T _ {\mathrm {I I}} ^ {\mathrm {P T}} := \inf  \left\{t > T _ {\mathrm {I I}}: \forall k \in \mathcal {K} _ {+}, \operatorname {s g n} _ {k} ^ {-} (t) = 0 \right\},
$$
and we call  $t\in (T_{\mathrm{II}},T_{\mathrm{II}}^{\mathrm{PT}}]$  the phase transition from Phase II to Phase III.
Theorem 4.6 (Phase Transition). (S1)  $T_{\mathrm{II}}^{\mathrm{PT}} = \left(1 + \mathcal{O}\left(\sqrt{\kappa_1\kappa_2^3}\right)\right)T_{\mathrm{II}}$ ; (S2)  $\mathbf{sgn}_k^+ (T_{\mathrm{II}}^{\mathrm{PT}}) = 1$  and  $\mathbf{sgn}_k^- (T_{\mathrm{II}}^{\mathrm{PT}}) = 0$  for any  $k\in \mathcal{K}_{+}$ ;  $\mathbf{sgn}_k^+ (T_{\mathrm{II}}^{\mathrm{PT}}) = 0$  and  $\mathbf{sgn}_k^- (T_{\mathrm{II}}^{\mathrm{PT}}) = 1$  for any  $k\in \mathcal{K}_{-}$ .
Neuron deactivation. As shown in Theorem 4.6 (S2), after the phase transition, all the living positive neurons  $k \in \mathcal{K}_{+}$  undergo deactivation for  $x_{-}$ , i.e.,  $\mathrm{sgn}_k^- (t)$  changes from 1 to 0, while other activation patterns remain unchanged. Furthermore, Theorem 4.6 (S1) reveals that the phase transition is completed quite quickly by using sufficiently small initialization value  $\kappa_{1}, \kappa_{2}$ . A smaller initialization value leads to a more precise initial condensation  $w_{k}(T_{\mathrm{I}}) \approx \mu$ , causing all living positive neurons to remain closer together before  $T_{\mathrm{II}}$  and thus changing their patterns nearly simultaneously.
Self-simplifying. As a result, the network implicitly simplifies itself through the deactivation behavior. At  $T_{\mathrm{III}}^{\mathrm{PT}}$ , only living negative neurons  $k \in \mathcal{K}_{+}$  are used for predicting on  $x_{-}$ , i.e.,  $f_{-}(T_{\mathrm{II}}^{\mathrm{PT}}) = \frac{\kappa_2}{\sqrt{m}} \sum_{k \in \mathcal{K}_{-}} \sigma(\langle b_k(T_{\mathrm{II}}^{\mathrm{PT}}, x_{-} \rangle)$ . In contrast, during Phase II, both living positive and living negative neurons jointly predict on  $x_{-}$ , i.e.,  $f_{-}(t) = \frac{\kappa_2}{\sqrt{m}} \sum_{k \in \mathcal{K}_{+}} \sigma(\langle b_k(t), x_{-} \rangle) - \frac{\kappa_2}{\sqrt{m}} \sum_{k \in \mathcal{K}_{-}} \sigma(\langle b_k(t), x_{-} \rangle)$ . As indicated in Table 1, two classes of activation patterns are simplified from (1, 0) to (0, 0), while others do not change.
After this phase transition, we study the dynamics before the patterns of living neurons change again. Specifically, we define
$$
T _ {\mathrm {I I I}} := \inf  \{t > T _ {\mathrm {I I}} ^ {\mathrm {P T}}: \exists k \in \mathcal {K} _ {+} \cup \mathcal {K} _ {-}, \operatorname {s g n} _ {k} ^ {+} (t) \neq \operatorname {s g n} _ {k} ^ {+} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right) \text {o r} \operatorname {s g n} _ {k} ^ {-} (t) \neq \operatorname {s g n} _ {k} ^ {-} \left(T _ {\mathrm {I I}} ^ {\mathrm {P T}}\right) \},
$$
and call  $t\in (T_{\mathrm{II}},T_{\mathrm{III}}]$  Phase III.
Theorem 4.7 (End of Phase III).  $T_{\mathrm{III}} = \left(1 + \Theta (\Delta^2)\right)T_{\mathrm{II}}$
Learning by simplest network. During  $t \in (T_{\mathrm{II}}^{\mathrm{PT}}, T_{\mathrm{III}})$ , all activation patterns do not change. This ensures that  $f_{+}(t) = \frac{\kappa_{2}}{\sqrt{m}} \sum_{k \in \mathcal{K}_{+}} \sigma(\langle \pmb{b}_{k}(t), \pmb{x}_{+} \rangle)$ , while  $f_{-}(t) = -\frac{\kappa_{2}}{\sqrt{m}} \sum_{k \in \mathcal{K}_{-}} \sigma(\langle \pmb{b}_{k}(t), \pmb{x}_{-} \rangle)$ . Additionally, by using sufficiently small  $\kappa_{1}$ , the neurons in  $\mathcal{K}_{+}$  and  $\mathcal{K}_{-}$  keep close together respectively before  $T_{\mathrm{III}}$ , making the network close to a simple two-neuron network consisting of one positive neuron and one negative neuron. Please refer to Appendix E for more details. Furthermore, this pattern scheme is almost the "simplest" way to ensure binary classification: the living positive neurons only predict positive data  $x_{+}$  while the living negative neurons only predict negative data  $x_{-}$ . Therefore, GF tries to learn by this almost simplest network in this phase.
Please refer to Appendix E for the proof of Phase III.
4.5 Phase IV. Complicating by Neuron Reactivation, and Directional Convergence
Phase IV begins with an instantaneous phase transition at time  $T_{\mathrm{III}}$ .
Theorem 4.8 (Phase Transition). All living negative neuron  $k \in \mathcal{K}_{-}$ simultaneously change their patterns on  $x_{+}$ at  $T_{\mathrm{III}}$ :  $\lim_{t \to T_{\mathrm{III}}^{-}} \operatorname{sgn}_k^+(t) = 0$ ,  $\lim_{t \to T_{\mathrm{III}}^{+}} \operatorname{sgn}_k^+(t) = 1$ , while others remain unchanged.
Neuron reactivation. According to Theorem 4.8, all of living negative neurons  $k \in \mathcal{K}_{-}$  reactivate simultaneously on  $x_{+}$  at  $T_{\mathrm{III}}$ :  $\operatorname{sgn}_k^+(t)$  changes from 0 to 1, while other activation patterns remain unchanged.
Self-Complicating. Along with the reactivation behavior, GF implicitly complicates itself. In Phase III, only living negative neurons  $k \in \mathcal{K}_+$  are used to predict on  $x_+$ , i.e.,  $f_+(t) = \frac{\kappa_2}{\sqrt{m}} \sum_{k \in \mathcal{K}_+} \sigma(\langle b_k(t), x_+ \rangle)$ . In contrast, after the phase transition at  $T_{\mathrm{III}}$ , both living positive and living negative neurons jointly predict on  $x_+$ , i.e.,  $f_+(t) = \frac{\kappa_2}{\sqrt{m}} \sum_{k \in \mathcal{K}_+} \sigma(\langle b_k(t), x_+ \rangle) - \frac{\kappa_2}{\sqrt{m}} \sum_{k \in \mathcal{K}_-} \sigma(\langle b_k(t), x_+ \rangle)$ . As indicated in Table 1, two classes of activation patterns are complicated from  $(0,0)$  to  $(0,1)$ , while others do not change.
In this phase, we study the dynamics before activation patterns change again after the phase transition in Theorem 4.8. We define the hitting time:
$$
T _ {\mathrm {I V}} := \inf  \{t > T _ {\mathrm {I I I}}: \exists k \in \mathcal {K} _ {+} \cup \mathcal {K} _ {-}, \operatorname {s g n} _ {k} ^ {+} (t) \neq \lim  _ {s \to T _ {\mathrm {I I I}} ^ {+}} \operatorname {s g n} _ {k} ^ {+} (s) \text {o r} \operatorname {s g n} _ {k} ^ {-} (t) \neq \lim  _ {s \to T _ {\mathrm {I I I}} ^ {+}} \operatorname {s g n} _ {k} ^ {-} (s) \},
$$
and we call  $t\in (T_{\mathrm{III}},T_{\mathrm{IV}}]$  Phase IV.
Theorem 4.9 (Phase IV). (S1)  $T_{\mathrm{IV}} = +\infty$ . Moreover, for any  $t > T_{\mathrm{III}}$ , (S2) all activation patterns do not change; (S3) the loss converges with  $\mathcal{L}(\pmb{\theta}(t)) = \Theta \left( \frac{1}{p^{\frac{1}{1 - \alpha}\cos\Delta} + \kappa_2^2\Delta^2(t - T_{\mathrm{III}})} \right)$ .
Theorem 4.9 illustrates that all activation patterns never change again after the phase transition at  $T_{\mathrm{III}}$  with  $\operatorname{sgn}_k^+(t) = 1$ ,  $\operatorname{sgn}_k^- (t) = 0$  for any  $k \in \mathcal{K}_{+}$  and  $\operatorname{sgn}_k^+(t) = \operatorname{sgn}_k^- (t) = 1$  for any  $k \in \mathcal{K}_{-}$ . Additionally, the loss converges with the polynomial rate  $\Theta(1 / \kappa_2^2 \Delta^2 t)$ . Furthermore, we present the following theorem about the convergent direction of each neuron.
Theorem 4.10 (Directional Convergence). The limit  $\lim_{t\to +\infty}\frac{\pmb{\theta}(t)}{\|\pmb{\theta}(t)\|_2}$  exists and denoted by  $\overline{\pmb{\theta}} = (\overline{\pmb{b}}_1^\top, \dots, \overline{\pmb{b}}_m^\top)^\top \in \mathbb{S}^{md-1}$ . Moreover, (i) for any  $k \notin \mathcal{K}_+ \cup \mathcal{K}_-$ ,  $\overline{\pmb{b}}_k = \mathbf{0}$ ; (ii) for any  $k \in \mathcal{K}_+$ ,  $\overline{\pmb{b}}_k \equiv \pmb{v}_+ = C(\pmb{x}_+ - \pmb{x}_- \cos \Delta)$ ; (iii) for any  $k \in \mathcal{K}_-$ ,  $\overline{\pmb{b}}_k \equiv \pmb{v}_- = C((1 + \frac{\sin^2\Delta}{\alpha(1 + \cos\Delta)})\pmb{x}_- - \pmb{x}_+)$ , where  $C > 0$  is a scaling constant such that  $\left\| \overline{\pmb{\theta}} \right\|_2 = 1$ . (iv) Additionally,  $f_-(\overline{\pmb{\theta}}) = -f_+(\overline{\pmb{\theta}}) > 0$ .
Initialization-dependent Directional Convergence. As an asymptotic result, Theorem 4.10 provides the final convergent direction of GF. All living positive neurons  $(k\in \mathcal{K}_{+})$  directionally converge to  $\pmb{v}_{+}\parallel \pmb{x}_{-}^{\perp}$  with  $\langle \pmb{v}_{+},\pmb{x}_{+}\rangle >0$  and  $\langle \pmb{v}_{+},\pmb{x}_{-}\rangle = 0$  , while all living negative neurons  $(k\in \mathcal{K}_{-})$  directionally converge to  $\pmb{v}_{-}\in \operatorname {span}\{\pmb{x}_{+},\pmb{x}_{-}\}$  with  $\langle \pmb{v}_{-},\pmb{x}_{+}\rangle >0$  and  $\langle v_{-},x_{-}\rangle >0$  . It is worth noting that  $\pmb{v}_{-}$  directly depends not only on the data but also on the ratio  $\alpha = |\mathcal{K}_-\vert /|\mathcal{K}_+\vert$  (defined in Remark 4.2). Recalling the results in Phase I,  $\alpha$  lies in a certain range with high probability; but it is still a random variable due to its dependence on random initialization. Different initializations may lead to different values  $|\mathcal{K}_-| / |\mathcal{K}_+$  at the end of Phase I, eventually causing different convergent directions in Phase IV.
Non-(Local)-Max-Margin Direction. Lastly, we study the implicit bias of the final convergence rate. According to Lyu and Li (2019); Ji and Telgarsky (2020) and our results above,  $\overline{\theta}$  in Theorem 4.10 must be a KKT direction of some max-margin optimization problem. However, it is not clear whether the direction  $\overline{\theta}$  is actually an actual optimum of this problem. Surprisingly, in next Theorem, we demonstrate that the final convergent direction is not even a local optimal direction of this problem, which enlightens us to rethink the max margin bias of ReLU neural networks.
Theorem 4.11 (Implicit Bias). The final convergent direction  $\overline{\theta}$  (in Theorem 4.10) is a KKT direction of the max-margin problem  $\min : \frac{1}{2} \| \pmb{\theta} \|^2$  s.t.  $y_i f(\pmb{x}_i; \pmb{\theta}) \geq 1, i \in [n]$ . However,  $\overline{\theta}$  is not even a local optimal direction of this problem.
Please refer to Appendix F for the proof of Phase IV.
5 Discussion and Comparison on Nonlinear Behaviors
Throughout the whole training process in Section 4, we divide the phases based on the evolution of ReLU activation patterns. During Phase I, as well as the beginning of Phase II and III, numerous activation patterns undergo rapid changes. Table 1 summarizes the evolution of activation patterns for all living neurons after Phase I. These results are also numerically validated in Figure 1.
Table 1: The evolution of two classes of activation patterns of living neurons after Phase I. As for other two classes,  $\mathbf{sgn}_k^+(t) (k \in \mathcal{K}_+)$  and  $\mathbf{sgn}_k^-(t) (k \in \mathcal{K}_-)$ , they remain equal to 1 after Phase I.
t ∈ (T1, TII)	t ∈ (TII, TPT)	t ∈ (PTT, TIII)	t ∈ (TIII, +∞)
sgn-k(t) (k ∈ K+)	1	1 or 0	0	0
sgn+k(t) (k ∈ K-)	0	0	0	1
Simplifying-to-Complicating. In phase I, GF simplifies all the neurons from random directions into three categories: living positive neurons  $\kappa_{+}$  and living negative neurons  $\kappa_{-}$  condense in one direction each, which other neurons are deactivated forever. After Phase I, as shown in Table 1, the two classes of activation patterns change from  $(1,0)\stackrel{\text{simplify}}{\rightarrow}(0,0)\stackrel{\text{complicate}}{\rightarrow}(0,1)$ , while other patterns remain unchanged. Therefore, the evolution of activation patterns exhibits a simplifying-to-complicating learning trend, which also implies that the network trained by GF learn features in increasing complexity.
Comparison with NTK. In the lazy regime such as NTK, most neurons keep close to the initialization and most activation patterns do not change during training. Specifically, for any training data  $\boldsymbol{x}_i$ ,  $\frac{1}{m} \sum_{k \in [m]} \mathbb{I}\{\mathrm{sgn}(\langle \boldsymbol{b}_k(t), \boldsymbol{x}_i \rangle) \neq \mathrm{sgn}(\langle \boldsymbol{b}_k(0), \boldsymbol{x}_i \rangle)\} = o(1), \forall t > 0$  (Du et al., 2018). However, our work stands out from lazy regime as activation patterns undergo numerous changes during training. In Phase I, initial condensation causes substantial changes in activation patterns, which is similarly observed in (Phuong and Lampert, 2021). Furthermore, even after Phase I, there are notable modifications in activation patterns. As shown in Table 1, the proportion of changes in
activation patterns for any given training data is the  $\Theta(1)$ , as compared with the  $o(1)$  in NTK regime. Specifically, at any  $t > T_{\mathrm{III}}$ ,  $\frac{1}{m}\sum_{k\in [m]}\mathbb{I}\{\mathsf{sgn}_k^+(t)\neq \mathsf{sgn}_k^+ (T_\mathrm{I})\} = \frac{1}{m} |\mathcal{K}_-| = \Theta (1)$  and  $\frac{1}{m}\sum_{k\in [m]}\mathbb{I}\{\mathsf{sgn}_k^- (t)\neq \mathsf{sgn}_k^- (T_\mathrm{I})\} = \frac{1}{m} |\mathcal{K}_+| = \Theta (1)$ . On the other hand, in our analysis, the requirement on the network's width  $m$  is only  $m = \Omega (\log (1 / \delta))$  (Theorem 4.1), regardless of data parameters  $p,\Delta$ , while NTK regime requires a much larger width  $m = \Omega (\log (p / \delta) / \Delta^6)$  (Ji and Telgarsky, 2019).
Comparison with Phuong and Lampert (2021); Lyu et al. (2021); Wang and Ma (2022); Boursier et al. (2022). Beyond lazy regime and local analysis, these works also characterize the entire training dynamics and analyze a few nonlinear behaviors. Now we compare our results with these works in detail. (i) While Lyu et al. (2021) focuses on training Leaky ReLU NNs, our work and the other three papers study ReLU NNs. It is worth noting that the dynamics of Leaky ReLU NNs differ from ReLU due to the permanent activation of Leaky ReLU  $(\sigma'(\cdot) \geq \alpha > 0)$ . (ii) Initial condensation is also proven in (Lyu et al., 2021; Boursier et al., 2022), and the condensation directions are some types of data averages. In our work, neurons can aggregate towards not only the average direction  $\mu$ , but also another direction  $x_{+}^{\perp}$ . Moreover, we also estimate the number of neurons that condense into two directions. (iii) Saddle-to-saddle dynamics are proven in (Phuong and Lampert, 2021) for square loss, where the second saddle is about training loss and caused by incomplete fitting. However, our work focuses on classification with exponential loss and exhibit a similar saddle-to-plateau dynamics, where the plateau is about training accuracy, caused by incomplete feature learning. (iv) Phased feature learning. In (Phuong and Lampert, 2021; Wang and Ma, 2022), all features can be rapidly learned in Phase I (accuracy = 100%), followed by lazy training. However, for practical tasks on more complex data, NNs can hardly learn all features in such short time. In our work, the data is more difficult to learn, resulting in incomplete feature learning in Phase I (accuracy < 100%). Subsequently, NNs experience a long time to learn other features completely. Such multi-phase feature leaning dynamics are closer to practical training process. (v) Neuron reactivation and deactivation. For ReLU NNs, The evolution of activation patterns is one of the essential causes of nonlinear dynamics. In (Phuong and Lampert, 2021; Wang and Ma, 2022), activation patterns only change rapidly in Phase I, after which they remain unchanged. In (Boursier et al., 2022), their lemma 6 shows that their dynamics lack neuron reactivation. However, in our dynamics, even after Phase I, our dynamics exhibit significant neuron deactivation and reactivation as discussed in Table 1. (vi) The final convergent directions are also derived in (Phuong and Lampert, 2021; Lyu et al., 2021; Boursier et al., 2022), which only depend on the data. However, in our setting, the convergent direction is more complicated, determined by both data and random initialization. (vii) Furthermore, our four-phase dynamics demonstrate the whole evolution of activation patterns during training and reveal a general simplifying-to-complicating learning trend.