Title: Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: non-convex optimization

training dynamics

neural network
Abstract: The training process of ReLU neural networks often exhibits complicated nonlinear phenomena. 
The nonlinearity of models and non-convexity of loss pose significant challenges for theoretical analysis. Therefore, most previous theoretical works on the optimization dynamics of neural networks focus either on local analysis (like the end of training) or approximate linear models (like Neural Tangent Kernel). 
In this work, we conduct a complete theoretical characterization of the training process of a two-layer ReLU network trained by Gradient Flow on a linearly separable data. In this specific setting, our analysis captures the whole optimization process starting from random initialization to final convergence. 
Despite the relatively simple model and data that we studied, we reveal four different phases from the whole training process showing a general simplifying-to-complicating learning trend.
Specific nonlinear behaviors can also be precisely identified and captured theoretically, such as
initial condensation, saddle-to-plateau dynamics, plateau escape, changes of activation patterns, 
learning with increasing complexity, etc.

Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks

Abstract

The training process of ReLU neural networks often exhibits complicated nonlinear phenomena. The nonlinearity of models and non-convexity of loss pose significant challenges for theoretical analysis. Therefore, most previous theoretical works on the optimization dynamics of neural networks focus either on local analysis (like the end of training) or approximate linear models (like Neural Tangent Kernel). In this work, we conduct a complete theoretical characterization of the training process of a two-layer ReLU network trained by Gradient Flow on a linearly separable data. In this specific setting, our analysis captures the whole optimization process starting from random initialization to final convergence. Despite the relatively simple model and data that we studied, we reveal four different phases from the whole training process showing a general simplifying-to-complicating learning trend. Specific nonlinear behaviors can also be precisely identified and captured theoretically, such as initial condensation, saddle-to-plateau dynamics, plateau escape, changes of activation patterns, learning with increasing complexity, etc.

1 Introduction

Deep learning shows its remarkable capabilities across various fields of applications. However, the theoretical understanding of its great success still has a long way to go. Among all theoretical topics, one of the most crucial aspect is the understanding of the optimization dynamics of deep neural network (NN), particularly the dynamics produced by Gradient Descent (GD) and its variants. This topic is highly challenging due to the highly non-convex loss landscape and existing works usually work with settings that do not align well with realistic practices. For instance, the extensived studied Neural Tangent Kernel (NTK) theory (Jacot et al., 2018; Du et al., 2018; 2019; Zou et al., 2018; Allen-Zhu et al., 2019) proves the global convergence of Stochastic gradient descent (SGD) to zero training error for highly over-parameterized neural networks; however, the optimization behaviors are similar to kernel methods and do not exhibit nonlinear behaviors, because neurons remain close to their initialization throughout training.

In reality, however, the training of practical networks can exhibit plenty of nonlinear behaviors (Chizat and Bach, 2018; Mei et al., 2019; Woodworth et al., 2020). In the initial stage of the training, a prevalent nonlinear phenomenon induced by small initialization is initial condensation (Maennel et al., 2018; Luo et al., 2021), where neurons condense onto a few isolated orientations. At the end of training, NNs trained by GD can directionally converge to the KKT points of some constrained max-margin problem (Nacson et al., 2019; Lyu and Li, 2019; Ji and Telgarsky, 2020). However, KKT points are not generally unique, and determining which direction GD converges to can be challenging. Nonlinear training behaviors besides initial and terminating stages of optimization are also numerous. For example, for square loss, Jacot et al. (2021) investigates the saddle-to-saddle dynamics where GD traverses a sequence of saddles during training, but it is unclear whether similar behavior can

Chao Ma

occur for classification tasks using exp-tailed loss. Moreover, while in lazy regime, most activation patterns do not change during training ReLU networks, it remains uncertain when and how activation patterns evolve beyond lazy regime. Additionally, while it is generally conjectured that GD learns functions of increasing complexity (Nakkiran et al., 2019), this perspective has yet to be proven.

As reviewed in Section 2, works have been done to analyze and explain the nonlinear training behaviors listed above. However, due to the complexity of the training dynamics, most existing works only focus on one phenomenon and conduct analysis on a certain stage of the training process. Few attempts have been done to derive a full characterization of the whole training dynamics from the initialization to convergence, and the settings adopted by these works are usually too simple to capture many nonlinear behaviors (Phuong and Lampert, 2021; Lyu et al., 2021; Wang and Ma, 2022; Boursier et al., 2022).

In this work , we make an attempt to theoretically describe the whole neural network training dynamics beyond the linear regime, in a setting that many nonlinear behaviors manifest. Specifically, We analyze the training process of a two-layer ReLU network trained by Gradient Flow (GF) on a linearly separable data. In this setting, our analysis captures the whole optimization process starting from random initialization to final convergence. Despite the relatively simple model and data that we studied, we reveal multiple phases in training process, and show a general simplifying-to-complicating learning trend by detailed analysis of each phase. Specifically, by our meticulous theoretical analysis of the whole training process, we precisely identify four different phases that exhibit numerous nonlinear behaviors . In Phase I, initial condensation and simplification occur as living neurons rapidly condense in two different directions. Meanwhile, GF escapes from the saddle around initialization. In Phase II, GF gets stuck into the plateau of training accuracy for a long time, then escapes . In Phase III, a significant number of neurons are deactivated , leading to self-simplification of the network, then GF tries to learn using the almost simplest network. In Phase IV, a considerable number of neurons are reactivated , causing self-complication of the network. Finally, GF converges towards an initialization-dependent direction , and this direction is not even a local max margin direction . Overall, the whole training process exhibits a remarkable simplifying-to-complicating behavior.

2 Other Related Works

Initial condensation phenomenon are studied in (Maennel et al., 2018; Luo et al., 2021; Zhou et al., 2022a;b; Abbe et al., 2022a;b; Chen et al., 2023). Theoretically, Lyu et al. (2021); Boursier et al. (2022) analyze the condensation directions under their settings, which are some types of data average. Additionally, Atanasov et al. (2022) demonstrates that NNs in the rich feature learning regime learn a kernel machine due to the silent alignment phenomenon, similar to the initial condensation.

The end of training is extensively studied for classification tasks. Specifically, for classification with exponentially-tailed loss functions, if all the training data can be classified correctly, NNs trained by GD converge to the KKT directions of some constrained max-margin problem (Nacson et al., 2019; Lyu and Li, 2019; Chizat and Bach, 2020; Ji and Telgarsky, 2020; Kunin et al., 2023). In (Phuong and Lampert, 2021; Lyu et al., 2021), they analyze entire training dynamics and derive specific convergent directions that only depend on the data. Furthermore, another famous phenomenon in the end of training is the neural collapse (Papyan et al., 2020; Fang et al., 2021; Zhu et al., 2021; Han et al., 2021), which says the features represented by over-parameterized neural networks for data in a same class will collapse to one point, and such points for all classes converge to a simplex equiangular tight frame.

Saddle-to-saddle dynamics are explored for square loss in (Jacot et al., 2021; Zhang et al., 2022; Boursier et al., 2022; Pesme and Flammarion, 2023; Abbe et al., 2023). Furthermore, learning of increasing complexity, also called simplifying-to-complicating or frequency-principle, is investigated in (Arpit et al., 2017; Nakkiran et al., 2019; Xu et al., 2019; Rahaman et al., 2019).

Beyond lazy regime and local analysis, Phuong and Lampert (2021); Lyu et al. (2021); Wang and Ma (2022); Boursier et al. (2022) also characterize the whole training dynamics and exhibit a few of nonlinear behaviors. Specifically, Lyu et al. (2021) studies the training dynamics of GF on Leaky ReLU networks, which differ from ReLU networks because Leaky ReLU is always activated on any data. In (Safran et al., 2022), they studies the dynamics of GF on one dimensional dataset, and characterizes the effective number of linear regions. In (Brutzkus et al., 2017), they studies the

dynamics of SGD on Leaky ReLU networks and linearly separable dataset. Moreover, Boursier et al. (2022) characterizes the dynamics on orthogonally data for square loss. The studies closest to our work are Phuong and Lampert (2021); Wang and Ma (2022), exploring the complete dynamics on classifying orthogonally separable data. However, this data is easy to learn, and all the features can be learned rapidly (accuracy= 100% ) in initial training, followed by lazy training (activation patterns do not change). Unfortunately, this simplicity does not hold true for actual tasks on much more complex data, and NNs can only learn some features in initial training, which complicates the overall learning process. Furthermore, we provide a detailed comparison between our results and these works in Section 5. Another related work (Saxe et al., 2022) introduces a novel bias of learning dynamics: toward shared representations. This idea and the view of gating networks are enlightening for extending our two-layer theory to deep ReLU neural networks.

Our work also investigates the max-margin implicit bias of ReLU neural networks, and related works have been listed above. Although in homogenized neural networks such as ReLU, GD implicitly converges to a KKT point of the max-margin problem, it is still unclear where it is an actual optimum. A recent work (Vardi et al., 2022) showed that in many cases, the converged KKT point is not even a local optimum of the max margin problem. Besides, there are many other attempts to explain the implicit bias of deep learning (Vardi, 2023). Another popular implicit bias is the flat minima bias (Hochreiter and Schmidhuber, 1997; Keskar et al., 2016). Recent studies (Wu et al., 2018; Blanc et al., 2020; Ma and Ying, 2021; Li et al., 2021; Mulayoff et al., 2021; Wu et al., 2022; Wu and Su, 2023) provided explanations for why SGD favors flat minima and flat minima generalize well.

3 Preliminaries

Basic Notations. We use bold letters for vectors or matrices and lowercase letters for scalars, e.g. x = ( x 1 , · · · , x d ) ⊤ ∈ R d and P = ( P ij ) m 1 × m 2 ∈ R m 1 × m 2 . We use ⟨· , ·⟩ for the standard Euclidean inner product between two vectors, and ∥·∥ for the l 2 norm of a vector or the spectral norm of a matrix. We use progressive representation O , Ω , Θ to hide absolute positive constants. For any positive integer n , let [ n ] = { 1 , · · · , n } . Denote by N ( µ , Σ ) the Gaussian distribution with mean µ and covariance matrix Σ , U ( S ) the uniform distribution on a set S . Denote by I { E } the indicator function for an event E .

3.1 Binary Classification with Two-layer ReLU Networks

Binary classification. In this paper, we consider the binary classification problem. We are given n training data S = { ( x i , y i ) } n i =1 ⊂ R d ×{± 1 } . Let f ( · ; θ ) be a neural network model parameterized by θ , and aim to minimize the empirical risk given by:

where ℓ ( · ) : R → R is the exponential-type loss function (Soudry et al., 2018; Lyu and Li, 2019) for classification tasks, including the most popular classification losses: exponential loss, logistic loss, and cross-entropy loss. Our analysis focuses on the exponential loss ℓ ( z ) = e -z , while our method can be extended to logistic loss and cross-entropy loss.

Two-layer ReLU Network. Throughout the following sections, we consider two-layer ReLU neural networks comprising m neurons defined as

where σ ( z ) = max { z, 0 } is the ReLU activation function, b 1 · · · , b m ∈ R d are the weights in the first layer, a 1 , · · · , a m are the weights in the second layer. And we consider the case that the weights in the second layer are fixed, which is a common setting used in previous studies (Arora et al., 2019; Chatterji et al., 2021). We use θ = ( b ⊤ 1 , · · · , b ⊤ m ) ⊤ ∈ R md to denote the concatenation of all trainable weights.

3.2 Gradient Flow Starting from Random Initialization

Gradient Flow. As the limiting dynamics of (Stochastic) Gradient Descent with infinitesimal learning rate (Li et al., 2017; 2019), we study the following Gradient Flow (GF) on the objective function (1):

Notice that the ReLU is not differentiable at 0 , and therefore, the dynamics is defined as a subgradient inclusion flow (Bolte et al., 2010). Here, ∂ ◦ denotes the Clarke subdifferential, which is a generalization of the derivative for non-differentiable functions. Additionally, to address the potential non-uniqueness of gradient flow trajectories, we adopt the definition of solutions for discontinuous systems (Filippov, 2013). For formal definitions, please refer to Appendix B, G, and H.

Random Initialization. We consider GF (2) starting from the following initialization:

Here, 0 < κ 1 < κ 2 ≤ 1 control the initialization scale. It is worth noting that since the distribution N ( 0 , I d /d ) is close to U ( S d -1 ) in high-dimensional settings, our result can be extended to the initialization b k i.i.d. ∼ N ( 0 , κ 2 1 I d /md ) with high probability guarantees.

3.3 Linearly Separable Data beyond Orthogonally Separable

In previous works (Phuong and Lampert, 2021; Wang and Ma, 2022), a special case of the linearly separable dataset was investigated, namely 'orthogonally separable'. A training dataset is orthogonally separable when ⟨ x i , x j ⟩ ≥ 0 for i, j in the same class, and ⟨ x i , x j ⟩ ≤ 0 for i, j in different classes. As mentioned in Section 2, in this case, GF can learn all features and achieve 100% training accuracy quickly, followed by lazy training. In this work, we consider data that is more difficult to learn, which leads to more complicated optimization dynamics. Specifically, we consider the following data.

Assumption 3.1. Consider the linearly separable dataset S = { ( x i , y i ) } i ∈ [ n ] ⊂ R d × R such that angle ∆ ∈ (0 , π/ 2) , and n + , n -are the numbers of positive and negative samples, respectively, with n = n + + n -. We also use p := n + /n -to denote the ratio of n + and n -, which measures the class imbalance. Furthermore, we assume p cos ∆ > 1 .

Remark 3.2. We focus on the training dataset satisfying Assumption 3.1 with a small ∆ ≪ 1 . The margin of the dataset is sin(∆ / 2) , which implies that the separability of this data is much weaker than that of orthogonal separable data. Additionally, the condition p cos ∆ > 1 merely requires a slight imbalance in the data. These two properties work together to produce rich nonlinear behaviors during training.

4 Characterization of Four-phase Optimization Dynamics

In this section, we study the whole optimization dynamics of GF (2) starting from random initialization when training the two-layer ReLU network on linearly separable dataset satisfying Assumption 3.1 and using the loss function (1). To begin with, we introduce some additional notations.

Additional Notations. First, we identify several crucial data-dependent directions under Assumption 3.1. These include two directions that are orthogonal to the data, defined as x ⊥ + := x --⟨ x -, x + ⟩ x + ∥ x --⟨ x -, x + ⟩ x + ∥ and x ⊥ -:= x + -⟨ x + , x -⟩ x -∥ x + -⟨ x + , x -⟩ x -∥ , which satisfy 〈 x + , x ⊥ + 〉 = 〈 x -, x ⊥ -〉 = 0 . Additionally, we define the label-average data direction as µ := z ∥ z ∥ where z = 1 n ∑ n i =1 y i x i . One can verify that ⟨ µ , x + ⟩ > 0 and ⟨ µ , x -⟩ > 0 under the condition p cos ∆ > 1 . In Figure 3, we visualize these directions.

Second, we use the following notations to denote important quantities during the GF training process. We denote the prediction on x + and x -by f + ( t ) := f ( x + ; θ ( t )) , f -( t ) := f ( x -; θ ( t )) . We use

Acc( t ) := 1 n ∑ n i =1 I { y i f ( x i ; θ ( t )) > 0 } to denote the training accuracy at time t . For each neuron k ∈ [ m ] , we use w k ( t ) := b ( t ) / ∥ b ( t ) ∥ and ρ k ( t ) := ∥ b ( t ) ∥ to denote its direction and norm, respectively. To capture the activation dynamics of each neuron k ∈ [ m ] on each data, we use sgn + k ( t ) := sgn ( ⟨ b k ( t ) , x + ⟩ ) to record whether the k -th neuron is activated with respect to x + , and sgn -k ( t ) := sgn ( ⟨ b k ( t ) , x -⟩ ) defined similarly, which we call ReLU activation patterns.

4.1 A Brief Overview of four-phase Optimization Dynamics

We illustrate different phases in the training dynamics by a numerical example. Specifically, we train a network on the dataset that satisfies Assumption 3.1 with p = 4 and ∆ = π/ 15 . The directions and magnitudes of the neurons at some important times are shown in Figure 1, reflecting four different phases on the training behavior and activation patterms. More experiment details and results can be found in Appendix A.1.

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

(a)

t

= 0

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

02

0

.

04

0

.

06

0

.

08

0

.

10

(b)

t

= 200

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

(c)

t

= 50000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

(d)

t

= 60000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

1

2

3

4

5

(e)

t

= 150000

From Fig 1(a) to (b) is the Phase I of the dynamics, marked by a condensation of neurons. Although the initial directions are random, we see that all neurons are rapidly divided into three categories: living positive neurons ( k ∈ K + ) and living negative neurons ( k ∈ K -) condense in one direction each ( µ and x ⊥ + ), while other neurons ( k / ∈ K + ∪ K -) are deactivated forever. From the perspective of loss landscape, GF rapidly escapes from the saddle near 0 where the loss gradient vanishes.

From Fig 1(b) to (c) is the Phase II of the dynamics, in which GF gets stuck into a plateau with training accuracy p 1+ p for a long time T plat before escaping. Once the dynamics escapes from the plateau, the training accuracy rises to a perfect 100% . Moreover, activation patterns do not change in this phase.

From Fig 1(c) to (d) is the Phase III of the dynamics. The phase transition from phase II to phase III sees a rapid deactivation of all the living positive neurons k ∈ K + on x -rapidly, while other activation patterns are unchanged. This leads to a simpler network in phase III, in which only living positive neurons (in K + ) predict x + , and only living negative neurons (in K -) predict x -. Hence, in this phase the GF tries to learn the training data using almost the simplest network by only changing the norms of the neurons.

Finally, Fig 1(d) to (e) shows Phase IV , starting from another 'phase transition' when all the living negative neurons ( k ∈ K -) reactivate simultaneously on x + . This leads to a more complicated network. After the phase transition, the activation patterns no longer change, and the neurons eventually converges towards some specific directions dependent on both data and initialization. Additionally, this direction is not even the local optimal max margin direction.

Overall , the whole dynamics exhibit a simplifying-to-complicating learning trend.

In the following four subsections, we present a meticulously detailed and comprehensive depiction of the whole optimization dynamics and nonlinear behaviors. For clarity, in Figure 2, we first display the timeline of our dynamics and some nonlinear behaviors.

In Appendix A.2, we further validate our theoretical bounds on the key time points in Figure 2 numerically. Additionally, in Appendix A.3, we relax the data Assumption 3.1 by perturbing the

Phase I

Phase II

Phase III

Phase IV

T

plat

Θ

(

p

κ

2

2

∆

2

)

0

T

I

Θ

( √

κ

1

κ

2

)

T

II

Θ

(

p

1

1

-

α

cos ∆

κ

2

2

∆

2

)

T

III

(

1 + Θ(∆

2

)

)

T

II

∞

t

➀➁

➂

➃

➄

➅➆

data with random noise, and our experimental results illustrate that similar four-phase dynamics and nonlinear behaviors persist.

4.2 Phase I. Initial Condensation and Saddle Escape

Let T I = 10 √ κ 1 κ 2 , and we call t ∈ [0 , T I ] Phase I. The theorem below is our main result in Phase I.

Theorem 4.1 (Initial Condensation) . Let the width m = Ω(log(1 /δ )) , the initialization κ 1 , κ 2 = O (1) and κ 1 /κ 2 = O (∆ 8 ) . Then with probability at least 1 -δ , the following results hold at T I :

(S1) Let K + be the index set of living positive neurons at T I , i.e. K + := { k ∈ [ m/ 2] : sgn + k ( T I ) = 1 or sgn -k ( T I ) = 1 } . Then, (i) 0 . 21 m ≤ |K + ( T I ) | ≤ 0 . 29 m . Moreover, for any k ∈ K + , (ii) its norm is small but significant: ρ k ( T I ) = Θ (√ κ 1 κ 2 m ) ; (iii) Its direction is strongly aligned with µ : ⟨ w k ( T I ) , µ ⟩ ≥ 1 -O ( √ κ 1 κ 2 ) -O ( ( κ 1 /κ 2 ) 0 . 55 ) ; (iv) sgn + k ( T I ) = sgn -k ( T I ) = 1 .

(S2) Let K -be the index set of living negative neurons at T I , i.e. K -:= { k ∈ [ m ] -[ m/ 2] : sgn + k ( T I ) = 1 or sgn -k ( T I ) = 1 } . Then, (i) 0 . 075 m ≤ |K -| ≤ 0 . 205 m . Moreover, for any k ∈ K -, (ii) its norm is tiny: ρ k ( T I ) = O ( √ κ 1 κ 2 √ m ( √ κ 1 κ 2 + ∆ p ) ) ; (iii) its direction is aligned with x ⊥ + : 〈 w k ( T I ) , x ⊥ + 〉 1 ( ( √ κ 1 κ 2 p ∆ ) 1 . 6 ) ; (iv) sgn -k ( T I ) = 1 , but sgn + k ( T I ) = 0 .

(S3) For other neuron k / ∈ K + ∪ K -, it dies and remains unchanged during the remaining training process: sgn + k ( t ) ≤ 0 , sgn -k ( t ) ≤ 0 , b k ( t ) ≡ b k ( T I ) , ∀ t ≥ T I .

Initial condensation and simplification. Theorem 4.1 (S1)(S2)(S3) show that, after a short time T I = Θ( √ κ 1 /κ 2 ) , all neurons are implicitly simplified to three categories: K + , K -and others. The living positive neurons k ∈ K + align strongly with µ , and the living negative neurons k ∈ K -align with x ⊥ + and lie on the manifold orthogonal to x + . Other neurons die and remain unchanged during the remaining training process. Moreover, we also estimate tight bounds for |K + | and |K + | . Actually, κ 1 /κ 2 = O (1) can ensure Theorem 4.1 and initial condensation hold (please refer to Appendix C), and we write κ 1 /κ 2 = O (∆ 8 ) here to ensure that the dynamics of later phases hold. In Phase I, the dynamics exhibit a fast condensation phenomenon, i.e., in addition to dead neurons, living positive and negative neurons condense in one direction each.

Saddle-to-Plateau. The network is initially close to the saddle point at 0 (where the loss gradient vanishes). However, Theorem 4.1 (S1) reveals that despite being small, there is a significant growth in the norm of living positive neuron k ∈ K + from Θ( κ 1 / √ m ) to Θ( √ κ 1 κ 2 / √ m ) and the predictions also experience substantial growth (S4). This means that GF escapes from this saddle rapidly. Furthermore, it is worth noting that initial training accuracy can randomly be 0 , 1 1+ p , p 1+ p , or 1 . However, after Phase I, the training accuracy reaches Acc( T I ) = p 1+ p which we will prove as a plateau in the next subsection. Therefore, Phase I exhibits saddle-to-plateau dynamics.

Remark 4.2. Throughout the following subsections, we call the neuron k ∈ K + the 'living positive neuron', the neuron k ∈ K -the 'living negative neuron', and the neuron k / ∈ K + ∪ K -the

'dead neuron'. Moreover, we denote m + := |K + | , m -:= |K -| , and α := m -m + . Notice that Theorem 4.1(S1)(S2) guarantee that 0 < 0 . 075 0 . 29 ≤ α ≤ 0 . 205 0 . 21 < 1 .

Remark 4.3. The results in the following subsections are all based on the occurrence of the events in Theorem 4.1 and with the same settings as Theorem 4.1. So they all hold with probability at least 1 -δ .

Please refer to Appendix C for the proof of Phase I.

4.3 Phase II. Getting Stuck in and Escaping from Plateau

In this phase, we study the dynamics before the patterns of living neurons change again after Phase I. Specifically, we define

and call t ∈ ( T I , T II ] Phase II.

̸

̸

(S3) One of living positive neuron k 0 ∈ K + precisely changes its pattern on x -at T II : lim t → T -II sgn -k 0 ( t ) = 1 and lim t → T + II sgn -k 0 ( t ) = 0 , while all other activation patterns remain unchanged.

Recalling the results in Theorem 4.1, during Phase II, the activation patterns do not change with sgn + k ( t ) = sgn -k ( t ) = 1 for k ∈ K + and sgn + k ( t ) = 0 , sgn -k ( t ) = 1 for k ∈ K -. Theorem 4.4 demonstrates that at the end of Phase II, except for one of living positive neuron k 0 ∈ K + precisely changes its pattern on x -, all other activation patterns remain unchanged.

Theorem 4.5 (Plateau) . We define the hitting time T plat := inf { t ∈ [ T I , T II ] : Acc( t ) = 1 } . Then, (S1) T plat = Θ ( p κ 2 2 ∆ 2 ) ; (S2) ∀ t ∈ [ T I , T plat ] , Acc( t ) ≡ p 1+ p ; (S3) ∀ t ∈ ( T plat , T II ] , Acc( t ) ≡ 1 .

Plateau of training accuracy. According to Theorem 4.5, during Phase II, the training accuracy gets stuck in a long plateau p 1+ p , which lasts for Θ ( p κ 2 2 ∆ 2 ) time. However, once escaping from this plateau, the training accuracy rises to 100% . It is worth noting that this plateau is essentially induced by the dataset. All that's required is only mild imbalance ( p is slightly greater than 1 such that p cos ∆ > 1 ) and a small margin sin(∆ / 2) of two data classes. Notably, if the dataset has an extremely tiny margin (∆ → 0) , then the length of this plateau will be significantly prolonged ( T plat → + ∞ ) , which implies how the data separation can affect the training dynamics. Additionally, using a smaller initialization scale κ 1 of the input layers cannot avoid this plateau.

Please refer to Appendix D for the proof of Phase II.

4.4 Phase III. Simplifying by Neuron Deactivation, and Trying to Learn by Simplest Network

Building upon Phase II, we demonstrate that within a short time, all the living positive neurons K + change their activation patterns, corresponding to a 'phase transition'. Specifically, we define

and we call t ∈ ( T II , T PT II ] the phase transition from Phase II to Phase III.

Theorem 4.6 (Phase Transition) . (S1) T PT II = ( 1 + O ( √ κ 1 κ 3 2 )) T II ; (S2) sgn + k ( T PT II ) = 1 and sgn -k ( T PT II ) = 0 for any k ∈ K + ; sgn + k ( T PT II ) = 0 and sgn -k ( T PT II ) = 1 for any k ∈ K -.

Neuron deactivation. As shown in Theorem 4.6 (S2), after the phase transition, all the living positive neurons k ∈ K + undergo deactivation for x -, i.e., sgn -k ( t ) changes from 1 to 0, while other activation patterns remain unchanged. Furthermore, Theorem 4.6 (S1) reveals that the phase transition is completed quite quickly by using sufficiently small initialization value κ 1 , κ 2 . A smaller initialization value leads to a more precise initial condensation w k ( T I ) ≈ µ , causing all living positive neurons to remain closer together before T II and thus changing their patterns nearly simultaneously.

Self-simplifying. As a result, the network implicitly simplifies itself through the deactivation behavior. At T PT III , only living negative neurons k ∈ K + are used for predicting on x -, i.e., f -( T PT II ) = κ 2 √ m ∑ k ∈K -σ ( 〈 b k ( T PT II ) , x -〉 ) . In contrast, during Phase II, both living positive and living negative neurons jointly predict on x -, i.e., f -( t ) = κ 2 √ m ∑ k ∈K + σ ( ⟨ b k ( t ) , x -⟩ ) -κ 2 √ m ∑ k ∈K -σ ( ⟨ b k ( t ) , x -⟩ ) . As indicated in Table 1, two classes of activation patterns are simplified from (1 , 0) to (0 , 0) , while others do not change.

After this phase transition, we study the dynamics before the patterns of living neurons change again. Specifically, we define

̸

Theorem 4.7 (End of Phase III) . T III = ( 1 + Θ(∆ 2 ) ) T II .

Learning by simplest network. During t ∈ ( T PT II , T III ) , all activation patterns do not change. This ensures that f + ( t ) = κ 2 √ m ∑ k ∈K + σ ( ⟨ b k ( t ) , x + ⟩ ) , while f -( t ) = -κ 2 √ m ∑ k ∈K -σ ( ⟨ b k ( t ) , x -⟩ ) . Additionally, by using sufficiently small κ 1 , the neurons in K + and K -keep close together respectively before T III , making the network close to a simple two-neuron network consisting of one positive neuron and one negative neuron. Please refer to Appendix E for more details. Furthermore, this pattern scheme is almost the "simplest" way to ensure binary classification: the living positive neurons only predict positive data x + while the living negative neurons only predict negative data x -. Therefore, GF tries to learn by this almost simplest network in this phase.

Please refer to Appendix E for the proof of Phase III.

4.5 Phase IV. Complicating by Neuron Reactivation, and Directional Convergence

Phase IV begins with an instantaneous phase transition at time T III .

Theorem 4.8 (Phase Transition) . All living negative neuron k ∈ K -simultaneously change their patterns on x + at T III : lim t → T -III sgn + k ( t ) = 0 , lim t → T + III sgn + k ( t ) = 1 , while others remain unchanged.

Neuron reactivation. According to Theorem 4.8, all of living negative neurons k ∈ K -reactivate simultaneously on x + at T III : sgn + k ( t ) changes from 0 to 1, while other activation patterns remain unchanged.

Self-Complicating. Along with the reactivation behavior, GF implicitly complicates itself. In Phase III, only living negative neurons k ∈ K + are used to predict on x + , i.e., f + ( t ) = κ 2 √ m ∑ k ∈K + σ ( ⟨ b k ( t ) , x + ⟩ ) . In contrast, after the phase transition at T III , both living positive and living negative neurons jointly predict on x + , i.e. f + ( t ) = κ 2 √ m ∑ k ∈K + σ ( ⟨ b k ( t ) , x + ⟩ ) -κ 2 √ m ∑ k ∈K -σ ( ⟨ b k ( t ) , x + ⟩ ) . As indicated in Table 1, two classes of activation patterns are complicated from (0 , 0) to (0 , 1) , while others do not change.

In this phase, we study the dynamics before activation patterns change again after the phase transition in Theorem 4.8. We define the hitting time:

̸

and we call t ∈ ( T III , T IV ] Phase IV.

̸

Theorem 4.9 (Phase IV) . (S1) T IV = + ∞ . Moreover, for any t > T III , (S2) all activation patterns do not change; (S3) the loss converges with L ( θ ( t )) = Θ ( 1 p 1 1 -α cos ∆ + κ 2 2 ∆ 2 ( t -T III ) ) .

Theorem 4.9 illustrates that all activation patterns never change again after the phase transition at T III with sgn + k ( t ) = 1 , sgn -k ( t ) = 0 for any k ∈ K + and sgn + k ( t ) = sgn -k ( t ) = 1 for any k ∈ K -. Additionally, the loss converges with the polynomial rate Θ(1 /κ 2 2 ∆ 2 t ) . Furthermore, we present the following theorem about the convergent direction of each neuron.

̸

Theorem 4.10 (Directional Convergence) . The limit lim t → + ∞ θ ( t ) ∥ θ ( t ) ∥ 2 exists and denoted by θ = ( b ⊤ 1 , · · · , b ⊤ m ) ⊤ ∈ S md -1 . Moreover, (i) for any k / ∈ K + ∪ K -, b k = 0 ; (ii) for any k ∈ K + , b k ≡ v + = C ( x + -x -cos ∆ ) ; (iii) for any k ∈ K -, b k ≡ v -= C (( 1 + sin 2 ∆ α (1+cos ∆) ) x --x + ) , where C > 0 is a scaling constant such that ∥ ∥ θ ∥ ∥ 2 = 1 . (iv) Additionally, f -( θ ) = -f + ( θ ) > 0 .

Initialization-dependent Directional Convergence. As an asymptotic result, Theorem 4.10 provides the final convergent direction of GF. All living positive neurons ( k ∈ K + ) directionally converge to v + ∥ x ⊥ -with ⟨ v + , x + ⟩ > 0 and ⟨ v + , x -⟩ = 0 , while all living negative neurons ( k ∈ K -) directionally converge to v -∈ span { x + , x -} with ⟨ v -, x + ⟩ > 0 and ⟨ v -, x -⟩ > 0 . It is worth noting that v -directly depends not only on the data but also on the ratio α = |K -| / |K + | (defined in Remark 4.2). Recalling the results in Phase I, α lies in a certain range with high probability; but it is still a random variable due to its dependence on random initialization. Different initializations may lead to different values |K -| / |K + | at the end of Phase I, eventually causing different convergent directions in Phase IV.

Non-(Local)-Max-Margin Direction. Lastly, we study the implicit bias of the final convergence rate. According to Lyu and Li (2019); Ji and Telgarsky (2020) and our results above, θ in Theorem 4.10 must be a KKT direction of some max-margin optimization problem. However, it is not clear whether the direction θ is actually an actual optimum of this problem. Surprisingly, in next Theorem, we demonstrate that the final convergent direction is not even a local optimal direction of this problem, which enlightens us to rethink the max margin bias of ReLU neural networks.

Theorem 4.11 (Implicit Bias) . The final convergent direction θ (in Theorem 4.10) is a KKT direction of the max-margin problem min : 1 2 ∥ θ ∥ 2 s . t . y i f ( x i ; θ ) ≥ 1 , i ∈ [ n ] . However, θ is not even a local optimal direction of this problem.

Please refer to Appendix F for the proof of Phase IV.

5 Discussion and Comparison on Nonlinear Behaviors

Throughout the whole training process in Section 4, we divide the phases based on the evolution of ReLU activation patterns. During Phase I, as well as the beginning of Phase II and III, numerous activation patterns undergo rapid changes. Table 1 summarizes the evolution of activation patterns for all living neurons after Phase I. These results are also numerically validated in Figure 1.

Simplifying-to-Complicating. In phase I, GF simplifies all the neurons from random directions into three categories: living positive neurons K + and living negative neurons K -condense in one direction each, which other neurons are deactivated forever. After Phase I, as shown in Table 1, the two classes of activation patterns change from (1 , 0) simplify → (0 , 0) complicate → (0 , 1) , while other patterns remain unchanged. Therefore, the evolution of activation patterns exhibits a simplifyingto-complicating learning trend, which also implies that the network trained by GF learn features in increasing complexity.

̸

Comparison with NTK. In the lazy regime such as NTK, most neurons keep close to the initialization and most activation patterns do not change during training. Specifically, for any training data x i , 1 m ∑ k ∈ [ m ] I { sgn ( ⟨ b k ( t ) , x i ⟩ ) = sgn ( ⟨ b k (0) , x i ⟩ ) } = o (1) , ∀ t > 0 (Du et al., 2018). However, our work stands out from lazy regime as activation patterns undergo numerous changes during training. In Phase I, initial condensation causes substantial changes in activation patterns, which is similarly observed in (Phuong and Lampert, 2021). Furthermore, even after Phase I, there are notable modifications in activation patterns. As shown in Table 1, the proportion of changes in

̸

activation patterns for any given training data is the Θ(1) , as compared with the o (1) in NTK regime. Specifically, at any t > T III , 1 m ∑ k ∈ [ m ] I { sgn + k ( t ) = sgn + k ( T I ) } = 1 m |K -| = Θ(1) and 1 m ∑ k ∈ [ m ] I { sgn -k ( t ) = sgn -k ( T I ) } = 1 m |K + | = Θ(1) . On the other hand, in our analysis, the requirement on the network's width m is only m = Ω(log(1 /δ )) (Theorem 4.1), regardless of data parameters p, ∆ , while NTK regime requires a much larger width m = Ω(log( p/δ ) / ∆ 6 ) (Ji and Telgarsky, 2019).

̸

Comparison with Phuong and Lampert (2021); Lyu et al. (2021); Wang and Ma (2022); Boursier et al. (2022). Beyond lazy regime and local analysis, these works also characterize the entire training dynamics and analyze a few nonlinear behaviors. Now we compare our results with these works in detail. (i) While Lyu et al. (2021) focuses on training Leaky ReLU NNs, our work and the other three papers study ReLU NNs. It is worth noting that the dynamics of Leaky ReLU NNs differ from ReLU due to the permanent activation of Leaky ReLU ( σ ′ ( · ) ≥ α > 0) . (ii) Initial condensation is also proven in (Lyu et al., 2021; Boursier et al., 2022), and the condensation directions are some types of data averages. In our work, neurons can aggregate towards not only the average direction µ , but also another direction x ⊥ + . Moreover, we also estimate the number of neurons that condense into two directions. (iii) Saddle-to-saddle dynamics are proven in (Phuong and Lampert, 2021) for square loss, where the second saddle is about training loss and caused by incomplete fitting. However, our work focus on classification with exponential loss and exhibit a similar saddle-to-plateau dynamics, where the plateau is about training accuracy, caused by incomplete feature learning. (iv) Phased feature learning. In (Phuong and Lampert, 2021; Wang and Ma, 2022), all features can be rapidly learned in Phase I (accuracy = 100% ), followed by lazy training. However, for practical tasks on more complex data, NNs can hardly learn all features in such short time. In our work, the data is more difficult to learn, resulting in incomplete feature learning in Phase I (accuracy < 100% ). Subsequently, NNs experience a long time to learn other features completely. Such multi-phase feature leaning dynamics are closer to practical training process. (v) Neuron reactivation and deactivation. For ReLU NNs, The evolution of activation patterns is one of the essential causes of nonlinear dynamics. In (Phuong and Lampert, 2021; Wang and Ma, 2022), activation patterns only change rapidly in Phase I, after which they remain unchanged. In (Boursier et al., 2022), their lemma 6 shows that their dynamics lack neuron reactivation. However, in our dynamics, even after Phase I, our dynamics exhibit significant neuron deactivation and reactivation as discussed in Table 1. (vi) The final convergent directions are also derived in (Phuong and Lampert, 2021; Lyu et al., 2021; Boursier et al., 2022), which only depend on the data. However, in our setting, the convergent direction is more complicated, determined by both data and random initialization. (vii) Furthermore, our four-phase dynamics demonstrate the whole evolution of activation patterns during training and reveal a general simplifying-to-complicating learning trend.

In summary, our whole four-phase optimization dynamics capture more nonlinear behaviors than these works. Furthermore, we conduct a more thorough and detailed theoretical analysis of these nonlinear behaviors, providing a more systematic and comprehensive understanding.

6 Conclusion and Future Work

In this work, we study the optimization dynamics of ReLU neural networks trained by GF on a linearly separable data. Our analysis captures the whole optimization process starting from random initialization to final convergence. Throughout the whole training process, we reveal four different phases and identify rich nonlinear behaviors theoretically. However, theoretical understanding of the training of NNs still has a long way to go. For instance, although we conduct a fine-grained analysis of GF, the dynamics of GD are more complex and exhibit other nonlinear behaviors such as progressive sharpening and edge of stability (Wu et al., 2018; Jastrz˛ ebski et al., 2019; Cohen et al., 2021; Ma et al., 2022; Li et al., 2022; Damian et al., 2022; Zhu et al., 2022; Ahn et al., 2022a;b). Additionally, unlike GD, SGD uses only mini-batches of data and injects noise (Zhu et al., 2019; Thomas et al., 2020; Feng and Tu, 2021; Liu et al., 2021; Ziyin et al., 2022; Wu et al., 2022; Wojtowytsch, 2023; Wang and Wu, 2023) in each iteration, which can have a pronounced impact on the optimization dynamics and nonlinear behaviors. Better understanding of the nonlinear behaviors during GD or SGD training is an important direction of future work.

Acknowledgments and Disclosure of Funding

We thank Prof. Weinan E, Prof. Lei Wu, Prof. Zhi-Qin John Xu and anonymous reviewers for helpful suggestions. Mingze Wang is supported in part by the National Key Basic Research Program of China: 2015CB856000.

References

Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory , pages 4782-4887. PMLR, 2022a.

Emmanuel Abbe, Elisabetta Cornacchia, Jan Hazla, and Christopher Marquis. An initial alignment between neural network and target is needed for gradient descent to learn. In International Conference on Machine Learning , pages 33-52. PMLR, 2022b.

Emmanuel Abbe, Enric Boix-Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. arXiv preprint arXiv:2302.11055 , 2023.

Kwangjun Ahn, Sébastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via the" edge of stability". arXiv preprint arXiv:2212.07469 , 2022a.

Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. In International Conference on Machine Learning , pages 247-257. PMLR, 2022b.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization. In International Conference on Machine Learning , pages 242-252. PMLR, 2019.

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pages 322-332. PMLR, 2019.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017.

Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. International Conference on Learning Representations , 2022.

Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on learning theory , pages 483-513. PMLR, 2020.

Jérôme Bolte, Aris Daniilidis, Olivier Ley, and Laurent Mazet. Characterizations of łojasiewicz inequalities: subgradient flows, talweg, convexity. Transactions of the American Mathematical Society , 362(6):3319-3363, 2010.

Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. Advances in Neural Information Processing Systems , 2022.

Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. Sgd learns overparameterized networks that provably generalize on linearly separable data. arXiv preprint arXiv:1710.10174 , 2017.

Niladri S Chatterji, Philip M Long, and Peter L Bartlett. When does gradient descent with logistic loss find interpolating two-layer networks? Journal of Machine Learning Research , 22(159):1-48, 2021.

Zhengan Chen, Yuqing Li, Tao Luo, Zhangchen Zhou, and Zhi-Qin John Xu. Phase diagram of initial condensation for two-layer neural networks. arXiv preprint arXiv:2303.06561 , 2023.

Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems , 31, 2018.

Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory , pages 1305-1338. PMLR, 2020.

Francis H Clarke, Yuri S Ledyaev, Ronald J Stern, and Peter R Wolenski. Nonsmooth analysis and control theory , volume 178. Springer Science & Business Media, 2008.

Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065 , 2021.

Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594 , 2022.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning , pages 1675-1685. PMLR, 2019.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054 , 2018.

Joydeep Dutta, Kalyanmoy Deb, Rupesh Tulshyan, and Ramnik Arora. Approximate kkt points and a proximity measure for termination. Journal of Global Optimization , 56(4):1463-1499, 2013.

Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences , 118(43):e2103091118, 2021.

Yu Feng and Yuhai Tu. The inverse variance-flatness relation in stochastic gradient descent is critical for finding flat minima. Proceedings of the National Academy of Sciences , 118(9), 2021.

Aleksei Fedorovich Filippov. Differential equations with discontinuous righthand sides: control systems , volume 18. Springer Science & Business Media, 2013.

XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. arXiv preprint arXiv:2106.02073 , 2021.

Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation , 9(1):1-42, 1997.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1806.07572 , 2018.

Arthur Jacot, François Ged, Berfin ¸ Sim¸ sek, Clément Hongler, and Franck Gabriel. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.15933 , 2021.

Stanisław Jastrz˛ ebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the relation between the sharpest directions of dnn loss and the sgd step length. International Conference on Learning Representations , 2019.

Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks. arXiv preprint arXiv:1909.12292 , 2019.

Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems , 33:17176-17186, 2020.

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations , 2016.

Daniel Kunin, Atsushi Yamamura, Chao Ma, and Surya Ganguli. The asymmetric maximum margin bias of quasi-homogeneous neural networks. International Conference on Learning Representations , 2023.

Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning , pages 2101-2110. PMLR, 2017.

Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. The Journal of Machine Learning Research , 20 (1):1474-1520, 2019.

Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss?-a mathematical framework. arXiv preprint arXiv:2110.06914 , 2021.

Zhouzi Li, Zixuan Wang, and Jian Li. Analyzing sharpness along gd trajectory: Progressive sharpening and edge of stability. arXiv preprint arXiv:2207.12678 , 2022.

Kangqiao Liu, Liu Ziyin, and Masahito Ueda. Noise and fluctuation of finite learning rate stochastic gradient descent. In International Conference on Machine Learning , pages 7045-7056. PMLR, 2021.

Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer relu neural networks at infinite-width limit. The Journal of Machine Learning Research , 22(1):3327-3373, 2021.

Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.05890 , 2019.

Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems , 34, 2021.

Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks. Advances in Neural Information Processing Systems , 34:16805-16817, 2021.

Chao Ma, Daniel Kunin, Lei Wu, and Lexing Ying. Beyond the quadratic approximation: the multiscale structure of neural network loss landscapes. arXiv preprint arXiv:2204.11326 , 2022.

Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient descent quantizes relu network features. arXiv preprint arXiv:1803.08367 , 2018.

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on Learning Theory , pages 2388-2464. PMLR, 2019.

Rotem Mulayoff, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability: A view from function space. Advances in Neural Information Processing Systems , 34:17749-17761, 2021.

Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, and Daniel Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning , pages 4683-4692. PMLR, 2019.

Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity. arXiv preprint arXiv:1905.11604 , 2019.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40): 24652-24663, 2020.

Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. arXiv preprint arXiv:2304.00488 , 2023.

Mary Phuong and Christoph H Lampert. The inductive bias of relu networks on orthogonally separable data. In International Conference on Learning Representations , 2021.

Aristide Rahaman, Nasim xd Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning , pages 5301-5310. PMLR, 2019.

Itay Safran, Gal Vardi, and Jason D Lee. On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. Advances in Neural Information Processing Systems , 35:32667-32679, 2022.

Andrew Saxe, Shagun Sodhani, and Sam Jay Lewallen. The neural race reduction: Dynamics of abstraction in gated networks. In International Conference on Machine Learning , pages 1928719309. PMLR, 2022.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1): 2822-2878, 2018.

Valentin Thomas, Fabian Pedregosa, Bart Merriënboer, Pierre-Antoine Manzagol, Yoshua Bengio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect on optimization and generalization. In International Conference on Artificial Intelligence and Statistics , pages 3503-3513. PMLR, 2020.

Gal Vardi. On the implicit bias in deep-learning algorithms. Communications of the ACM , 66(6): 86-93, 2023.

Gal Vardi, Ohad Shamir, and Nati Srebro. On margin maximization in linear and relu networks. Advances in Neural Information Processing Systems , 35:37024-37036, 2022.

Mingze Wang and Chao Ma. Early stage convergence and global convergence of training mildly parameterized neural networks. Advances in Neural Information Processing Systems , 2022.

Mingze Wang and Lei Wu. The noise geometry of stochastic gradient descent: A quantitative and analytical characterization. arXiv preprint arXiv:2310.00692 , 2023.

Stephan Wojtowytsch. Stochastic gradient descent with noise of machine learning type part i: Discrete time analysis. Journal of Nonlinear Science , 33(3):45, 2023.

Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory , pages 3635-3673. PMLR, 2020.

Lei Wu and Weijie J Su. The implicit regularization of dynamical stability in stochastic gradient descent. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 37656-37684. PMLR, 2023.

Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems , 31: 8279-8288, 2018.

Lei Wu, Mingze Wang, and Weijie Su. When does sgd favor flat minima? a quantitative characterization via linear stability. Advances in Neural Information Processing Systems , 2022.

Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523 , 2019.

Yaoyu Zhang, Yuqing Li, Zhongwang Zhang, Tao Luo, and Zhi-Qin John Xu. Embedding principle: A hierarchical structure of loss landscape of deep neural networks. Journal of Machine Learning , 1(1):60-113, 2022. ISSN 2790-2048.

Hanxu Zhou, Zhou Qixuan, Zhenyuan Jin, Tao Luo, Yaoyu Zhang, and Zhi-Qin Xu. Empirical phase diagram for three-layer neural networks with infinite width. Advances in Neural Information Processing Systems , 35:26021-26033, 2022a.

Hanxu Zhou, Zhou Qixuan, Tao Luo, Yaoyu Zhang, and Zhi-Qin Xu. Towards understanding the condensation of neural networks at initial training. Advances in Neural Information Processing Systems , 35:2184-2196, 2022b.

Xingyu Zhu, Zixuan Wang, Xiang Wang, Mo Zhou, and Rong Ge. Understanding edge-of-stability training dynamics with a minimalist example. arXiv preprint arXiv:2210.03294 , 2022.

Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference on Machine Learning , pages 7654-7663. PMLR, 2019.

Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems , 34:29820-29834, 2021.

Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of minibatch noise in SGD. In International Conference on Learning Representations , 2022.

Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks. arXiv preprint arXiv:1811.08888 , 2018.

Contents

Appendix

A Experimental Details

All experiments are conducted on a MacBook pro 13 (M2) only using CPU. See the code at https://github.com/wmz9/Understanding_Multi-phase_Optimization_NeurIPS2023 .

A.1 Experiments on standard Dataset

We train the two-layer network on the dataset that satisfies Assumption 3.1 with d = 20 , p = 4 and ∆ = π/ 15 . Specifically, we choose the network width m = 100 ; the initialization scale κ 1 = 0 . 1 , κ 2 = 1 ; the small learning rate η = 0 . 01 .

In Figure 3, we show some key data directions in this dataset, as well as the training accuracy, which contains a long plateau. Furthermore, in Figure 4, we provide the evolution of all neurons during training from t = 0 to t = 150000 , which is a more complete version of Figure 1.

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Data Directions

x

+

x

-

x

⊥

+

x

⊥

-

µ

0

50000

100000

150000

20

40

60

80

100

Acc

Acc(

t

)

training accuracy

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

(a)

t

= 0

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

02

0

.

04

0

.

06

0

.

08

0

.

10

(b)

t

= 200

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

(c)

t

= 2000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

(d)

t

= 30000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

(e)

t

= 50000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

(f)

t

= 60000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

(g)

t

= 80000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

1

2

3

4

(h)

t

= 120000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

1

2

3

4

5

(i)

t

= 150000

A.2 Numerical validation on our theoretical bounds

We conduct experiments to validate our theoretical bounds on T plat and T III under different p and ∆ , and the results are shown in 2.

In the first experiment (1st and 2nd subtable), we fix p = 4 for change ∆ ; in the second experiment (3rd and 4th subtable), we fix ∆ = π/ 15 and change p . As for other hyperparameters (such as κ 1 , κ 2 , d, m ), we keep the same scales as our setups in Appendix A.1.

We have two main conclusions: (1) four training phases in our theory persistently exist; (the same as Fig 1 in Appendix A, and be omitted due to the limited space) (2) our theoretical estimates on T plat and T III are relatively tight, basically in the same magnitude as the realistic time.

-

A.3 Experiments on Noisy Dataset

We conduct numerical experiments on the setting of adding small stochastic noise on top of x + and x -, a little bit more realistic setting. Specifically, in span { x + , x -} , we perturb the angles of n + -1 instances of x + and n --1 instances of x -using stochastic noise ξ ∼ Unif([0 , ∆ / 4]) .

In Figure 5, we visualize (i) the evolution of each neuron throughout the training process; (ii) some key data directions; (iii) the evolution of training accuracy.

From the numerical results in Figure 5, we have two main conclusions: (1) we ascertain that the same four-phase optimization dynamics and nonlinear behaviors persist, even for our dataset with small stochastic noise; (2) a slight difference is that there is more than one plateau of training accuracy in Phase II. The reason is that for noisy data, GF needs to learn negative data one by one in Phase II. For example, three distinct negative data are employed in this experiment, so three plateaus of training accuracy emerge ( 12 / 15 , 13 / 15 , and 14 / 15 ).

µ

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Data Directions

x

+

x

+

+noise

x

-

x

-

+noise

x

⊥

+

x

⊥

-

(a) key data directions

0

100000

200000

300000

400000

20

40

60

80

100

Acc

Acc(

t

)

training accuracy

(b) training acc

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

001

0

.

002

0

.

003

0

.

004

0

.

005

0

.

006

0

.

007

(c)

t

= 0

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

02

0

.

04

0

.

06

0

.

08

0

.

10

(d)

t

= 200

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

(e)

t

= 2000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

1

.

6

(f)

t

= 30000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

(g)

t

= 65000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

(h)

t

= 90000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

1

2

3

4

5

6

(i)

t

= 200000

0

°

45

°

90

°

135

°

180

°

225

°

270

°

315

°

2

4

6

8

(j)

t

= 400000

B Proof Preparation

Selection of initialization parameters.

For the data satisfying Assumption 3.1, we consider the regime that ∆ ≪ 1 with p cos ∆ > 1 . During the entire proof, we select the initialization scale κ 1 , κ 2 as follows:

Gradient Flow. In general, for any k ∈ [ m ] , the GF dynamics of b k ( t ) can be written as

where ∂ ◦ is Clarke's subdifferential defined in Defnition G.1. Notice that if L is continuously differentiable at θ , then ∂ ◦ L ( θ ) = {∇L ( θ ) } is unique.

However, for discontinuous differentiable points of L , the differential inclusion flow d θ d t ∈ ∂ ◦ L ( θ ) may not be unique. To study a more specific dynamics, we also utilize Definition H.1 to determine GF at some of such points, which overcomes non-uniqueness of GF trajectories to some extent. It is worth noting that Definition H.1 and Definition G.1 are compatible and specifically, the dynamics defined in Definition H.1(Case I, III) lie in the convex hull defined in Definition G.1.

Remark B.1. In (Lyu et al., 2021), the non-branching starting point Assumption is employed to address the technical challenge of non-uniqueness in GF trajectories. By comparison, in this work, we do not need this assumption. We adopt Definition H.1 to uniquely determine the Gradient Flow trajectories theoretically near some discontinuous differential regions, such as 'Ridge', 'Valley', and 'Refraction edge' discussed in Section I.2 in (Lyu et al., 2021).

Additionally, in the following sections, we may rewrite this dynamics accordingly, such as specific forms and dynamics decomposition.

Additional Notations. As a similar description to sgn + k ( · ) and sgn -k ( · ) , we also employ the following six manifolds to characterize activation patterns (by judging which manifold the neuron w k belongs to):

As one of our interested manifolds, the border ∂ ( M + + ∩ M + -) can be divided into

Furthermore, we also utilize the following notations:

̸

Notice that for the direction of a neuron b k = 0 , using P + + , P 0 + , P -+ , P + -, P 0 -, P --to describe b k is equivalent to using M + + , M 0 + , M -+ , M + -, M 0 -, M --to describe w k .

Lemma B.2 (Dead neurons keep dead) . For the k -th neuron, if there exists a t 0 ≥ 0 , s.t. w k ( t 0 ) ∈ M -+ ∩M --, then it dies and remains unchanged during the remaining training: w k ( t ) ∈ M -+ ∩M --and b k ( t ) ≡ b k ( t 0 ) for any t ≥ t 0 .

Proof of Lemma B.2. A straightforward calculation.

The above fact is a basic fact in our setting, which illustrates that if the neuron b k is deactivated for both data x + and x -at some time, then it remains 'dead' forever.

It is worth mentioning that if the neuron b k is deactivated on x + but activated on x -at some time, it can still reactivate on x + later, which is one of the important reasons why our ReLU optimization dynamics are complicated.

C Proofs of Optimization Dynamics in Phase I

In this section, we conduct a detailed analysis of the training dynamics of each neuron in Phase I. The main proof idea is to decompose neurons' dynamics into tangential and radial dynamics. For small initialization, in Phase I, the radial increasing of neurons is much slower than their tangential velocity, which can result in condensation. However, the main challenges arise from the initial direction's randomness and ReLU's discontinuous derivative, leading to eight categories of neuron dynamics. Moreover, it is also nontrivial and requires meticulous analysis to estimate the number of two classes of living neurons at the end of Phase I.

We define the time and call t ∈ [0 , T I ] 'Phase I'.

Recall that the selection (3) about initialization parameters can guarantee the whole four-phase optimization dynamics. Nevertheless, when we focus on Phase I, κ 2 = O (1) and κ 1 /κ 2 = O (1) suffice to ensure the dynamics in Phase I. Specifically, during the proof of Phase I, we can use the following selection (6) on κ 1 , κ 2 , which is much weaker than (3):

For simplicity, we assume ∆ ≤ 1 5 . And for convenience, we assume p ≥ 5 . It is worth mentioning that our proof approach also applies for p = 2 , 3 , 4 , with at most one absolute constant difference.

Prepared for the analysis in Phase I, we decompose the dynamics of b k ( t ) into the radial movement ρ k ( t ) ∈ R and the tangential movement w k ( t ) ∈ S d -1 satisfied to b k ( t ) = ρ k ( t ) w k ( t ) .

Lemma C.1 (Dynamics decomposition) . For any k ∈ [ m ] , the dynamics of b k ( t ) can be decomposed into the radial movement ρ k ( t ) ∈ R and the tangential movement w k ( t ) ∈ S d -1 :

where ρ k ( t ) = ∥ b k ( t ) ∥ , w k ( t ) = b k ( t ) / ∥ b k ( t ) ∥ and

Proof of Lemma C.1.

From the dynamics of b k ( t ) :

we have

Prepared for the analysis of the neurons' dynamics, we establish a rough estimate about the norm and prediction growth of each neuron in Phase I, and we will improve it later.

Lemma C.2 (A Rough Estimate of Norm and Prediction in Phase I) .

For any t ≤ T I , k ∈ [ m ] , and i ∈ [ n ] , we have the following estimates:

Proof of Lemma C.2.

First, we define the hitting time

κ κ √

From | f i (0) | ≤ m 2 √ m 1 √ m = κ 1 κ 2 < κ 1 κ 2 and the continuity of f i ( · ) , we know T √ κ 1 κ 2 > 0 . Then we will prove T I = 10 √ κ 1 κ 2 T √ κ 1 κ 2 .

For any k ∈ [ m ] , i ∈ [ n ] , and t ≤ T √ κ 1 κ 2 , we have the following estimates.

Recalling the definition of F k ( t ) (8), we have

Recalling the dynamics (7), for any t ≤ T √ κ 1 κ 2 ,

Then combining ρ k (0) = κ 1 √ m , for any t ≤ T √ κ 1 κ 2 ,

So for any t ≤ T I , we have

From the definition of T √ κ 1 κ 2 , we have proved T I ≤ T √ κ 1 κ 2 .

Moreover, from this proof, we know (9) holds for any t ≤ T I . Moreover, by Mean Value Theorem,

Now we delve into the optimization dynamics of all neurons in Phase I in the following Section C.1 and C.2.

C.1 The Dynamics of Positive Neurons

According to the initial direction, all positive neurons ( s k = 1 ) can be divided into the following four classes.

In the following four lemmas, we will prove the dynamics for these four classes of positive neurons. In summary, in Phase I ( t < T I ), some of positive neurons align well with the direction µ , and their norms experiment a small but significant increase, while other positive neurons go dead.

Lemma C.3 (Positive, M + + ∩ M + -) .

For positive neuron k ∈ { k ∈ [ m/ 2] : w k (0) ∈ M + + ∩ M + -} , at the end of Phase I, it holds that

Proof of Lemma C.3.

Step I. The neuron stays in M + + ∩ M + -for any t ≤ T I .

We do the following analysis for any k ∈ { k ∈ [ m/ 2] : w k (0) ∈ M + + ∩ M + -} , i.e. s k = 1 , ⟨ w k (0) , x + ⟩ > 0 , and ⟨ w k (0) , x -⟩ > 0 .

First, we define the hitting time

and we aim to prove that T hit does not exist. From the definition of T hit and (4), the dynamics of the neuron is:

From T hit ≤ T I and Lemma C.2, we have ∣ ∣ e -y i f i ( t ) -1 ∣ ∣ ≤ 1 . 1 √ κ 1 κ 2 for any t ≤ T hit . Then we have:

Hence, for any t ≤ T hit , ⟨ b k ( t ) , x + ⟩ > ⟨ b k (0) , x + ⟩ > 0 and ⟨ b k ( t ) , x -⟩ > ⟨ b k (0) , x -⟩ > 0 . According to the definition of T hit , we have proved that T hit does not exist, which means the neuron stays in M + + ∩ M + -for any t ≤ T I .

Step II. Estimate the evolution of ⟨ w k ( t ) , µ ⟩ .

With the help of Step I , we were able to determine the dynamics for t ≤ T I . For any t ≤ T I , we can do the following estimate.

From (7), the tangential dynamics of the neuron is

Recalling the definitions of µ and z , we can estimate the difference between F k ( t ) and z :

The dynamics of ⟨ w k ( t ) , µ ⟩ is:

And it can be estimated by:

Noticing w k (0) ∈ M + + ∩ M + -, we have:

Now we consider the following auxiliary ODE:

and let U ( t ) is the solution of (13). Due to (11) (12), we know that ⟨ w k ( t ) , µ ⟩ is an upper solution of ODE (13). From the Comparison Principle of ODEs, we know this means:

Hence, in order to estimate ⟨ w k ( t ) , µ ⟩ , we only need to study the solution of ODE (13). It is easy to verify that the solution of (13) satisfies

Then we have:

which means

Hence, we have the estimate of ⟨ w k ( t ) , µ ⟩ :

Specifically, we have:

Step III. A finer estimate of ρ k ( T I ) .

In this step, we will estimate the lower bound and upper bound for ρ k ( T I ) .

First, lemma C.2 gives us the upper bound for ρ k ( T I ) :

Now we focus on the estimate of the lower bound. Recalling the dynamics of ρ k ( t ) (7), for any t ≤ T I ,

We denote T 0 = 10 κ 1 κ 2 , and it is easy to verify:

So we have:

Lemma C.4 (Positive, M + + ∩ M --) . For positive neuron k ∈ { k ∈ [ m/ 2] : w k (0) ∈ M + + ∩ M --} , at the end of Phase I, it holds that

Proof of Lemma C.4.

Step I. The neuron must arrives in M + + ∩ M 0 -in O ( κ 1 ∆ κ 2 ) time.

We do the following analysis for any k ∈ { k ∈ [ m/ 2] : w k (0) ∈ M + + ∩ M --} , i.e. s k = 1 , ⟨ w k (0) , x + ⟩ > 0 , and ⟨ w k (0) , x -⟩ ≤ 0 .

The case ⟨ w k (0) , x -⟩ = 0 is trivial. Then we only need to consider the case ⟨ w k (0) , x -⟩ < 0 . First, we define the hitting time

and we aim to estimate T hit and prove w k ( T hit ) ∈ M + + ∩ M 0 -.

We focus on the dynamics of ⟨ b k ( t ) , x + ⟩ and ⟨ b k ( t ) , x -⟩ .

From the definition of T hit and (4), the dynamics of the neuron is:

Then we have

It is clear d ⟨ b k ( t ) , x + ⟩ d t > 0 , so ⟨ b k ( t ) , x + ⟩ > ⟨ b k (0) , x + ⟩ > 0 for any t ≤ T hit . If we denote T hit , -:= inf { t ∈ (0 , T I ] : ⟨ w k ( t ) , x -⟩ ≥ 0 } ,

then it holds:

So we only need to estimate T hit , -. Due to T hit ≤ T I ≤ T init and Lemma C.2, for any t ≤ T hit , we have ∣ ∣ e -y i f i ( t ) -1 ∣ ∣ ≤ 0 . 11 . Then for any t ≤ T hit , we have:

Recalling ⟨ w k (0) , x + ⟩ > 0 and ⟨ w k (0) , x -⟩ < 0 , with the help of Lemma I.2, we have ⟨ w k (0) , x -⟩ > -sin ∆ . Combining the two estimate, we have:

Hence,

Step II. Dynamics after arriving in the manifold M + + ∩ M 0 -.

In this step, we analyze the training dynamics after w k ( T hit ) ∈ M + + ∩ M 0 -.

First, we will prove w k ( t ) passes immediately from one side of the surface M + + ∩M 0 -to the other, i.e. w k ( t ) enters into M + + ∩ M + -at time T hit . Equivalently, we only need to prove b k ( t ) passes immediately from one side of the surface P + + ∩ P 0 -to the other, i.e. b k ( t ) enters into P + + ∩ P + -at time T hit .

For any ˜ b ∈ P + + ∩ P 0 -and 0 < δ 0 ≪ 1 , we know that P + + ∩ P 0 -separates its neighborhood B ( ˜ b , δ 0 ) into two domains G -= { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x -⟩ < 0 } and G + = { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x -⟩ > 0 } . Following Definition H.1, we calculate the limited vector field on ˜ b from G -and G + .

(i) The limited vector field F -on ˜ b (from G -):

(ii) The limited vector field F + on ˜ b (from G + ):

(iii) Then we calculate the projections of F -and F + onto x -(the normal to the surface P + + ∩ P 0 -):

From T I < T init and Lemma C.2, we know | e -y i f i ( t ) -1 | ≤ 0 . 11 , so pe -f + ( t ) cos ∆ -e f -( t ) ≥ 0 . 89 p cos ∆ -1 . 11 > 0 , which means F + N > 0 . And it is clear that F -N > 0 . Hence, the dynamics corresponds to Case (II) in Definition H.1 ( F -N > 0 and F + N > 0 ).

(iv) Hence, b k ( t ) passes immediately from one side of the surface P + + ∩ P 0 -to the other, i.e. b k ( t ) enters into P + + ∩ P + -at time T hit .

Second, proceeding as in the proof of Step I ∼ III of the Proof of Theorem C.3, we have the results:

Lemma C.5 (Positive, M -+ ∩ M + -) .

For positive neuron k k [ m/ 2] : w k (0) -+ + -, after ( κ 1 p ∆ κ 2 ) time, it goes dead:

∈ { ∈ ∈ M ∩ M } O

Proof of Lemma C.9.

We do the following analysis for any k ∈ { k ∈ [ m/ 2] : w k (0) ∈ M + + ∩ M --} , i.e. s k = 1 , ⟨ w k (0) , x + ⟩ > 0 , and ⟨ w k (0) , x -⟩ ≤ 0 .

First, we define the hitting time

and we aim to estimate T hit and prove w k ( T hit ) ∈ M -+ ∩ M --.

From the definition of T hit and (4), the dynamics of the neuron is:

Then we have

It is clear d ⟨ b k ( t ) , x + ⟩ d t < 0 , so ⟨ b k ( t ) , x + ⟩ < ⟨ b k (0) , x + ⟩ ≤ 0 for any t ≤ T hit . If we denote

So we only need to estimate T hit , -. Due to T hit ≤ T I ≤ T init and Lemma C.2, for any t ≤ T hit , we have ∣ ∣ e -y i f i ( t ) -1 ∣ ∣ ≤ 0 . 11 . Then for any t ≤ T hit , we have:

then it holds:

Recalling ⟨ w k (0) , x + ⟩ ≤ 0 and ⟨ w k (0) , x -⟩ > 0 , with the help of Lemma I.2, we have ⟨ w k (0) , x -⟩ ≤ sin ∆ . Combining the two estimate, we have:

Hence,

Moreover, the analysis gives us w k ( T hit ) ∈ M -+ ∩ M --. By Lemma B.2, we obtain:

Lemma C.6 (Positive, M -+ ∩ M --) .

For positive neuron k ∈ { k ∈ [ m/ 2] : w k (0) ∈ M -+ ∩ M --} , it keeps dead forever.

Proof of Lemma C.6. Due to Lemma B.2, this lemma is trivial.

C.2 The Dynamics of Negative Neurons

According to the initial direction, all negative neurons (d k = -1) can be divided into the following four classes.

In the following four lemmas, we will prove the dynamics of these four classes of negative neurons. In summary, in Phase I ( t < T I ), some of the negative neurons move to the manifold M 0 + ∩M + -in a shorter time and then remain on this manifold, and their norms grow slowly, while other negative neurons go dead.

For negative neuron k ∈ { k ∈ [ m ] -[ m/ 2] : w k (0) ∈ M + + ∩ M + -} , in Phase I ( t ≤ T I ) , it's dynamics must belong to one of the following two cases:

Moreover , if ⟨ w k (0) , x -⟩ > (1+ O ( κ 1 κ 2 )) p cos ∆ -(1 -O ( κ 1 κ 2 )) (1 -O ( κ 1 κ 2 )) p -(1+ O ( κ 1 κ 2 )) cos ∆ ⟨ w k (0) , x + ⟩ , it must belongs to Case (i) ; if ⟨ w k (0) , x + ⟩ > (1+ O ( κ 1 κ 2 )) p -(1 -O ( κ 1 κ 2 )) cos ∆ (1 -O ( κ 1 κ 2 )) p cos ∆ -(1+ O ( κ 1 κ 2 )) ⟨ w k (0) , x -⟩ , it must belongs to Case (ii) .

Proof of Lemma C.7.

We do the following analysis for any k ∈ { k ∈ [ m ] -[ m/ 2] : w k (0) ∈ M + + ∩ M + -} , i.e. s k = -1 , ⟨ w k (0) , x + ⟩ > 0 , and ⟨ w k (0) , x -⟩ > 0 .

Step I. Neuron must arrives in the border ∂ ( M + + ∩ M + -) in O ( κ 1 κ 2 ) time.

First, we define the hitting time

and we aim to prove T hit exists and estimate T hit .

Recalling the decoupling ∂ ( M + + ∩ M + -) = ( M 0 + ∩ M + -) ∪ ( M 0 -∩ M + + ) ∪ ( M 0 + ∩ M 0 -) , we only need to focus on the dynamics of ⟨ b k ( t ) , x + ⟩ and ⟨ b k ( t ) , x -⟩ .

From the definition of T hit and (4), the dynamics of the neuron is:

Then we have

Due to T hit ≤ T I ≤ T init and Lemma C.2, for any t ≤ T hit , we have ∣ ∣ e -y i f i ( t ) -1 ∣ ∣ ≤ 1 . 1 √ κ 1 κ 2 . Then for any t ≤ T hit , we have:

Now we consider the time

If we assume T test < T hit , then we have the estimate:

which is contradict to the definition of T hit . Hence, we have:

which means neurons must arrive in the border ∂ ( M + + ∩ M + -) in O ( κ 1 κ 2 ) time.

Because ∂ ( M + + ∩M + -) = ( M 0 + ∩M + -) ∪ ( M 0 -∩M + + ) ∪ ( M 0 + ∩M 0 -) , the neuron must arrives in M 0 + ∩M + -or M 0 -∩M + + or M 0 + ∩M 0 -. If the neuron arrives in M 0 + ∩M 0 -, it goes dead forever (Lemma B.2). We will analyze the training dynamics after arriving in M 0 + ∩ M + -or M 0 -∩ M + + in the following Step II and Step III.

Step II. Dynamics after arriving in the manifold M 0 + ∩ M + -.

In this step, we will analyze the training dynamics after w k ( T hit ) ∈ M 0 + ∩M + -, i.e. after b k ( T hit ) ∈ P 0 + ∩ P + -.

For any ˜ b ∈ P 0 + ∩ P + -and 0 < δ 0 ≪ 1 , we know that P 0 + ∩ P + -separates its neighborhood B ( ˜ b , δ 0 ) into two domains G -= { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x + ⟩ < 0 } and G + = { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x + ⟩ > 0 } . Following Definition H.1, we calculate the limited vector field on ˜ b from G -and G + .

We first analysis the vector field around the manifold P 0 + ∩ P + -for T hit ≤ t ≤ T I .

(i) The limited vector field F -on ˜ b (from G -):

(ii) The limited vector field F + on ˜ b (from G + ):

(iii) Then we calculate the projections of F -and F + onto x + (the normal to the surface P 0 + ∩ P + -):

From T I < T init and Lemma C.2, we know | e -y i f i ( t ) -1 | ≤ 0 . 11 , so pe -f + ( t ) cos ∆ -e f -( t ) ≥ 0 . 89 p cos ∆ -1 . 11 > 0 , which means F + N < 0 . And it is clear that F -N > 0 . Hence, the dynamics corresponds to Case (I) in Definition H.1 ( F -N > 0 and F + N < 0 ).

(iv) Hence, b k ( t ) can not leave P 0 + ∩ P + -for T hit ≤ t ≤ T I .

(v) Moreover, the dynamics of b k on P 0 + ∩ P + -satisfies:

By Lemma C.1, we know that the dynamics of w k ( t ) on M 0 + ∩ M + -and the dynamics of ρ k ( t ) are:

(vi) In this step, we aim to estimate ρ k ( T I ) and 〈 w k ( t ) , x ⊥ + 〉 .

From w k ( T hit ) ∈ M 0 + ∩ M + -, it holds that ⟨ w k ( T hit ) , x + ⟩ = 0 and ⟨ w k ( T hit ) , x -⟩ > 0 . Using lemma I.2, we have 0 < ⟨ w k ( T hit ) , x -⟩ ≤ sin ∆ .

which is

Recalling Lemma C.2 and the estimate of T hit in Step I , we have:

and we can estimate the dynamics for T hit ≤ t ≤ T I by (v)(vi):

Then we obtain the estimate of ρ k ( t ) for any T hit < t ≤ T I :

Specifically, we have:

For any w ∈ M 0 + ∩ M + -, we have ⟨ w , x + ⟩ = 0 , so

So we only need to focus on the dynamics of ⟨ w k ( t ) , x -⟩ to derive the dynamics of 〈 w k ( t ) , x ⊥ + 〉 . By (15) and the estimate of ρ k ( t ) , for any T hit ≤ t ≤ T I we have:

And we have 0 < w k ( 1 κ 2 ) , x -< sin ∆ .

Now we consider the following auxiliary ODE:

and let U ( t ) is the solution of (13). We know that ⟨ w k ( t ) , x -⟩ is an upper solution of ODE (17). From the Comparison Principle of ODEs, we know this means:

In order to estimate ⟨ w k ( t ) , x -⟩ , we only need to study the solution of ODE (17). It is easy to verify that the solution of (17) satisfies

Then we have:

which means

Hence, we have the estimate of 〈 w k ( t ) , x ⊥ + 〉 :

Step III. Dynamics after arriving in the manifold M + + ∩ M 0 -.

In this step, we analyze the training dynamics after w k ( T hit ) ∈ M + + ∩M 0 -, i.e. b k ( T hit ) ∈ P + + ∩P 0 -.

For any ˜ b ∈ P + + ∩ P 0 -and 0 < δ 0 ≪ 1 , we know that P + + ∩ P 0 -separates its neighborhood B ( ˜ b , δ 0 ) into two domains G -= { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x -⟩ < 0 } and G + = { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x -⟩ > 0 } . Following Definition H.1, we calculate the limited vector field on ˜ b from G -and G + .

For any ˜ b ∈ P + + ∩ P 0 -and 0 < δ 0 ≪ 1 , we know that P + + ∩ P 0 -separates its neighborhood B ( ˜ b , δ 0 ) into two domains G -= { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x -⟩ < 0 } and G + = { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x -⟩ > 0 } . Following Definition H.1, we calculate the limited vector field on ˜ b from G -and G + .

(i) The limited vector field F -on ˜ b (from G -):

(ii) The limited vector field F + on ˜ b (from G + ):

(iii) Then we calculate the projections of F -and F + onto x -(the normal to the surface P + + ∩ P 0 -):

From T I < T init and Lemma C.2, we know | e -y i f i ( t ) -1 | ≤ 0 . 11 , so pe -f + ( t ) cos ∆ -e f -( t ) ≥ 0 . 89 p cos ∆ -1 . 11 > 0 , which means F + N > 0 . And it is clear that F -N > 0 . Hence, the dynamics corresponds to Case (II) in Definition H.1 ( F -N > 0 and F + N > 0 ).

(iv) Hence, b k ( t ) passes immediately from one side of the surface P + + ∩ P 0 -to the other, i.e. b k ( t ) enters into P + + ∩ P + -at time T hit .

Then the dynamics of b k in P + + ∩ P --satisfies:

(v) We define the following time, and our aim is to estimate T test , 1 :

It is clear T test , 1 ≤ T test , 2 . Moreover, due to d ⟨ b k ( t ) , x -⟩ d t = -κ 2 √ m p 1+ p e -f + ( t ) cos ∆ < 0 and ⟨ b k ( T hit ) , x -⟩ = 0 , we know ⟨ b k ( t ) , x -⟩ < 0 holds for any t ≤ T test , 1 . Hence, we have

And we only need to estimate T test , 2 . For any T hit < t ≤ T test , 1 = T test , 2 , we have

Recalling Lemma C.2 and the estimate of T hit in Step I , we have:

Then for any T hit < t ≤ T test , 2 , we have:

So we have the estimate

Step IV. Which subspace does the neuron select?

Recalling w k ( T test , 1 ) ∈ M -+ ∩ M --and Lemma B.2, the neuron b k ( t ) keeps dead for any t ≥ T I .

From Step II , we know that the neuron w k ( t ) must arrives in M 0 + ∩M + -or M 0 -∩M + + or M 0 + ∩M 0 -. In this step, we will analyze which subspace does the neuron select.

We only need to compare the following two times:

From the definition of T hit , we know T hit , + = T hit or T hit , -= T hit .

Recalling the proof in Step II , we compare the following two dynamics for t < T hit :

With the help of Lemma C.2 and the estimate of T hit , for any t ≤ T hit ,

Hence, we have the estimate of the dynamics:

(i) If the initialization satisfies ⟨ b k (0) , x -⟩ > (1+4 . 73 κ 1 κ 2 ) p cos ∆ -(1 -4 . 73 κ 1 κ 2 ) (1 -4 . 73 κ 1 κ 2 ) p -(1+4 . 73 κ 1 κ 2 ) cos ∆ ⟨ b k (0) , x + ⟩ , we will prove that the neuron selects M 0 + ∩ M + -at T hit .

For any t < T hit , we have the estimate:

Comparing these two inequalities, we have:

which means

So the neuron w k ( T hit ) ∈ M 0 + ∩ M + -.

(ii) If the initialization satisfies ⟨ b k (0) , x + ⟩ > (1+4 . 73 κ 1 κ 2 ) p -(1 -4 . 73 κ 1 κ 2 ) cos ∆ (1 -4 . 73 κ 1 κ 2 ) p cos ∆ -(1+4 . 73 κ 1 κ 2 ) ⟨ b k (0) , x -⟩ , we will prove that the neuron selects M + + ∩ M --at T hit .

For any t < T hit , we have the estimate:

Comparing these two inequalities, we have:

which means

So the neuron w k ( T hit ) ∈ M + + ∩ M 0 -.

Lemma C.8 (Negative, M -+ ∩ M + -) .

For negative neuron k ∈ { k ∈ [ m ] -[ m/ 2] : w k (0) ∈ M -+ ∩ M + -} , in Phase I ( t ≤ T I ) , we have:

We do the following analysis for any k ∈ { k ∈ [ m ] -[ m/ 2] : w k (0) ∈ M -+ ∩ M + -} , i.e. s k = -1 , ⟨ w k (0) , x + ⟩ ≤ 0 , and ⟨ w k (0) , x -⟩ > 0 .

The case ⟨ w k (0) , x + ⟩ = 0 is trivial. Then we only need to consider the case ⟨ w k (0) , x + ⟩ < 0 . First, we define the hitting time

and we aim to estimate T hit and prove w k ( T hit ) ∈ M 0 + ∩ M + -.

We focus on the dynamics of ⟨ b k ( t ) , x + ⟩ and ⟨ b k ( t ) , x -⟩ .

From the definition of T hit and (4), the dynamics of the neuron is:

Then we have

It is clear d ⟨ b k ( t ) , x -⟩ d t > 0 , so ⟨ b k ( t ) , x -⟩ > ⟨ b k (0) , x -⟩ > 0 for any t ≤ T hit . If we denote

So we only need to estimate T hit , + . Due to T hit ≤ T I ≤ T init and Lemma C.2, for any t ≤ T hit , we have ∣ ∣ e -y i f i ( t ) -1 ∣ ∣ ≤ 0 . 11 . Then for any t ≤ T hit , we have:

Recalling ⟨ w k (0) , x + ⟩ < 0 and ⟨ w k (0) , x -⟩ > 0 , with the help of Lemma I.2, we have ⟨ w k (0) , x + ⟩ > -sin ∆ and ⟨ w k (0) , x -⟩ < sin ∆ . Combining the two estimate, we have:

Hence,

Moreover, we can estimate of ρ k ( T hit ) .

Since ⟨ w k ( t ) , x + ⟩ < 0 and ⟨ w k ( t ) , x -⟩ > 0 hold for any t ≤ T hit , with the help of Lemma I.2, we have ⟨ w k ( t ) , x -⟩ < sin ∆ . Combining (7), for any t ≤ T hit , we have

then it holds:

Step II. Dynamics after arriving in the manifold M 0 + ∩ M + -.

Proceeding as in the proof of Step II in the Proof of Theorem C.7, we have:

w k ( t ) can not leave M 0 + ∩ M + -for T hit ≤ t ≤ T I . Moreover, the dynamics of w k on M 0 + ∩ M + -satisfies:

Recalling the estimate of T hit in Step I , we have

As the proof of Step II in the Proof of Theorem C.7, for any T hit < t ≤ T I , we have

Combining the estimate in Step I , for any 0 < t ≤ T I , we have:

Specifically, we have:

Similar to the proof of Step II in the Proof of Theorem C.7, we have the estimate of the dynamics of ⟨ w k ( t ) , x -⟩ :

In the same way, we can derive

Hence, we have the estimate of 〈 w k ( t ) , x ⊥ + 〉 :

Lemma C.9 (Negative, M + + ∩ M --) . For negative neuron k ∈ { k ∈ [ m ] -[ m/ 2] : w k (0) ∈ M + + ∩ M --} , it keeps dead:

We do the following analysis for any k ∈ { k ∈ [ m ] -[ m/ 2] : w k (0) ∈ M + + ∩ M --} , i.e. s k = -1 , ⟨ w k (0) , x + ⟩ > 0 , and ⟨ w k (0) , x -⟩ ≤ 0 .

First, we define the hitting time

and we aim to estimate T hit and prove w k ( T hit ) ∈ M -+ ∩ M --.

From the definition of T hit and (4), the dynamics of the neuron is:

Then we have

It is clear d ⟨ b k ( t ) , x -⟩ d t < 0 , so ⟨ b k ( t ) , x -⟩ < ⟨ b k (0) , x -⟩ ≤ 0 for any t ≤ T hit . If we denote

So we only need to estimate T hit , + . Due to T hit ≤ T I ≤ T init and Lemma C.2, for any t ≤ T hit , we have ∣ ∣ e -y i f i ( t ) -1 ∣ ∣ ≤ 0 . 11 . Then for any t ≤ T hit , we have:

Recalling ⟨ w k (0) , x + ⟩ > 0 and ⟨ w k (0) , x -⟩ ≤ 0 , with the help of Lemma I.2, we have ⟨ w k (0) , x + ⟩ ≤ sin ∆ . Combining the two estimate, we have:

Hence,

Moreover, the analysis gives us w k ( T hit ) ∈ M -+ ∩ M --. By Lemma B.2, we obtain:

Lemma C.10 (Negative, M -+ ∩ M --) . For negative neuron k ∈ { k ∈ [ m ] -[ m/ 2] : w k (0) ∈ M -+ ∩ M --} , it keeps dead: w k ( t ) ∈ M -+ ∩ M --for any t ≥ 0 .

Proof of Lemma C.10. Due to Lemma B.2, this lemma is trivial.

then it holds:

C.3 Initialization Estimation and Proof of Theorem 4.1

To get the number of neurons in the eight classes in the subsection above, we also need to estimate the initial positions of these neurons under the random initialization.

Lemma C.11 (Initialization Estimation) .

If m = Ω ( log(1 /δ ) ) , then with probability at least 1 -δ , we have:

where A = (1+4 . 73 κ 1 κ 2 ) p cos ∆ -(1 -4 . 73 κ 1 κ 2 ) (1 -4 . 73 κ 1 κ 2 ) p -(1+4 . 73 κ 1 κ 2 ) cos ∆ and B = (1+4 . 73 κ 1 κ 2 ) p -(1 -4 . 73 κ 1 κ 2 ) cos ∆ (1 -4 . 73 κ 1 κ 2 ) p cos ∆ -(1+4 . 73 κ 1 κ 2 ) (mentioned in Lemma C.7).

Proof of Lemma C.11.

(i) By Hoeffding's Inequality (Lemma I.1), for any ϵ > 0 we have:

(ii) From w k (0) ∼ U ( S d -1 ) , without loss of generality, we can let x -= e 1 and x + = e 1 cos ∆ + e 2 sin ∆ .

So we have:

From (6), we have A > 0 and

For simplicity, we denote the event

Then we have the estimate:

We first estimate the lower bound for E [ I { A m } ] :

Secondly, by Hoeffding's inequality (Lemma I.1), for any ϵ > 0 , we have

(iii) This proof is similar to (ii). From w k (0) ∼ U ( S d -1 ) , without loss of generality, we can let x -= e 1 and x + = e 1 cos ∆ + e 2 sin ∆ . so we have:

From (6), we have B > 0 and

For simplicity, we denote the event

Then we have the estimate:

We first estimate the lower bound for E [ I { B m } ] :

Secondly, by Hoeffding's inequality (Lemma I.1), for any ϵ > 0 , we have

let ϵ = 0 . 08 and δ = 4exp( -mϵ 2 / 2) . Combining the uniform bounds in (i)(ii)(iii), we obtain this theorem:

If m ≥ 2 log(4 /δ ) 0 . 08 2 , then with probability at least 1 -δ , we have:

So far, Lemma C.3, C.4, C.5, C.6, C.7, C.8, C.9, C.10 characterize the training dynamics of each neuron in Phase I, and Lemma C.11 estimate the initial positions of the neurons. Now we can prove our main theorem in Phase I.

Theorem C.12 (Restatement of Theorem 4.1) .

Under the data Assumption 3.1, let the two-layer network trained by Gradient Flow (2) starting from random initialization. Let the width m = Ω(log(1 /δ )) , the initialization scales satisfy (6) . Then with probability at least 1 -δ , the following results (S1) ∼ (S5) hold at the end of Phase I ( T I = 10 √ κ 1 κ 2 ) :

(S1). For positive neurons k ∈ [ m/ 2] (s k = 1) , let K + be the index set of living neurons, i.e. K + := { k ∈ [ m/ 2] : w k ( T I ) ∈ M + + ∪ M + -} . Then 0 . 21 m ≤ |K + | ≤ 0 . 29 m . Moreover, for any neuron k ∈ K + , it has the following properties (P1)(P2) .

(P2). Its direction is strongly aligned with µ :

(S2). For negative neurons k ∈ [ m ] -[ m/ 2] (s k = -1) , let K -be the index set of living neurons, i.e. K -:= { k ∈ [ m ] -[ m/ 2] : w k ( T I ) ∈ M + + ∪ M + -} . Then 0 . 075 m ≤ |K -| ≤ 0 . 205 m . Moreover, for any neuron k ∈ K -, it has the following properties (N1)(N2)(N3) .

(N2). It lies on a manifold perpendicular to x + : w k ( t ) ∈ M 0 + ∩ M + -.

(S3). For other neurons k / ∈ K + ∪ K -, it will remain dead forever:

(S4). The predictions for x + and x -have the estimate:

and the training accuracy is Acc( T I ) = p 1+ p .

(S5). 0 < 0 . 258 ≤ 0 . 075 0 . 29 ≤ |K -| |K + | ≤ 0 . 205 0 . 21 ≤ 0 . 977 < 1 .

Proof of Theorem C.12.

This theorem is a corollary of Lemma C.3, Lemma C.4, Lemma C.5, Lemma C.6, Lemma C.7, Lemma C.8, Lemma C.9, Lemma C.10, and Lemma C.11. We focus on the end of Phase I: T I = 10 √ κ 1 κ 2 .

Proof of (S1)(S2) . From Lemma C.11, we know that: if m = Ω(log(1 /δ )) , then with probability at least 1 -δ , we have:

where A = (1+4 . 73 κ 1 κ 2 ) p cos ∆ -(1 -4 . 73 κ 1 κ 2 ) (1 -4 . 73 κ 1 κ 2 ) p -(1+4 . 73 κ 1 κ 2 ) cos ∆ and B = (1+4 . 73 κ 1 κ 2 ) p -(1 -4 . 73 κ 1 κ 2 ) cos ∆ (1 -4 . 73 κ 1 κ 2 ) p cos ∆ -(1+4 . 73 κ 1 κ 2 ) .

Recalling the dynamics analysis in Lemma C.3, C.4, C.5, and C.6, we have:

Recalling the dynamics analysis in Lemma C.7, C.8 C.9, and C.10, we have:

Moreover, the estimates in Lemma C.3, C.4, C.5, and C.6 ensure that for any k ∈ K + , the following results hold:

Similarly, the estimates in Lemma C.7, C.8 C.9, and C.10 ensure that for any k ∈ K -, the following results hold:

Proof of (S3) . A direct corollary of Lemma C.3, C.4, C.5, C.6, C.7, C.8, C.9, C.10.

Proof of (S4) . (S4) are direct corollaries of (S1)(S2) .

For f + ( T I ) , we have the following estimate:

Then we have:

In the same way, we can estimate f ( T ) :

Them we have:

Due to f + ( T I ) > 0 and f -( T I ) > 0 , we obtain ACC( T I ) = p 1+ p .

Moreover, from Theorem C.12 (S1)(S2), we have

Remark C.13. The results in the following proofs are all based on the occurrence of the events in Theorem C.12. All of these results use the same settings as Theorem 4.1, except using a stronger condition on the initialization parameters (3) than (6) in Theorem C.12. So they all hold with probability at least 1 -δ .

D Proofs of Optimization Dynamics in Phase II

In this phase, we study the dynamics before the patterns of living neurons change again after Phase I. Specifically, we define

̸

̸

Recalling the results in Theorem 4.1, during Phase II, the activation patterns do not change with sgn + k ( t ) = sgn -k ( t ) = 1 for k ∈ K + and sgn + k ( t ) = 0 , sgn -k ( t ) = 1 for k ∈ K -. Theorem 4.4 demonstrates that at the end of Phase II, except for one of living positive neuron k 0 ∈ K + precisely changes its pattern on x -, all other activation patterns remain unchanged.

Recall that at the end of Phase I, (i) the neuron k / ∈ K + ∪ K -is dead forever; (ii) as for the living neuron k ∈ K + ∪ K -, the activation patterns are:

In this section, we will focus on the Phase when the negative neuron k ∈ K -still stays on the manifold M 0 + ∩ M + -(hence is still dead for x + ) and the positive neuron k ∈ K + is still activated for x -. In general, we aim to estimate the

As stated in the main text, we define as following hitting time:

̸

and we call T I ≤ t ≤ T II 'Phase II'.

First, we define a more relaxed hitting time than T II , only about the change of living positive neurons:

Noticing that the changes in activation partitions are essentially caused by the change of discontinuous vector fields, we first define the following auxiliary hitting time:

We call T I ≤ t ≤ T ∗ II 'Phase II*'.

In the subsequent proof, we first meticulously characterize the optimization dynamics in Phase II* and then prove T II = T + II = T ∗ II . The crucial proof technique is fine-grained prior estimations for 2 d ODEs on f + ( t ) and f -( t ) , leading to the vector field estimation.

To begin with, we establish the following lemma about the optimization dynamics of living neurons.

Lemma D.1 (Dynamics of living neurons in Phase II*) .

In Phase II*, t ∈ [ T I , T ∗ II ] , we have the following dynamics for each neuron k ∈ K -∪ K + .

(S1) For positive neuron k ∈ K + , we have:

(S2). For negative neuron k ∈ K -, we have:

̸

Proof of Lemma D.1.

(S1) Let k ∈ K + . Recalling the definition of T ∗ II , it holds that ⟨ w k ( t ) , x + ⟩ > 0 and ⟨ w k ( t ) , x -⟩ > 0 for any T I ≤ t ≤ T ∗ II , so the dynamics holds.

For any ˜ b ∈ P 0 + ∩ P + -and 0 < δ 0 ≪ 1 , we know that P 0 + ∩ P + -separates its neighborhood B ( ˜ b , δ 0 ) into two domains G -= { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x + ⟩ < 0 } and G + = { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x + ⟩ > 0 } . Following Definition H.1, we calculate the limited vector field on ˜ b from G -and G + .

(S2) Let k ∈ K -. Recalling the definition of T ∗ II , it holds ⟨ F + ( t ) , x + ⟩ > 0 for any T I ≤ t ≤ T ∗ II . Due to w k ( T I ) ∈ M 0 + ∩ M + -, we first analysis the vector field around the manifold P 0 + ∩ P + -for T I ≤ t ≤ T ∗ II .

(i) The limited vector field F -on ˜ b (from G -):

(ii) The limited vector field F + on ˜ b (from G + ):

(iii) Then we calculate the projections of F -and F + onto x + (the normal to the surface P 0 + ∩ P + -):

We further define the hitting time to check whether w k ( t ) ∈ M 0 + ∩ M + -for T I ≤ t ≤ T ∗ II .

From the definition of T ∗ II , we know ⟨ F + ( t ) , x + ⟩ > 0 for any T I ≤ t ≤ T ∗ II , so F + N < 0 . And it is clear that F -N > 0 . Hence, the dynamics corresponds to Case (I) in Definition H.1 ( F -N > 0 and F + N < 0 ), which means b k ( t ) can not leave P 0 + (i.e., ⟨ b k ( t ) , x + ⟩ = 0 ) for t ∈ [ T I , τ + -] , and the dynamics of b k ( t ) satisfies:

which is

By Lemma C.1, we know that the dynamics of w k ( t ) on M 0 + ∩ M + -and the dynamics of ρ k ( t ) are:

From this dynamics, for any t ∈ [ T I , τ + -] , we have

which means that T ∗ II = τ + -.

Noticing that Lemma D.1 determines the activation patterns for living neurons in Phase II*, the next lemma gives the first-order dynamics of f + ( t ) and f -( t ) .

Lemma D.2 (First-order dynamics of predictions in Phase II*) . In Phase II* ( T I ≤ t ≤ T ∗ II ) , we have the following dynamics for f + ( t ) and f -( t ) :

From the definition of T ∗ II , for any T I ≤ t ≤ T ∗ II , we have

With the help of Lemma D.1, we have the dynamics of predictions:

Due to the specificity of the first-order dynamics, the following lemma gives an second-order autonomous dynamics of predictions, which is is the core dynamics in this phase.

Lemma D.3 (Second-order Autonomous Dynamics of predictions in Phase II*) .

Consider the following two variables:

Then the following autonomous dynamics of U ( t ) and V ( t ) hold in Phase II* ( T I ≤ t ≤ T ∗ II ) :

Recall the first-order dynamics in Lemma D.2:

Then this proof is a straight-forward calculation:

Lemma D.3 enlighten us that we only need to study the dynamics of U ( t ) and V ( t ) to study the dynamics in Phase II, where U ( t ) , V ( t ) satisfies the following autonomous dynamics:

The next lemma provides a fine-grained prior estimate of the dynamics (22).

Lemma D.4 (Fine-grained prior estimate of the dynamics (22)) . For the dynamics (22) , then we have the following results:

(S2). For any t ≥ T I , we have U ( t ) > V ( t ) > 0 .

(S3). If we define the hitting time τ 1 := inf { t ≥ T I : U ( t ) cos ∆ ≤ V ( t ) ( 1 + α sin 2 ∆ ) } , then

(S4). For any t ≥ τ 1 , we have

(S5). For any t ≥ τ 1 , we have U ( t ) -V ( t ) cos ∆ = Θ ( ∆ 2 V ( t ) ) > 0 .

Proof of Lemma D.4.

For simplicity, in this proof, we denote

Step I. Preparation. From Theorem C.12 (S4), we know 0 < f + ( T I ) , f -( T I ) ≤ 3 . 85 κ 2 √ κ 1 κ 2 ≤ 0 . 04 log(1 . 1) , so

Notice that U ( T I ) = κ 2 2 m + m p 1+ p e -f + ( T I ) and V ( T I ) = κ 2 2 m + m 1 1+ p e f -( T I ) . Then we have the estimate:

From Theorem C.12 (S1)(S2), we have

Step II. A rough estimate on U ( t ) and V ( t ) . In this step, we aim to prove:

Then we consider the dynamics of U ( t ) + V ( t ) . From we have

Then we consider the dynamics of U ( t ) -V ( t ) . We define the hitting time

First, from the definition of U ( t ) and V ( t ) , we have U ( t ) > 0 and V ( t ) > 0 .

For any t ∈ [ T I , τ U-V ) , we have:

We consider the auxiliary ODE: d d t P ( t ) = -( U ( T I )+ V ( T I )) P ( t ) , where P ( T I ) = U ( T I ) -V ( T I ) > 0 . From the Comparison Principle of ODEs, we have:

From the definition of τ U-V , we have proved

Step III. Finer estimate in the early Phase t ∈ [ T I , τ 1 ] . Define the following hitting time

For t = T I , it holds that

From Step I, we know τ 1 exists and τ 1 > T I . From (22), we have d U ( t ) d t < 0 and d V ( t ) d t > 0 when t ∈ [ T I , τ 1 ) . Moreover, we have the following dynamics for t ∈ [ T I , τ 1 ) :

If we define Z ( t ) := U ( t ) V ( t ) , then we have d U = Z d V + V d Z .

The dynamics above can be transformed to:

which means

Integrating this equation from T I to t ∈ [ T I , τ 1 ) , we have:

From the continuity of U ( t ) , V ( t ) and Z ( t ) , we have

Combining (25) and (24), let t → τ -1 . Then we have:

Z

(

τ

1

) =

1 +

ϵ

cos ∆

;

where

Therefore,

and where 1 ≤ V ( T I ) κ 2 2 m + m 1 1+ p ≤ 1 + 3 . 87 κ 2 √ κ 1 κ 2 is estimated in Step I.

Step IV. Nearly tight bounds for τ 1

From the definition of τ 1 , we have d V ( t ) d t > 0 for any t ∈ [ T I , τ 1 ) , thus V ( T I ) < V ( t ) < V ( τ 1 ) , ∀ t ∈ ( T I , τ 1 ) . So we have

We first estimate the upper bound for τ 1 . Consider the following dynamics and the hitting time

Then τ 1 < τ u 1 . From the dynamics of ϕ ( t ) , for any t ∈ ( T I , τ u 1 ] , it holds

Therefore,

With the help of Theorem C.12, we have m + m = Θ(1) and m -m = Θ(1) . From Step I, we have C = Θ ( κ 2 2 ∆ 2 ) , U ( T I ) = Θ( κ 2 2 ) and V ( T I ) = Θ ( κ 2 2 p ) . Moreover, it holds

It is easy to verify

Hence, we obtain the upper bound for τ 1 :

In a similar way, we can derive the lower bound for τ 1 . Consider the following dynamics and the hitting time

Then τ 1 > τ l 1 . From the dynamics of ϕ ( t ) , for any t ∈ ( T I , τ l 1 ] , it holds

Therefore,

It is easy to verify thus

Hence, we obtain the lower bound for τ 1 :

Step V. Finer Estimate in the late Phase t > τ 1 .

In this step, we focus on the dynamics when t > τ 1 .

First, we will prove the following nearly tight bound about the ratio of U ( t ) to V ( t ) :

For the right inequality, we define the hitting time

From U ( τ 1 ) V ( τ 1 ) = 1+ ϵ cos ∆ < 1+2 ϵ cos ∆ , we know τ r U / V exists and τ r U / V > τ 1 .

For any t ∈ ( τ 1 , τ r U / V ) , consider

For simplicity, we denote C 1 := ( (1 + 2 ϵ )(1 + 1 cos ∆ ) -cos ∆ ) ( U ( T I ) + V ( T I )) > 0 . We consider the auxiliary ODE: d d t P ( t ) = -C 1 P ( t ) , where P ( τ 1 ) = U ( τ 1 ) -1+2 ϵ cos ∆ V ( τ 1 ) < 0 . From the Comparison Principle of ODEs, we have:

From the definition of τ r U / V , we have proved

In the same way, it can be proved that

Moreover, we also need to derive a tight bound for U ( t ) and V ( t ) when t > τ 1 , respectively. For any t > τ 1 , we have

We consider the auxiliary ODE: d d t P ( t ) = -( (1+ ϵ ) -(1+ ϵ 2 ) cos ∆ ) P 2 ( t ) , where P ( τ 1 ) = V ( τ 1 ) . From the Comparison Principle of ODEs, we have the lower bound for V ( t ) :

In the same way, for any t > τ 1 , we have

We consider the auxiliary ODE: d d t P ( t ) = -( 1 -cos ∆ 1+ ϵ 2 ) P 2 ( t ) , where P ( τ 1 ) = U ( τ 1 ) . From the Comparison Principle of ODEs, we have the upper bound for U ( t ) :

The upper bound for V ( t ) and the lower bound for U ( t ) can be estimated by:

Hence, we obtain the tight bound for U ( t ) and V ( t ) :

It means

Step VI. The tight bound for U ( t ) -V ( t ) cos ∆ .

From 1 + ϵ 2 < U ( t ) V ( t ) < 1+2 ϵ cos ∆ proved in Step V, the two-side bound is straight-forward: for any t ≥ τ 1 ,

Then we obtain

Step VII. The tight bound of U ( t ) cos ∆ -V ( t ) .

If we follow the proof in Step VI, 1 + ϵ 2 < U ( t ) V ( t ) < 1+2 ϵ cos ∆ can only gives us a loose two-side bound for U ( t ) cos ∆ -V ( t ) :

Hence, we need more fine-grained analysis to derive its sharper bounds.

We first focus on its sharper upper bound. From the dynamics (22), for any t ≥ T I , we have

We define the hitting time

From U ( τ 1 ) V ( τ 1 ) = 1+ ϵ cos ∆ , we know τ + U / V exists and τ + U / V > τ 1 .

Then for any t ∈ ( τ 1 , τ + U / V ) , we have U ( t ) cos ∆ -V ( t ) > 0 , so

For simplicity, we denote A = 1 V ( τ 1 ) , B = (1 + ϵ ) -(1 + ϵ 2 ) cos ∆ , C 1 = sin 2 ∆ cos ∆ + 2 + ϵ 2 , C 2 = ( 1 cos ∆ -α ) sin 2 ∆ . And we consider the auxiliary ODE:

Its solution is

From the Comparison Principle of ODEs, for any t ∈ ( τ 1 , τ + U / V ) , we have

Let T E -τ 1 = A B ( ( 1 + ( C 1 -B ) ϵ C 2 ) 1 C 1 B -1 -1 ) , it is easy to verify E ( T E ) = 0 . Moreover,

Therefore,

Then we define the next hitting time

From U ( τ + U / V ) cos ∆ -V ( τ + U / V ) = 0 and d d t ( U ( t ) cos ∆ -V ( t ) )∣ ∣ ∣ t = τ + U / V < 0 , we know τ -U / V exists and τ -U / V > τ + U / V .

For any t ∈ ( τ + U / V , τ -U / V ) , we have U ( t ) cos ∆ -V ( t ) < 0 , so

For simplicity, we denote C 3 = 3 ( 1 + 1 cos ∆ +1 -cos ∆ 1+ ϵ 2 ) . And we consider the auxiliary ODE:

Its solution is

Let τ ′ 1 = τ 1 +Θ ( p 1 1+cos ∆ log(1 / ∆) κ 2 2 ) ≥ 2 τ 1 . Then for any t ≥ τ ′ 1 , it holds

thus,

If we let τ 2 = Θ ( p 1 1+cos ∆ log(1 / ∆) κ 2 2 ) ≥ 2 τ ′ 1 , then we have:

From the Comparison Principle of ODEs, for any t ∈ ( τ + U / V , τ -U / V ) , we have

From the definition of τ -U / V , we obtain

Moreover,

Recalling the lower bound at the beginning of Step VII, we obtain the tight bound:

Lemma D.5 (Hitting time relationship) .

where T + II and T ∗ II are defined in (20)(21) , and U ( t ) , V ( t ) satisfy (22) .

Proof of Lemma D.5.

Recall the definitions of T + II and T ∗ II :

Notice

And for any k ,

So we have

Proof of Lemma D.6.

With the help of Theorem C.12 (S1), for any k ∈ K + , we have:

With the help of Lemma I.4, we have the estimate of ⟨ w k ( T I ) , x -⟩ :

where

Then we have:

which means

From the dynamics (22), we have

With the help of Lemma D.4, we know that U ( t ) -V ( t ) cos ∆ > 0 for any t ≥ T I . So ⟨ b k ( T I ) , x + ⟩ + ∫ t T I √ m κ 2 m + ∆ ( U ( s ) -V ( s ) cos ∆ ) d s > ⟨ b k ( T I ) , x + ⟩ > 0 , ∀ k ∈ K + , ∀ t ≥ T I . Hence, we have the transformation of the hitting time:

Lemma D.6 (Time Estimate of Phase II) .

Taking integral, we obtain:

From Lemma D.5, we have:

Recalling the definition of τ 1 in Lemma D.4, we know U ( s ) cos ∆ -V ( s ) > 0 for any t ≤ τ 1 , so T ∗ II > τ 1 .

From the proof of Lemma D.4, we know U ( T I ) = Θ( κ 2 2 ) and V ( T I ) = Θ ( κ 2 2 p ) . Therefore, solving

we obtain

A straight-forward calculation gives us:

Hence, we get

Combining Lemma D.4 (S3), we obtain

Recall the relationship between T II and T + II (19)(20):

Then using Lemma D.1 (S2), we obtain:

plat

Lemma D.7 (Length of Plateau) .

If we define the hitting time T plat := inf { t ∈ [ T I , T II ] : Acc( t ) = 1 } , then we have:

(S1).

T

= Θ

p

(

)

κ

2

.

2

∆

2

(S3).

∀

t

∈

(

T

plat

, T

II

]

,

Proof of Lemma D.7.

It is easy to verify

From Theorem C.12 (S4), we know f + ( T I ) > 0 and f -( T I ) > 0 . From Lemma D.3, we have

From the proof of Lemma D.4, we know d d t U ( t ) < 0 , ∀ t ∈ [ T I , T II ] , so

Recall the definition of τ 1 and τ 2 in Lemma D.4. From the proof of Lemma D.4, we know d d t V ( t ) > 0 , ∀ t ∈ [ T I , τ 1 ) , so

With the help of Lemma D.4 (S4), we know

Acc(

t

)

≡

1

.

Because ( T plat ) = κ 2 2 m + m 1 1+ p = Θ ( κ 2 2 p ) , we have

It is easy to verify T plat = Θ ( p κ 2 2 ∆ 2 ) < T II = Θ ( p 1 1 -α cos ∆ κ 2 2 ∆ 2 ) .

Because τ 2 -τ 1 = Θ ( p 1 1+cos ∆ log(1 / ∆) κ 2 2 ) , for any t ∈ ( τ 1 , τ 2 ] , we have

Thus,

From Lemma D.4 (S6), we know that U ( t ) cos ∆ - V ( t ) < 0 , ∀ t ∈ ( τ 2 , T II ] . Then d V ( t ) d t < 0 , ∀ t ∈ ( τ 2 , T II ] . Thus,

In summary, we have proved (S1)(S2)(S3).

Lemma D.8 (Prediction at end of Phase II) .

(S1) For the predictions, we have:

(S2). For any k ∈ K + , we have:

(S3). For any k ∈ K -, we have ⟨ b k ( T II ) , x + ⟩ = 0 .

Hence, we know

Proof of Lemma D.8.

Proof of (S1). Recall the definitions in Lemma D.3:

From Lemma D.4 (S4) and Lemma D.6, we have

Therefore, we obtain the estimate:

Moreover, Lemma D.4 (S4)(S5)(S6) give us

Hence,

Proof of (S2). Denote K 0 + := { k ∈ K + : ⟨ w k ( T II ) , x -⟩ = 0 } . From the definition of T II and the proof in Phase II, we know that ⟨ w k ( T II ) , x -⟩ > 0 holds for any k ∈ K + -K 0 + . From the proof in Lemma D.5, for any k ∈ K + , it holds

Thus for any k ∈ K + -K 0 + , we have

so

From the proof of Lemma D.6, we know

where ϵ = 4 . 2 √ κ 1 κ 2 + 2 1+0 . 7 ( 1+9 . 9 √ κ 2 κ 1 ) 1 . 15 . This means

Hence, for any k ∈ K + -K 0 + ,

Proof of (S3). Due to the dynamics of the neuron k ∈ K -in Phase II, this conclusion is clear.

As simple corollaries of these lemmas, we can prove two theorems in Phase II.

Proof of Theorem 4.4 and 4.5.

Theorem 4.5 is Lemma D.7. Theorem 4.4 (S1) has been proven in Lemma D.6; Theorem 4.4 (S2) has been proven in Lemma D.8. Additionally, combining (i) T II = T + II in Lemma D.6, (ii) the transformation in Lemma D.5, and (iii) the definition of T II , we obtain Theorem 4.4 (S3).

E Proofs of Optimization Dynamics in Phase III

E.1 Optimization Dynamics during Phase Transition

Building upon Phase II, we will demonstrate that within a short time, all the living positive neurons K + change their activation patterns, corresponding to a 'phase transition'. After the phase transition, all the living positive neurons k ∈ K + undergo deactivation for x -, i.e., sgn -k ( t ) changes from 1 to 0, while other activation patterns remain unchanged.

Specifically, we define the hitting time

and we call t ∈ ( T II , T PT II ] 'Phase Transition' from Phase II to Phase III.

Notice that the dynamics during phase transition is highly nonlinear with |K + | = Θ( m ) changes on activation partitions. Fortunately, we can keep the neurons of K + and K -close enough respectively in Phase I by using sufficiently small initialization κ 1 . Moreover, their differences do not enlarge in Phase II. As a result, the phase transition can be completed quickly without significant changes in the vector field.

In order to analyze the dynamics of neurons and vector fields, we introduce the auxiliary hitting time:

We call T II ≤ t ≤ T PT ∗ II 'Phase Transition*'.

Lemma E.1 (Dynamics of living neurons during Phase Transition*) .

In Phase Transition*, i.e., t ∈ [ T II , T PT ∗ II ] , we have the following dynamics for each neuron k ∈ K -∪ K + .

(S1). For living negative neuron k ∈ K -, we have:

(S2) For living positive neuron k ∈ K + , we define the hitting time:

Then it holds that:

(P1) For any t ∈ [ T II , T PT ∗ II ,k ) , we have

(P2) If T PT ∗ II ,k < T PT ∗ II strictly, then for any t ∈ [ T PT ∗ II ,k , T PT ∗ II ] , we have w k ( t ) ∈ M + + ∩ M 0 -and

(P3) Regardless of the relationship between T PT ∗ II ,k and T PT ∗ II , for any t ∈ [ T II , T PT ∗ II ] , we have

Proof of (S1). Recalling the definition of T PT ∗ II , it holds ⟨ F + ( t ) , x + ⟩ > 0 for any T II ≤ t ≤ T PT ∗ II . So (S1) can be proved in the same way as employed in the proof of Lemma D.1 (S2) and is omitted.

Proof of (S2)(P0) and (S2)(P1). (S2)(P0) is obvious. Moreover, for any k ∈ K + and t ∈ [ T II , T PT ∗ II ,k ) , we have ⟨ w k ( t ) , x + ⟩ > 0 and ⟨ w k ( t ) , x -⟩ > 0 for any T I ≤ t ≤ T ∗ II , so (S2)(P1) can be proved in the same method as shown in the proof of Lemma D.1 (S1), and we have the dynamics:

Additionally, recalling the definition of T PT ∗ II ,k , we know ⟨ F + ( t ) , x + ⟩ > 0 holds for any t ∈ [ T II , T PT ∗ II ,k ) . Combining the dynamics of b k ( t ) , we further have:

Proof of (S2)(P2). Let k + . If T PT ∗ < T PT ∗

∈ K II ,k II , we have the following results:

Recalling the definition of T PT ∗ II ,k and T PT ∗ II , T PT ∗ II ,k < T PT ∗ II implies that 〈 w k ( T PT ∗ II ,k ) , x -〉 = 0 .

Then recalling our proof of (S2)(P1), we obtain 〈 b k ( T PT ∗ II ,k ) , x + 〉 > 0 .

Hence, we obtain w ( T PT ∗ II ,k ) ∈ M + + ∩ M 0 -.

Step II. Dynamics after t = T PT ∗ II ,k .

In this step, we will analyze the training dynamics after w k ( T PT ∗ II ,k ) ∈ M + + ∩ M 0 -, i.e. b k ( T PT ∗ II ,k ) ∈ P + + ∩ P 0 -. We first analysis the vector field around the manifold P + + ∩ P 0 -. For any ˜ b ∈ P + + ∩ P 0 -and 0 < δ 0 ≪ 1 , we know that P 0 + ∩ P + -separates its neighborhood B ( ˜ b , δ 0 ) into two domains G -= { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x -⟩ < 0 } and G + = { b ∈ B ( ˜ b , δ 0 ) : ⟨ b , x -⟩ > 0 } . Following Definition H.1, we calculate the limited vector field on ˜ b from G -and G + .

(i) The limited vector field F -on ˜ b (from G -):

(ii) The limited vector field F + on ˜ b (from G + ):

(iii) Then we calculate the projections of F -and F + onto x -(the normal to the surface P + + ∩ P 0 -):

We further define the hitting time to check whether w k ( t ) ∈ M + + ∩ M 0 -for T PT ∗ II ,k ≤ t ≤ T PT ∗ II .

From the definition of T PT ∗ II , we know that ⟨ F + ( t ) , x -⟩ = p 1+ p e -f + ( t ) cos ∆ -1 1+ p e f -( t ) < 0 for any t ∈ [ T II , T PT ∗ II ] , which means F + N < 0 . And it is clear that F -N > 0 . Hence, the dynamics

corresponds to Case (I) in Definition H.1 ( F -N > 0 and F + N < 0 ), which means that b k ( t ) can not leave P 0 -for any t ∈ [ T PT ∗ II ,k , τ + + ,k ] , and the dynamics of b k for t ∈ [ T PT ∗ II ,k , τ + + ,k ] satisfies:

which is

By Lemma C.1, we know that the dynamics of w k ( t ) on M + + ∩ M 0 -and the dynamics of ρ k ( t ) are:

Moreover, The dynamics above also ensures that:

which means τ + + ,k = T PT ∗ II . Hence, we have proved (S2)(P2).

Proof of (S2)(P3). Our proof for (S2)(P1) and (S2)(P2) imply this result directly.

For any t ∈ [ T II , T PT ∗ II ] , we have

where

Proof of Lemma E.2.

Step I. Preparation. With the help of Lemma E.1(S1) and (S2)(P3), we know that

(i) For k ∈ K -, we have

(ii) For k ∈ K + , we have

So f + ( t ) and f -( t ) have the following representation for any t ∈ [ T II , T PT ∗ II ] :

Step II. Evolution of f -( t ) .

To begin with, we need to do a rough estimate of ∑ k ∈K + κ 2 √ m b ⊤ k ( t ) x -. Let k ∈ K + . For any t ∈ [ T PT ∗ II ,k , T PT ∗ II ] , we have ⟨ b k ( t ) , x -⟩ = 0 . And for any t ∈ [ T II , T PT ∗ II ,k ) , we have:

Therefore, for any t ∈ [ T II , T PT ∗ II ] , we have 0 ≤ ⟨ b k ( t ) , x -⟩ ≤ ⟨ b k ( T II ) , x -⟩ , so

According to Lemma E.1, it follows that for any k ∈ K -, its dynamics is d b k ( t ) d t = κ 2 e f -( t ) √ m (1+ p ) ( x --x + cos ∆ ) , thus

Therefore, we have two-side bounds of f -( t ) :

For simplicity, we denote C 0 := m -κ 2 2 sin 2 ∆ m (1+ p ) , C --:= ∑ k ∈K -κ 2 √ m b ⊤ k ( T II ) x -and C + -:= ∑ k ∈K + κ 2 √ m b ⊤ k ( T II ) x -. Then we have:

Let Ψ( t ) := ∫ t T II e f -( s ) d s , then dΨ( t ) d t = e f -( t ) . So Ψ( T II ) = 0 and

For the right hand, for any ϵ ∈ (0 , 1) , we consider the auxiliary ODE:

The solution of this ODE is P ( t ) = 1 C 0 log ( 1 + C 0 e -C --+(1+ ϵ ) C + -( t -T II ) ) . From the Comparison Principle of ODEs, we have the upper bound for Ψ( t ) :

Taking ϵ → 0 , we obtain

In the similar way, we can derive the lower bound for Ψ( t ) :

Consequently, we infer that

Noticing f -( T II ) = C + --C --, we obtain

Noticing f -( T II ) = C + --C --, this inequality means

where C 0 = m -κ 2 2 sin 2 ∆ m (1+ p ) = Θ ( κ 2 2 ∆ 2 p ) . Moreover, according to Lemma D.8 (S1)(S2), we have

Step III. Evolution of f + ( t ) .

Let k ∈ K + . According to Lemma E.1 (S2)(P2) and (S2)(P3), it follows that for any k ∈ K + , its dynamics during t ∈ [ T II , T PT ∗ II ] is

Notice that

On the one hand, for any t ∈ [ T II , T PT ∗ II ] , we have the lower bound:

On the other hand, for any t ∈ [ T II , T PT ∗ II ] , we can derive an upper bound:

Hence, we obtain where

Lemma E.3 (Nearly fixed vector filed in Phase III*) .

There exist absolute constants Q 1 , Q 2 > 0 , such that: For any time T fix ∈ [ T II , + ∞ ) , if we choose κ 1 , κ 2 s.t.

then for any t ∈ [ T II , T fix ∧ T PT ∗ II ] , we have

Proof of Lemma E.3.

For simplicity, we denote δ T := T fix ∧ T PT ∗ II -T II From Lemma E.2, for any t ∈ [ T II , T fix ∧ T PT ∗ II ] , we have

Recalling Lemma D.8 (S1), there exists absolute constants Q 1 , Q 2 > 0 such that

To ensure |⟨ F + ( t ) , x + ⟩ - ⟨ F + ( T II ) , x + ⟩| ≤ 1 2 Q 1 ∆ 2 p -1 1 -α cos ∆ , we need only select parameters such that

From Lemma E.2 and Lemma D.8, we have:

Therefore, if we take then we have

If we can take

1

2

then κ 2 2 ∆ 2 p -1 1 -α cos ∆ δ T = O (1) and

Hence,

Step II. Bounding the term ⟨ + -⟩

The proof can be completed by the method analogous to that used in Step I, and we omit it. The result is

Lemma E.4 (The end of Phase Transition) .

If we choose

(S1) (Time).

(S2) (Prediction).

(S3) (Activation patterns).

κ

, κ

s.t

κ

=

O

(1)

and

√

κ

κ

1

2

2

=

O

(

∆

4

)

(3)

, then it holds that

Proof of Lemma E.4. Step I. Time Estimate. Let k ∈ K + .

Recalling the definition of T PT ∗ II ,k in Lemma E.1, Lemma E.1 (S2)(P0) also gives us

From Lemma D.8 (S2), we know that there exists an absolute constant Q 3 > 0 , s.t. 0 ≤ ⟨ b k ( T II ) , x -⟩ ≤ Q 3 √ κ 1 κ 2 √ m . And we let Q 2 > 0 be the absolute constant Q 2 in Lemma E.3.

First, we choose the time then we choose κ 1 , κ 2 s.t.

It can ensure

Then according to Lemma E.3, it follows that

Now we consider the dynamics for t ∈ [ T II , T PT ∗ II ,k ∧ T fix ) .

We claim T PT ∗ II ,k -T II ≤ 2 Q 3 Q 2 ∆ 2 √ κ 1 κ 2 p 1 1 -α cos ∆ . If otherwise, then

From the definition of T PT ∗ II ,k , we know T PT ∗ II ,k < T PT ∗ II ,k ∧ T fix , which leads to a contradiction. therefore, we have proved that for any k ∈ K + ,

With the help of Lemma E.1 (S2)(P0), we obtain

Recalling Lemma E.3, ⟨ F + ( t ) , x + ⟩ > 0 and ⟨ F + ( t ) , x -⟩ < 0 hold for any t ∈ [ T II , T PT ∗ II ∧ T fix ] = [ T II , T PT ∗ II ] . From the definitions of T PT II and T PT ∗ II (26)(27), we obtain

In conclusion, we have proved:

Step II. Prediction Estimate. Step I gives us the result:

Recalling the proof of Lemma E.2, we know

where

Then a straightforward calculation gives us:

With the help of Lemma D.8, we obtain the prediction estimate at the end of Phase III:

Moreover,

which means

In the same way, we can obtain

Step III. Activation Patterns.

Recall our proofs in Step I, we know that

Moreover, from the dynamics in Lemma E.1 (S1) and (S2)(P3), we obtain:

Proof of Theorem 4.6.

Theorem 4.6 (S1) has been proven in Lemma E.4 (S1), and Theorem 4.6 (S2) has been proven in Lemma E.4 (S3).

E.2 Optimization Dynamics after Phase Transition

After Phase Transition ( t > T PT II ) , we study the dynamics before the patterns of living neurons change again. Specifically, we define the following hitting time

̸

̸

and we call t ∈ ( T PT II , T III ) 'L-Phase III'.

Moreover, we call t T , T 'Phase III', i.e.. 'Phase Transition' + 'L-Phase III'.

∈ [ II III )

In order to analyze the dynamics of neurons and vector fields, we introduce the auxiliary hitting time:

We call t ∈ ( T PT II , T ∗ III ) 'L-Phase III*'.

Due to the almost simplest activation patterns, this phase is easier to analyze, and we only need to estimate the time and size of the changes in the vector field. Nevertheless, our challenge is to prove that all living negative neurons simultaneously change their activation patterns at T ∗ III , which also implies that T III = T ∗ III .

Lemma E.5 (Dynamics of activate neurons during L-Phase III*) .

In L-Phase III* ( t ∈ [ T PT II , T ∗ III )) , we have the following dynamics for each neuron k ∈ K -∪ K + . k , we have:

(S1). For negative neuron ∈ K -

(S2) For positive neuron k ∈ K + , we have:

Proof of Lemma E.5.

From the definition of T ∗ III , we know that ⟨ F + ( t ) , x + ⟩ > 0 and ⟨ F + ( t ) , x -⟩ < 0 hold for any t ∈ [ T PT II , T ∗ III ) . Moreover, Lemma E.4 ensures that for k ∈ K + , w k ( T PT II ) ∈ M 0 + ∩ M + -; for k ∈ K -, w k ( T PT II ) ∈ M + + ∩ M 0 -. Hence, this lemma can be proved in the same way as shown in the proof of Lemma E.1 (S1) and (S2)(P2). We do not repeat it here.

Lemma E.6 (Time and prediction estimate at the end of L-Phase III*) . (S1) (Time).

(S2) (Prediction).

Proof of Lemma E.6.

Step I. Explicit Solution to f + ( t ) and f -( t ) .

For any t ∈ [ T PT II , T ∗ III ) , we have:

Let us consider the dynamics of f + ( t ) and f -( t ) . With the help of Lemma E.5, these two dynamics are nearly independent:

Their solutions are:

Step II. Time Estimate of T ∗ III .

For simplicity, we denote G + := κ 2 2 m + p sin 2 ∆ m (1+ p ) and G -:= κ 2 2 m -sin 2 ∆ m (1+ p ) .

First, we consider the evolution of the vector field ⟨ F + ( t ) , x -⟩ :

Hence, the hitting time T ∗ III can be converted to the following T ∗∗ III :

Then we consider ⟨ F + ( t ) , x + ⟩ :

From Lemma E.4, we know

These imply the hitting time:

Step III. Prediction estimate.

From the explicit solution in Step I and the time estimate in Step II, it is easy to verify

Recalling the calculation in Step II, we have:

Lemma E.7 (Hitting time relationship) . If we define the following hitting time:

then it holds that T III = T ∗ III = T W III .

Proof of Lemma E.7.

We define the following hitting time:

From the proof in Lemma E.4, we know ⟨ F + ( T ∗ III ) , x -⟩ < 0 . From the continuity of ⟨ F + ( · ) , x -⟩ , we know that there exists τ 1 > 0 , such that ⟨ F + ( t ) , x -⟩ < 0 holds for any t ∈ [ T ∗ III , T ∗ III + τ 1 ) . Then in the same way as the proof of Lemma E.5 (S2), we know that for k ∈ K + , w ( t ) ∈ M + + ∩ M 0 -for any t ∈ [ T ∗ III , T ∗ III + τ 1 ) .

Recalling that for any k ∈ K -, ⟨ b k ( T ∗ III ) , x -⟩ > 0 , from the continuity, we know that there exists τ 2 > 0 such that ⟨ b k ( t ) , x -⟩ > 0 holds for any t ∈ [ T ∗ III , T ∗ III + τ 2 ) .

Hence, we have:

It is obvious that T N III ≥ T F III ∧ T N III = T ∗ III . Now we prove T N III = T ∗ III .

Then we consider the vector field around the manifold M 0 + ∩ M + -for t ∈ [ T ∗ III , T N III ) . In the same way as the proof of Lemma E.1 (S1), we can prove that the two-side projections onto x + (the normal to the surface M + + ∩ M 0 -) satisfies f + N ( t, ˜ w ) , f -N ( t, ˜ w ) > 0 for any t ∈ [ T ∗ III , T ∗ III + τ 1 ) , which satisfies (Case II) in Definition H.1. This implies that w k ( t ) enter the manifold M + + , i.e., ⟨ w k ( t ) , x + ⟩ > 0 for any t ∈ [ T ∗ III , T N III ) , which is contradict to the definition of T N III . Hence, we have proved

If we assume T N III > T ∗ III strictly, then the dynamics about f + ( t ) and f -( t ) in the proof (Step I) of Lemma E.6 still hold for any t ∈ [ T ∗ III , T N III ) . Using the same calculate about ⟨ F + ( t ) , x + ⟩ in the proof (Step II, III) of Lemma E.6, we can obtain: ⟨ F + ( t ) , x + ⟩ < 0 , t ∈ [ T ∗ III , T N III ) .

Noticing that the change of activation patterns of sgn + k ( t ) ( k ∈ K -) is due to the change of the vector field ⟨ F + ( t ) , x + ⟩ , it is easy to verify that T F III = T N III ∧ T F III ∗ . Then we have T N III = T F III = T N III ∧ T F III = T ∗ III .

Lastly, noticing that all living negative neurons ( k ∈ K -) belong to M 0 + ∩ M + -at time T III . As discussed above, for each living negative neuron k ∈ K -, the vector field near b k ( T III ) is the same, with f -N > 0 and f + N = 0 in Definition H.1 (Case II). Hence, each living positive neuron w k leaves from M 0 + and enter M + + instantly at T III , which means T W III = T N III .

Moreover, noticing that T III ≤ T N III and T ∗ III ≤ T III , we obtain T III = T ∗ III = T N III = T F III .

Proof of Theorem 4.7.

F Proofs of Optimization Dynamics in Phase IV

Proof of Theorem 4.8.

From Lemma E.7, we know that all living negative neuron k ∈ K -simultaneously change their patterns on x + at T III : lim t → T -III sgn + k ( t ) = 0 , lim t → T + III sgn + k ( t ) = 1 . Moreover, from our proof of Lemma E.7, we know that other activation patterns remain unchanged at T III .

In this phase, we study the dynamics before activation patterns change again after the phase transition in Theorem 4.8. We define the hitting time:

̸

̸

and we call t ∈ ( T III , T IV ) Phase IV.

In order to analyze the dynamics of neurons and vector fields, we introduce the auxiliary hitting time:

and we call t ∈ ( T III , T ∗ IV ) Phase IV*.

First, we will provide meticulous prior estimations for 2 d ODEs on f + ( t ) and f -( t ) , similar to Phase II, which can imply T ∗ IV = + ∞ . Additionally, we can prove T IV = T ∗ IV . Lastly, with the help of our fine-grained analysis for the 2D dynamics and the results in (Lyu and Li, 2019; Ji and Telgarsky, 2020), we can determine the unique convergent direction from numerous KKT directions.

F.1 Non-asymptotic Analysis of Optimization Dynamics in Phase IV*

Lemma F.1 (Dynamics of activate neurons in Phase IV*) . In Phase IV* ( t ∈ ( T III , T ∗ IV )) , we have the following dynamics for each neuron k ∈ K -∪ K + .

(S1). For negative neuron k ∈ K -, we have:

(S2) For positive neuron k ∈ K + , we have:

Proof of Lemma F.1.

Using the definition of T ∗ IV , this lemma can be proved in the same way as shown in the proof of Lemma D.1, E.1 and E.5.

The next lemma gives the first-order dynamics of f + ( t ) and f -( t ) .

Lemma F.2 (First-order Dynamics of predictions in Phase IV*) . In Phase IV* ( T III ≤ t ≤ T ∗ IV ) , we have the following dynamics for f + ( t ) and f -( t ) :

Proof of Lemma F.2.

From the definition of T IV , for any T III ≤ t ≤ T IV , we have

With the help of Lemma F.1, we have the dynamics of predictions:

Following the proof in Phase II, we focus on the dynamics about predictions. Due to the specificity of the first-order dynamics, the following lemma gives an second-order autonomous dynamics of predictions.

Lemma F.3 (Second-order Autonomous Dynamics of predictions in Phase IV*) . If we consider the following two variables:

then the following autonomous dynamics of U ( t ) and V ( t ) hold in Phase IV* ( T III ≤ t ≤ T ∗ IV ) :

Proof of Lemma F.3.

With the help of the first-order dynamics in Lemma F.2, the proof is straight-forward.

Lemma F.3 enlighten us that we only need to study the dynamics of I ( t ) and J ( t ) to study the dynamics in Phase IV*, where I ( t ) , J ( t ) satisfies the following autonomous dynamics:

The next lemma studies the dynamics (31) for any t ∈ [ T III , + ∞ ) .

Lemma F.4 (Fine-grained analysis of the dynamics (31)) .

For the dynamics (31) , we have the following results:

(S1). Initialization.

(S2). Fine-grained two-side bound for I ( t ) / J ( t ) .

(S3). The limit of I ( t ) / J ( t ) .

(S4). Tight estimate of I ( t ) and J ( t ) .

For simplicity, in this proof, we denote

Step I. Preparation. Recalling Lemma E.6, we have:

Step II. A rough estimate on I ( t ) and J ( t ) . In this step, we aim to prove:

Then we consider the dynamics of I ( t ) + J ( t ) . From we have

Then we consider the dynamics of J ( t ) -I ( t ) . We define the hitting time

From Step I, we know J ( T III ) -I ( T III ) = (1 -cos ∆) J ( T III ) > 0 . From the continuity, τ J-I exists and τ J-I > T III .

First, from the definition of I ( t ) and J ( t ) , we have I ( t ) > 0 and J ( t ) > 0 .

For any t ∈ [ T III , τ J-I ) , we have J ( t ) -I ( t ) > 0 and

We consider the auxiliary ODE: d d t P ( t ) = -( J ( T III ) + I ( T III )) P ( t ) , where P ( T III ) = J ( T III ) -I ( T III ) > 0 . From the Comparison Principle of ODEs, we have:

From the definition of τ J-I , we have proved

Step III. A rough two-side bound for I ( t ) / J ( t ) .

In Step II, we have given a rough upper bound for I ( t ) / J ( t ) : I ( t ) / J ( t ) < 1 , ∀ t ≥ T III . And we want to derive a lower bound for I ( t ) / J ( t ) in this step. We aim to prove:

First, we define the hitting time:

From Step I, we know

From the continuity, τ l I / J exists and τ l I / J > T III .

For any t ∈ [ T III , τ l I / J ) , we have I ( t ) -1+cos∆ 1+cos∆+ ϵ J ( t ) > 0 and

We consider the auxiliary ODE: d d t Q ( t ) = -(1 + ϵ )( I ( T III ) + J ( T III )) Q ( t ) , where Q ( T III ) = I ( T III ) -1+cos∆ 1+cos∆+ ϵ J ( T III ) > 0 . From the Comparison Principle of ODEs, we have:

=

I

(

T

III

)

-

(

1 + cos ∆

)

(

1 + cos ∆ +

ϵ

J

(

T

III

)

exp

From the definition of τ J-I , we have proved

Hence, we obtain the two-side bound for I ( t ) / J ( t ) :

Step IV. I ( t ) cos ∆ -J ( t ) and I ( t ) -J ( t ) cos ∆ are both negative.

The estimate on I ( t ) cos ∆ -J ( t ) is straight-forward:

As for I ( t ) -J ( t ) cos ∆ , we will actually prove a tighter upper bound:

We need to do finer analysis using the specific dynamics (31). First, we define the following hitting time:

Define the following hitting time

From I ( T III ) -J ( T III ) cos ∆ = 0 , d d t ( I ( T III ) -J ( T III ) cos ∆) < 0 and the continuity, we know τ u I / J exists and τ u I / J > T III .

Recalling I ( t ) cos ∆ -J ( t ) < 0 , we have

So we can consider the following dynamics for t ∈ [ T III , τ u I / J ] :

If we define Z ( t ) := I ( t ) J ( t ) , then we have d I = Z d J + J d Z .

The dynamics above can be transformed to:

Recalling the result in Step III, we have (1 + cos ∆) -(1 + cos ∆ + ϵ ) Z ( t ) < 0 holds for any t ≥ T III . So the dynamics is equal to:

Integrating this equation from T III to t ∈ [ T III , τ u I / J ) , we have:

-

(1 +

ϵ

)(

I

(

T

III

) +

J

(

T

III

))(

t

-

T

III

)

)

>

0

,

∀

t

∈

T

[

III

, τ

l

I

/

J

)

.

If we assume τ u I / J < + ∞ , the continuity gives us

Then letting t → τ u I / J -in (32), we have

which means J ( τ u I / J ) = J ( T III ) .

But on the other hand, we have:

which leads to a contradiction. Hence, we have proved

Moreover, we obtain a sharper two-side bound for t / t :

Step V. Tight bound for I ( t ) and J ( t ) .

In this step, we aim to give a tight bound for I ( t ) + J ( t ) . With the help of the two-side bound (33), we have

For the first inequality, we consider the auxiliary ODE: d d t P ( t ) = -∆ 2 6 P 2 ( t ) , where P ( T III ) = I ( T III ) + J ( T III ) > 0 . From the Comparison Principle of ODEs, we have:

In the same way, we can obtain the lower bound:

Recalling Step I, we have 1 I ( T III )+ J ( T III ) = Θ ( p 1 1 -α cos ∆ /κ 2 2 ) . Hence, we obtain the tight bound:

Taking (33) into the equation above, we have:

Step VI. The limit of I ( t ) / J ( t ) .

By Step V and the proof of Step IV, we know lim t →∞ J ( t ) = 0 and d d t J ( t ) < 0 holds for any t > T III .

Then for any ϵ ′ > 0 , there exists T ′ > T III such that

Taking it into (32), we obtain that for any t > T ′ ,

By the definition of the limit, we get

Lemma F.5 (Time and prediction estimate) .

(S1). For any t ∈ ( T III , + ∞ )

(S2). For any t ∈ ( T III , + ∞ ) ,

Moreover, pe -( f + ( T III )+ f -( T III )) = cos ∆ and lim t →∞ pe -( f + ( t )+ f -( t )) = 1+cos∆ 1+cos∆+ m + m -sin 2 ∆ .

(S3). For any t ∈ ( T III , + ∞ ) ,

(S4) (Time).

Proof of Lemma F.5.

Notice the relationships: pe -f + ( t ) = κ 2 2 m -m I ( t ) 1+ p , e f -( t ) = κ 2 2 m -m J ( t ) 1+ p and pe -( f + ( t )+ f -( t )) = I ( t ) / J ( t ) . Then Lemma F.4 implies that T ∗ IV = + ∞ . Recalling the dynamics in Lemma F.1, then lemma (S3)(S4) hold. Then using Lemma F.4 again, we obtain (S1)(S2).

Proof of Theorem 4.9.

Theorem 4.9 (S1), (S2), and (S3) are obtained in Lemma F.5 (S4), (S3), and (S1), respectively.

F.2 Asymptotic Directional Convergence

In this section, we will study the final convergence direction in our setting. It mainly depends on our prior fine-grained analysis of the training dynamics in Phase IV and the following result about the final convergence direction at the end of training.

Lemma F.6. Let f ( · ; θ ) be a homogeneous neural network parameterized by θ . Consider minimizing the exponential loss over a binary classification dataset { ( x i , y i ) } n i =1 ( ∥ x i ∥ 2 ≤ 1 , y i ∈ {± 1 } ) using Gradient Flow. Assume that there exists time t 0 such that L ( θ ( t 0 )) < 1 n . Then,

(I) (Paraphrased from (Lyu and Li, 2019; Ji and Telgarsky, 2020)). θ ( t ) converges in direction to a KKT point (Definition G.3) of the following maximum margin problem:

(II) (Lyu and Li, 2019; Ji and Telgarsky, 2020)). ∥ θ ( t ) ∥ 2 →∞ and L ( θ ( t )) → 0 .

Lemma F.7 (Final Convergence Direction) . The limit lim t → + ∞ θ ( t ) ∥ θ ( t ) ∥ 2 exists, and denoted by θ = ( ¯ b ⊤ , , ¯ b ⊤ ) ⊤ S md -1 , then it satisfies

(III) (Ji and Telgarsky (2020)). -∇L ( θ ( t )) and θ ( t ) converge to the same direction, meaning the angle between θ ( t ) and -∇L ( θ ( t )) converges to 0 .

where C > 0 is a scaling constant such that ∥ ∥ θ ∥ ∥ 2 = 1 . Moreover, f ( x + ; θ ) = -f ( x -; θ ) > 0 .

Proof of Lemma F.7. Let θ = ( b ⊤ 1 , · · · , b ⊤ m ) ⊤ ∈ S md -1 be the limits point of { θ ( t ) ∥ θ ( t ) ∥ 2 : t ≥ t 0 } . From Lemma F.6 (I), we know that there exists a scaling factor α > 0 such that α θ satisfies KKT conditions (Definition G.3) of the maximum-margin problem

For simplicity, we denote θ ∗ := α ¯ θ , where

Moreover, let λ ∗ + , λ ∗ -≥ 0 be the corresponding Lagrange multipliers (with respect to θ ∗ ) in Definition G.3.

Step I. The rough direction of each neuron.

Recalling the training dynamics about the dead neurons in Theorem 4.1 (S3),

Noticing Lemma F.6 (II) or Lemma F.5 (S1), ∥ θ ( t ) ∥ 2 →∞ , so

Then we only need to focus on θ ∗ k for k ∈ K + ∪ K -.

Recalling in Lemma F.5 (S3), for any t > T IV ,

then we have

Moreover,

Step II. Determine the direction of the neurons k ∈ K + .

Since θ ∗ is a possible point, f ( x + ; θ ∗ ) ≥ 1 , which gives us

Hence, there exists k 1 ∈ K + , s.t. 〈 b ∗ k 1 , x + 〉 > 0 strictly.

Then we study the neuron k 2 ∈ K + ( k 2 = k 1 ). Lemma F.1 and Lemma F.5 (S3) give use that

̸

Multiplying the above formula by c/ ∥ θ ( t ) ∥ 2 and taking t go to infinity, we obtain

Due to the arbitrariness of k 2 , we know

̸

Then we can write the KKT condition about the gradient of b ∗ k ( k ∈ K + ) of Problem (34):

It is clear that b ∗ k ∈ span { x + , x -} . Then combining two formulations above, we obtain:

Step III. Determine the direction of the neurons k ∈ K -.

Since θ ∗ is a possible point, f ( x -; θ ∗ ) ≤ -1 , which gives us

Hence, there exists k 1 ∈ K -, s.t. 〈 b ∗ k 1 , x -〉 > 0 strictly.

Then we study the neuron k 2 ∈ K -( k 2 = k 1 ). Lemma F.1 and Lemma F.5 (S3) give use that

̸

Multiplying the above formula by c/ ∥ θ ( t ) ∥ 2 and taking t go to infinity, we obtain

Due to the arbitrariness of k 2 , we know

̸

The next difficulty in this step is to determine whether ⟨ b ∗ k , x + ⟩ can be 0 . To prove this, we will use our fine-grained analysis of training dynamics (Lemma F.5) and Lemma F.6 (III).

Let k ∈ K -. Recalling the dynamics of b k ( t ) in Lemma F.1, we know

Recalling Lemma Lemma F.5 (S2), lim t →∞ pe -( f + ( t )+ f -( t )) = 1+cos∆ 1+cos∆+ m + m -sin 2 ∆ . Then using Lemma F.6 (III), there exists c 1 > 0 , s.t.

Hence, we have proved

Then writing the KKT condition about the gradient of b ∗ k of Problem (34):

Combining the two equations about b ∗ k , we obatin

In summary, we have proved the final convergence direction θ = ( ¯ b ⊤ 1 , · · · , ¯ b ⊤ m ) ⊤ ∈ S md -1 satisfies

where C > 0 is a scaling constant such that ∥ ∥ θ ∥ ∥ 2 = 1 .

Moreover, a straight-forward calculation gives us that

Proof of Theorem 4.10.

Lemma F.7 implies Theorem 4.10 directly.

Proof of Theorem 4.11.

From Lemma F.7, the final convergence direction θ = ( ¯ b ⊤ 1 , · · · , ¯ b ⊤ m ) ⊤ ∈ S md -1 satisfies

where C > 0 is a scaling constant such that ∥ ∥ θ ∥ ∥ 2 = 1 and f + ( θ ) = -f -( θ ) > 0 .

It is easy to verify that There exists a scaling factor C 1 > 0 such that f + ( ˆ θ ) = -f -( ˆ θ ) = 1 , where ˆ θ = C 1 θ . For simplicity, we denote Q := CC 1 , then ˆ θ = ( ˆ b ⊤ 1 , · · · , ˆ b ⊤ m ) ⊤ ∈ R md satisfies

Therefore, ˆ θ is a feasible point of Problem (34). Moreover, from

we have

For any ϵ ≥ 0 , now we consider another solution ˆ θ ( ϵ ) near ˆ θ : ˆ θ ( ϵ ) = ( ˆ b ⊤ 1 ( ϵ ) , · · · , ˆ b ⊤ m ( ϵ )) ⊤ , where

and it holds that ˆ θ (0) = ˆ θ . Moreover, it is easy to verify that ˆ θ ( ϵ ) is also a feasible point of Problem (34).

Then we compare the norm of ˆ θ ( ϵ ) and ˆ θ (0) .

At ϵ = 0 , we can calculate that

Then combining the continuity of d ∥ ˆ θ ( ϵ ) ∥ 2 d ϵ , there exists δ > 0 such that the following inequality holds:

Hence, we have proved that θ is not a local optimal direction of the max-margin problem (34).

G Clarke Subdifferential and KKT Conditions for Non-smooth Optimization

Definition G.1 (Clarke's Subdifferential (Clarke et al., 2008)) . For a locally Lipschitz function L : Ω → R , the Clarke's subdifferential at θ ∈ Ω is the convex set

Remark G.2. Notice that if L is continuously differentiable at θ , then ∂ ◦ L ( θ ) = {∇L ( θ ) } is unique. However, for discontinuous differentiable points of L , the differential inclusion flow d θ d t ∈ ∂ ◦ L ( θ ) defined by Definition G.1 may not be unique. To study a more specific dynamics, we also utilize Definition H.1 to determine GF at some of such points.

Now we review the definition of Karush-Kuhn-Tucker (KKT) conditions for non-smooth optimization problems (Dutta et al., 2013). Consider the following constrained optimization problem (P):

where f, g 1 , · · · , g N : R d → R are locally Lipschitz functions. We say that x ∈ R is a feasible point of (P) if x satisfies g i ( x ) ≤ 0 for all i ∈ [ N ] .

Definition G.3 (KKT Point for Non-smooth Optimization) . We say that a feasible point of (P) is a KKT point if there exists λ 1 , · · · , λ N ≥ 0 such that

H Solution of Discontinuous System

In this section, we add some supplements about the definitions of solutions of discontinuous systems, which can overcomes non-uniqueness of GF trajectories (2) to some extent. Many definitions of solutions of differential equations with discontinuous systems have been proposed. In this paper, we adopt a widely used definition of the solutions in Chapter 2.4 in (Filippov, 2013).

Definition H.1 (Solutions of Discontinuous Systems, Chapter 2.4 in (Filippov, 2013)) . Consider a n -dimensional equation or a system ( x ∈ R n ): d x d t = f ( x ) with a piecewise continuous function f in a domain G . We aim to define the dynamics near some discontinuous regions.

Let the function f be discontinuous on a smooth surface S given by the equation ϕ ( x ) = 0 . Let x ∗ ∈ S and the surface S separate the neighborhood of x ∗ into domains G -and G + . Let the function f ( x ) have the limit values:

Here one should distinguish between two main cases. Let f -N ( x ∗ ) and f + N ( x ∗ ) be projections of the vectors f -( x ∗ ) and f + ( x ∗ ) onto the normal to the surface S at the point x ∗ , where the normal is directed towards the domain G + .

(Case I) . If the vectors f ( x ∗ ) are directed to the surface S on both sides, i.e. f -N ( x ∗ ) > 0 , f + N ( x ∗ ) < 0 , then the solution the solution starting from x ∗ can not leave S for some time. Moreover, its dynamics on S can be defined in the following way:

(Case II) . If f -N ( x ∗ ) ≥ 0 , f + N ( x ∗ ) ≥ 0 , but f -N ( x ∗ ) and f + N ( x ∗ ) are not both 0 , then the solution starting from x ∗ passes from one side of the surface S to the other instantly.

(Case III). If f -N ( x ∗ ) < 0 , f + N ( x ∗ ) > 0 , then the dynamics is defined in the similar way as (Case I).

(Case IV). If f -N ( x ∗ ) ≤ 0 , f + N ( x ∗ ) ≤ 0 , but f -N ( x ∗ ) and f + N ( x ∗ ) are not both 0 , then the dynamics is defined in the similar way as (Case II).

Remark H.2. Notice that Definition H.1 overcomes non-uniqueness of GF trajectories to some extent. It is worth noting that Definition H.1 and Definition G.1 are compatible and specifically, the dynamics defined in Definition H.1(Case I, III) lie in the convex hull defined in Definition G.1.

Remark H.3. In (Lyu et al., 2021), the non-branching starting point Assumption is employed to address the technical challenge of non-uniqueness in GF trajectories. By comparison, in this work, we do not need this assumption. We adopt Definition H.1 to uniquely determine the Gradient Flow trajectories theoretically near some discontinuous differential regions, such as 'Ridge', 'Valley', and 'Refraction edge' discussed in Section I.2 in (Lyu et al., 2021).

I Some Basic Inequalities

Lemma I.1 (Hoeffding's Inequality) . Let X 1 , · · · , X n are independent random variables, and X i ∈ [ a i , b i ] for any i ∈ [ n ] . Define ¯ X = 1 n ∑ n i =1 X i . Then for any ϵ > 0 , we have the following probability inequalities:

Lemma I.2. Consider x 1 , x 2 , y ∈ S d -1 , where ⟨ x 1 , x 2 ⟩ = cos ∆ (∆ ∈ (0 , π/ 2)) . If ⟨ y , x 1 ⟩ ≥ 0 and ⟨ y , x 2 ⟩ ≤ 0 , then we have 0 ≤ ⟨ y , x 1 ⟩ ≤ sin ∆ and -sin ∆ ≤ ⟨ y , x 2 ⟩ ≤ 0 .

Proof of Lemma I.2.

Denote M x := span { x 1 , x 2 } . We can do the orthogonal decomposition of y :

where y M ∈ M x and y ⊥ M ⊥ M x . From y ∈ span { x 1 , x 2 } , there exist α, β ∈ R , s.t. y M = α x 1 + β x 2 .

Due to the orthogonal decomposition, we know ∥ y M ∥ ≤ 1 , which means α 2 + β 2 +2 αβ cos ∆ ≤ 1 . Noticing α 2 + β 2 +2 αβ cos ∆ = ( α + β cos ∆) 2 + α 2 sin 2 ∆ , we know α 2 sin 2 ∆ ≤ 1 .

Due to ⟨ y , x 1 ⟩ ≥ 0 and ⟨ y , x 2 ⟩ ≤ 0 , we have ⟨ y M , x 1 ⟩ ≥ 0 and ⟨ y M , x 2 ⟩ ≤ 0 , which means .

So α ≥ 0 and α sin ∆ ≥ 0 . Recalling α 2 cos 2 ∆ ≤ 1 , we know 0 ≤ α sin ∆ ≤ 1 . Hence, we have α + β cos ∆ ≤ α -α cos 2 ∆ = α sin 2 ∆ ≤ sin ∆ , i.e. ⟨ y , x 1 ⟩ ≤ sin ∆ .

In the same way, we have -sin ∆ ≤ ⟨ y , x 2 ⟩ ≤ 0 .

Lemma I.3. If p ≥ 5 , we have ∥ z ∥ ≥ p -1 p +1 ≥ 2 3 .

Proof of Lemma I.3.

Lemma I.4. Let w ∈ S d -1 . If ⟨ w , µ ⟩ ≥ 1 -ϵ ( ϵ ∈ (0 , 1)) , p ≥ 5 and cos ∆ ≥ 4 / 5 , then we have

Proof of Lemma I.4.

It is easy to verify

Thus,

Then we have the bound:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper conducts a comprehensive theoretical analysis of the training dynamics of two-layer ReLU neural networks on a linearly separable data. The authors isolate four discrete phases of the training procedure and describe specific nonlinear behaviors that occur during each phase. Moreover, they derive explicit formulae for the evolution of the network parameters and prove convergence results for the network's output. The paper's contributions include a complete theoretical characterization of the training process, a better understanding of the role of initialization and regularization, and insights into the generalization properties of ReLU networks. In brief, the research provides a valuable addition to the domain of deep learning theory and sheds light on the training process of ReLU.\n\nSTRENGTHS:\nThe originality of the paper lies in its complete theoretical characterization of the training process of the two-layer ReLU network. While previous work has focused on local analysis or approximate linear models, this paper provides a more comprehensive understanding of the optimization dynamics of ReLU networks. The authors identify four distinct stages of the training process and describe the specific nonlinear behavior that occurs at each stage. They also derive explicit formulations for the evolution of network parameters and demonstrate convergent results for network outputs. Besides, the authors provide rigorous mathematical proofs and derive explicit formulas that shed light on the complex dynamics of ReLU networks.\n\nWEAKNESSES:\nFirst, a weakness of the paper is that it focuses on the specific setting of two-layer ReLU networks trained on linearly separable data. While this setting is useful for providing a full theoretical characterization of the training process, it may not generalize to more complex networks or datasets. \n\nSecond, another weakness of the paper is that it does not provide empirical verification of its theoretical results. While the authors provide rigorous mathematical proofs and derive explicit formulas, it would be useful to validate these results on real-world datasets.\n\nQUESTIONS:\nCan the authors provide empirical validation of their theoretical results? While the authors provide rigorous mathematical proofs and derive explicit formulas, it would be useful to validate these results on real-world datasets.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe training of ReLU neural networks involves complex nonlinear phenomena, challenging theoretical analysis due to the nonlinearity of models and non-convexity of loss. This study provides a comprehensive theoretical characterization of the training process of a two-layer ReLU network using Gradient Flow on linearly separable data. From random initialization to final convergence, the analysis identifies four different training phases indicating a trend from simplification to complication. Specific nonlinear behaviors, like initial condensation, saddle-to-plateau dynamics, plateau escape, activation pattern changes, and increasing learning complexity, are accurately captured.\n\nSTRENGTHS:\n1. The paper presents a complete theoretical characterization of the training dynamics of 2-layer ReLU networks, which addresses the nonlinearity and non-convexity challenges.\n2. The precise characterization of four distinct phases of the training dynamics.\n\nWEAKNESSES:\n- A missing related work on network's training dynamics and phases [1]\n- Could authors run more hyperparameter settings (\\kappa_1, \\kappa_2, \\delta, etc.) and see if simulated training times (T1 ~ T4) are aligned with estimated ones?\n\n[1] \"Neural Networks as Kernel Learners: The Silent Alignment Effect\" Atanasov et al. 2021\n\nQUESTIONS:\n- Do these four phases persistently exist across different settings (\\kappa_1, \\kappa_2, \\delta, etc.)?\n- For Phase III: “deactivation of positive neurons on $\\mathbf{x}-$” — is this a sign of implicit bias of NN during gradient flow?\n- Phase IV: “reactivation of negative neurons on $\\mathbf{x}+$” — is this a sign of NN's overfitting?\n- It is a bit surprising to me that network width ($m$) and learning rate ($\\eta$) are not included in the form of T1~T4. Could authors intuitively explain why is it from the viewpoint of the proof strategy in this paper?\n- One previous work on NN convergence stated that \"strong correlations\" (of input patches) \"imply faster convergence\", which contradicts the conclusions in this work (i.e. smaller $\\delta$ indicates slower convergence). Could authors compare their work?\n\n[2] \"When is a Convolutional Filter Easy To Learn?\" Du et al. 2017.\n\nLIMITATIONS:\n- Assumption 3.1 is oversimplified. Basically, all training data collapse into two signals. How will the results change when we switch to a little bit more realistic setting by adding some stochastic noises on top of $\\mathbf{x}+$ and $\\mathbf{x}-$ to each data point?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to provide a theoretical understanding of the dynamics involved in training neural networks beyond the linear regime. The authors focus on a specific scenario where a two-layer ReLU network is trained using Gradient Flow (GF) on linearly separable data. The analysis encompasses the entire optimization process, starting from random initialization and concluding with final convergence.\n\nDespite the simplicity of the model and data used in the study, the authors uncover multiple phases within the training process. By conducting a meticulous theoretical analysis, they precisely identify four distinct phases that exhibit various nonlinear behaviors.\n\nIn Phase I, the initial stage of training, there is a phenomenon of condensation and simplification as active neurons rapidly gather in two different directions. Simultaneously, the GF successfully escapes from the saddle point around the initialization.\n\nPhase II involves a prolonged period where the GF becomes trapped in a plateau of training accuracy. However, it eventually manages to escape from this stagnation.\n\nDuring Phase III, a significant number of neurons are deactivated, leading to self-simplification of the network. The GF then adapts its learning approach using this simplified network configuration.\n\nIn Phase IV, a considerable number of previously deactivated neurons are reactivated, resulting in self-complication of the network. Finally, the GF converges towards an initialization-dependent direction.\n\nOverall, the training process exhibits a remarkable behavior of transitioning from simplification to complication. The detailed analysis of each phase sheds light on the intricate dynamics involved in the learning process beyond the linear regime.\n\nSTRENGTHS:\nThe manuscript demonstrates excellent organization, comprehensiveness, and clarity, effectively covering crucial aspects such as a thorough review of related works and a comprehensive discussion of limitations. Moreover, the paper stands out by successfully combining empirical and theoretical approaches.\n\nOverall, this work makes significant contributions that advance the current state of the literature.\n\nWEAKNESSES:\nThese suggestions are intended to improve the overall clarity and accessibility of the research.\n\nThe manuscript does not provide sufficient details regarding the resources and computational aspects involved in the research. Additionally, the work would greatly benefit from a clear delineation of its limitations.\n\nOutlined below are some limitations that underscore the weaknesses of the paper, despite its current status as the best effort in the field.\n\nThe paper lacks a significant number of experimental results to demonstrate the robustness of the findings. It would greatly enhance the study's credibility to include additional examples where the predictions have been validated and to provide measures of their robustness.\n\nThe submission would be strengthened by the addition of the code ran for the experiments\n\nQUESTIONS:\nDear authors, I have a few questions regarding your work that would greatly aid my comprehension:\n\nHow reliable are the predictions made in this study, and what was the extent of experimentation conducted to support them?\n\nCan general principles be derived from this research that would benefit machine learning practitioners?\n\nHave alternative types of architectures been tested to determine if the findings hold true across different models as well (i.e., more realistic architectures and data sets)?\n\nLIMITATIONS:\nThe limitations of the proposed method are summarized as follows:\n\n* Narrow Scope: This study focuses exclusively on ReLU neural networks trained with Gradient Flow (GF) on linearly separable data. Although the analysis comprehensively captures the optimization process and identifies four distinct phases with rich nonlinear behaviors, the generalizability of the findings to other neural network types or more complex datasets may be limited.\n\n* Limited Generalization to Gradient Descent (GD): While the paper provides a detailed analysis of GF dynamics, it acknowledges that Gradient Descent (GD) dynamics are more complex and can exhibit additional nonlinear behaviors like progressive sharpening and the edge of stability. Consequently, a comprehensive understanding of the nonlinear behaviors during GD training is not fully explored in this study, indicating the need for future research in this direction.\n\n* Theoretical Understanding of Neural Network Training: While this work makes significant strides in theoretically understanding the training dynamics of neural networks, it acknowledges that there is still much progress to be made in fully comprehending the entire training process. Theoretical advancements related to nonlinear behaviors in neural network training, beyond the specific focus of this study, present valuable opportunities for future investigations. With this in mind I would like to suggest works like' The Neural Race Reduction: Dynamics of Abstraction in Gated Networks' Saxe 2022, that get analytical solution to 'Relu like' Neural Networks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors attempt an exact analysis of the training dynamics of 2-layer ReLU networks trained via gradient flow on linearly separable data. Specifically, the authors aim to build on related work (e.g., Boursier et al. [2022] on square loss) to the case of:\n- Exponential loss (a more appropriate characteristic of classification problems)\n- With data having mild orthogonal separability\n\nDue to the additional non-linearity introduced by this specific loss type, and the complex data structure, the authors aim to characterize a richer non-linear structure.\n\nBy considering a 2-layer network, where the second-layer weights are held fixed, the authors demonstrate that under gradient flow, the tunable weights of the network evolve in 4 distinctive phases, indicative of a simple to complex learning phenomena.\n\nSTRENGTHS:\n- The authors extend the analytical formulation established in related previous literature to the case of the harder exponential loss type, which categorizes standard classification loss\n- The authors relax the data orthogonality assumption utilized in these previous works by considering a case where the angle of separability between data points is $<90^\\circ$\n- The authors demonstrate additional data condensation and alignment phases over existent work at the onset and the end of the training, thus attributing to the aim of capturing richer non-linear phenomena.\n- The authors establish bounds on the measure of the bounds on the count of the tunable *positive/negative* neurons in each phase along with the same on their norms, the time extent of these phases.\n\nWEAKNESSES:\n1. While the authors do draw similarities between their work and that of [Boursier et al. [2022]]*, they refer to the latter as performing substantial simplifications and, therefore, unable to capture a lot of non-linear phenomena. Nevertheless, the authors of this work too adopt an identical model and initialization strategy as in the above-cited work.\n1.1 **Most importantly** in Assumption 3.1, are the authors comprising a dataset using **just two data points**? If so, then that is an exceptionally restrictive assumption.\n1.2 The reference to the usage of data averaged direction over the Gram-Schmidt type orthogonal direction, between that work and this, as defining a key improvement, is again not entirely justified if 1.1 above is true. In the case of a set of *non-degenerate* data vectors, defining *useful* orthogonal directions is challenging\n\nQUESTIONS:\n- Can you detail how In Fig. 1, the projection to a 2D subspace span was performed?\n- If the specific dot product of Line 151 has to be true, then data should be normalized, yet that is not mentioned in Assumption 3.1\n- Can you explain the reason behind the assumption on $T_1$ in Line 191?\n- Can you at least intuitively explain why the positive neuron has initial condensation/alignment with the label averaged direction $\\mu$ and *then* transition to the expected $x_{-}^{\\perp}$ direction? On a similar note, is there a reason only the negative neurons eventually *converge toward some specific directions dependent on both data and initialization*? Is it due to $p>1$? If so, I would assume the alignment signal impacts the positive labels over the negative ones, though.\n\nLIMITATIONS:\nSee Questions and Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}