Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks
Authors: Mingze Wang
Chao Ma
Venue: NeurIPS 2022 Accept (2022)
Keywords: non-convex optimization
training dynamics
neural network

Abstract: The convergence of GD and SGD when training mildly parameterized neural networks starting from random initialization is studied. For a broad range of models and loss functions, including the widely used square loss and cross entropy loss, we prove an ''early stage convergence'' result. We show that the loss is decreased by a significant amount in the early stage of the training, and this decreasing is fast. Furthurmore, for exponential type loss functions, and under some assumptions on the training data, we show global convergence of GD. Instead of relying on extreme over-parameterization, our study is based on a microscopic analysis of the activation patterns for the neurons, which helps us derive gradient lower bounds. The results on activation patterns, which we call ``neuron partition'', help build intuitions for understanding the behavior of neural networks' training dynamics, and may be of independent interest. 

Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks
Mingze Wang
Chao Ma Department of Mathematics Stanford University Stanford, CA 94305
School of Mathematical Sciences Peking University
Beijing, 100081, P.R. China
mingzewang@stu. pku.ssstt.uk............ pkw.pdu.ccom cn
thaoma@stanford.c edu
Abstract
The convergence of GD and SGD when training mildly parameterized neural networks starting from random initialization is studied. For a broad range models and loss functions, including the most commonly used square loss and cross entropy loss, we prove an "early stage convergence" result. We show that the loss is decreased by a significant amount in the early stage of the training, and this decrease is fast. Furthurmore, for exponential type loss functions, and under some assumptions the training data, show global convergence of GD. Instead of relying on extreme over-param pwerraaramttrizzaioon our study is based on a microscopic analysis of the activation patterns for the neurons, which helps derive more powerful lower bounds for the gradient. The results on activation patterns, which we call "neuron partition", help build intuitions for understanding the behavior of neural networks' training dynamics, and may be of independent interest.
1 Introduction
As deep learning shows its capability in various fields of applications, extensive researches are done to theoretically understand and explain its great success. Among the topics covered by these theoretical studies, the optimization of deep neural networks one of the most crucial one, especially given the fact that simple optimization algorithms such as Gradient Descent (GD) and Stochastic Gradient Descent (SGD) can easily achieve zero training loss (Zhang et al., 2017), although the loss landscape of training neural networks is highly non-convex (Li et al., 2018; Sun et al., 2020). Existing works attempting to answer the surprising convergence ability usually work in settings that do not align well with realistic practices. For example, the notable Neural Tangent Kernel (NTK) theory (Jacot et al., 2018; Du et al., 2018; 8; 2019; Allen-Zhu et al., 2019b; Zou et al., 2018) shows the global convergence of SGD in highly over- parameterized regime, in which the loss landscape is approximated by quadratic function. In practice, though, the loss landscape is indeed highly non-convex with spurious local minima and saddle points (Safran and Shamir, 2018; Safran al., 2021), and the dynamics of neurons show nonlinear behaviors (Ma et al., 2020).
Nevertheless, fast decreasing of the loss value always happens when the network trained is not highly over- paramttrrimde at least in the early stage of training. drastic decreasing at the beginning of the training. In many cases, this decrease of loss even continues until the loss achieves zero, i.e. the optimization algorithm fully converges. In this paper, study the early stage or full convergence of GD and SGD when under aader-parameterized or mildly over parameeteette models are trained. Specifically, we answer the following two thcoretical questions:
common that the loss experiences
When train practical-sizz neural networks by GD or SGD, I. Does the fast convergence in the early stage of the training provably exist? If so, how
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

long will the phenomenon last and how much will loss descend in the early stage?
2. Can the global convergence be proved under some special conditions on loss finction and training data?
Our answers to the questions above are roughly summarized in the following main theorems:
Main Theorem 1 (Informal statement of Theorem 4.2 and 4.6), For mildly wver-parameterized or under-paranee pateeer-rraeeeiized two-layer neural networks with quadratic loss or general classification loss (Assumption 4.4), let parameters be trained by GD or SGD started with random initialization. If the learning rate 1) S 0.01, then in the first (1/1) iterations, the loss will descend (21).
Main Theorem (Informal statement of Theorem 5.2 and 5.3). For mildly over-parameterized or under-pa parraetrrreete two-layer neural networks with exponential-rype loss (Assumption 5.1), let parameters be trained by GD with random initialization and proper learning rate. If data is well-separated (satisfies Assumption 4.1), the loss converges to O at exponential rate or arbitrary polynomial rate, depending on the conditions.
In the first main theorem, we demonstrate that the fast convergence in the early stage of training happens under weak conditions. The model does not need to be highly over-parameterized, and the loss function can take quite general forms. In the second main theorem, we further prove the full convergence of GD for exponential-type loss and well-separated dataset. These assumptions on the loss function and training data are close to practice and widely picked by previous theoretical studies (Ji and Telgarsky. 2018; 2020; Phuong and Lampert, 2020; Chatterji et al., b;;;;;a; Lyu and Li, 2019; Lyu et al.. 2021). "Full convergence" means that our analysis covers all the stages of the training process, starting from initialization to the convergence to zero loss. This is different from existing works which focus on the convergence process after the training accuracy hits 100% (Lyu and Li, 2019; Ji and Telgarsky, 2020; Chatterji et al., 2021b). Moreover, we provide the convergence rate of GD for two classes of neural networks.
During the theoretical analysis, we study the dynamics of neurons in detail and capture the effect of each sample on each neuron. We call our results "neuron partition", which provides a accurate description to the behavior of neurons. With the neuron partition results, we derive novel gradient lower bound for mildly parameterized neural networks, which is vital for the convergence results. The neuron partition also provides rich intuition for how the convergence happens. When GD or SGD is used with small initialization, the network is initially close to the saddle point at 0, which might be hard for GD to escape. However, neurons will adjust their directions rapidly and enter good region which contains neither spurious local minima nor saddle points. Then, neurons will keep moving through the right directions for (11/n) iterations, during which the loss descends quickly and significantly. For ssponentia--yyp loss and well-separated training data, the second stage will proceed uninterruptedly until training is stopped.
2 Related Work
Due to the highly non-convex loss landscape (Li et al., 2018; Sun et al., 2020), classical optimization theories such convex optimization fail to characterize the convergence of training neural networks. Researchers have proposed theories specifically for neural networks. We list some below.
A popular line of works focus the highly peraattttttrr neural networks and derive the "Neural tanget kernel (NTK)" theory. In the NTK regime (Jacot et al., 2018; Du et al., 2018; 2019; Allen-Zhu et al., 201 9a;b; Arora et al., 2019a;b; Daniely, 2017: Zou et al., 2018; Li and Liang, 2018; Chizat et al., 2019; E et al., 2019; 2020), the neural network model is close to a kernel method, leading to the nearly convex optimization landscape. Global convergence is proven in this regime. Besides, for highly over- perrpaaraeteriieed neural networks, the mean- field approach (Chizat and Bach, 2018; Mei et al., 2018; 2019) is another line, which analyze the training dynamics by Wasserstein gradient flow.
However, the NTK regime is different from real practical neural networks in several aspects. First, practical neural networks are often mildly over-parameterized rather than highly werrparameterized (Livni et al., 2014). Second, the landscape for practical neural networks is usually more complicated, containing local minima and saddle points (Safran and Shamir, 2018; Safran et al., 2021; Ding et al., 2019). Finally, the empirical superiority of neural networks over kernel methods is obvious. For

---

example, neural networks can learn single neuron efficiently (Yehudai and Ohad, 2020; Wu, 2022), while kernel methods fail unless the network size is exponentially large with respect the input dimension (Yehudai and Shamir, 2019).
Furthermore, optimization theories for neural networks beyond the NTK regime has also been studied. Safran and Shamir (2018); Ding et al. (2019) proved that spurious local minima are common in the loss landscape. Safran et al. (2021) pointed out that neither one-point convexity nor Polyak- Lojasiewicz condition holds near the global minimum. In (Lyu and Li, 2019; Ji and Telgarsky. 2020; Chatterji et al., 202 b;a; Zhou et al.. 2021), local convergence results are given with cross-entropy loss or some special networks at the late stage of training, i.e. when training loss is small enough. In (Li al., 2020; Phuong and Lampert, 2020; Lyu al., 2021; Jentzen and Riekert, 2021: Cheridito et al., 2022), global convergence results are given for some special networks, special data distribution or special target functions.
Preliminaries
3.1 Notations
We use bold letters for vectors or matrices and lowercase letters for scalars, e.g. x = (#11,'. :
Rd and = (Pig) mm x E RR xma, We use (+.:) for the standard Euclidean inner product between two vectors, and ||-|| for the l2 norm of a vector or the spectral norm of a matrix. We use S b to indicate that there exists an absolute constant e > such that a I cb, and 2 b is similarly defined. We use standard progressive representation O $2, θ to hide absolute constants. For any positive integer n, let [n] = {1,- ... n). Denote by NJu, Σ) the Gaussian distribution with mean μ and covariance matrix Σ. U(S) the uniform distribution on set S. Denote by I{E) the indicator
function for an event E. For a square matrix P. we use Amin (P) to denote its smallest singular value.
3.2 Problem settings
In this paper, we consider supervised learning problems. Let P be a data distribution on Rd x RC, We are given n training data = (<x,,,))); y,));=1 IRd RC drawn i..dd from P. Without loss of generality, we assume ||>:|| < 1 for any (æ, y) sampled from P.
We consider the empirical risk minimization (ERM) problem, which tries minimize the empirical risk C(-) with loss function (()))):
n (y f(:; 0)),
((0)
n
where f(x;; 0) E RC is the model and 0 represents all parameters of the model. We will give specific forms to the loss function and the model in the analysis in later sections.
We use Gradient Descent (GD) or Stochastic Gradient Descent (SGD) starting from random initial- ization to solve the ERM problem. The update rules of GD batch SGD can be written
GD 0 + 1) 0() - me (())))
SGD
O(1 + 1) (()
Σ
Vily., [(x.; 0())),
(3)
B
SEB.
where B, = (_.,,... B is a batch, and 1t,1, R. iid. ~
U([z)) and are independent with
(1)). The random initialization will be specified in each theorem.
4 Early Stage Convergence
In this section, we state and discuss our main results on the fast convergence in the early stage of training. The models we focus on are mildly over-parameterized or under-parameterized neural networks. As a warm up, in Section 4.1, we study a simple case with binary classification problems and quadratic loss. Then, in Section 4.2, we extend our results to multi-class classification problems with general loss and one-hot labels. We provide discussions to our results in Section 4.3.
3

---

4.1 Binary classification with quadratic loss
In this subsection, we study the ERM problem (1) with quadratic loss {(y1. y2) 2/111 "22)² and the two-layer ReLU neural network model without bias:
m
f(z; Σ
ak (bb x),
(4)
where (2) = (z) = (a1, - am bj random initialization
b E gdm+m We consider the following
(0) i.d. U(+1/Vm), (x( iid N(O La),
for k E [n].
(5)
md
where k is constant that controls the initial scale of the first layer.
We focus on the training data with two classes with good separation given by the following assumption. Assumption 4.1. (i) is even. y/ = for E [n/2]]: yı = -1 for E [m] [n/2]. x. æjj 2 0 for j in the same class; x; 2 I 0 for i.j in different classes. (ii) There exists a constant fi0 >> 0, s.t. y æ 2 140-
min
max
[E]])).E6R'00 T 50) [je[n] my >0}
For Assumption 4.1 (i), similar assumption has been used in prior works (Phuong and Lampert, 2020).
For image classification problems, this assumption can be ensured by simple transformation on the data due to the non-negative pixels of images. Specifically, given an image dataset consisted of two classes, we just need to replace æ with - for any (æx, y) in the second class. Assumption 4.1 (ii) is a technical assumption used in the analysis. It is not a strong addition over Assumption 4.1 (i). For example, if there exists a (Zo: Vo) in the dataset S such that (-Zo. -Vo) is also in S, then the dataset S satisfies Assumption 4.1 (ii) with No = (the upper bound of μo is 1). Admittedly, this technical assumptions is limitation of our theory it restricts its applicability. We will search for the relaxation of this assumption future works.
Now we state the following result.
Theorem 4.2. Suppose Assumption 4.1 holds. Let (t) be the parameters of model (4) trained by Gradient Descent (2) with quadratic loss. If the width m ([log(n/8)) the imput dimension d = 2ilog m)) the learning rate 1ht = 11 S 0.01 and the initialization scale k = Ompoo/n /n) in (5), then, with probability at least 1 - 2me 2d, the loss will descend (2(1) in = O iterations.
From our proof in Appendix A, we have the following corollary with fixed numbers about 77, K, m instead of progressive expressions.
Corollary 4.3. Suppose Assumption 4.1 holds. Let 0(t) be the parameters of model (4) trained by Gradient Descent (2) with quadratic loss. Let 1) 0.01, k. S min /2000, m1o//3n, m >
max 144 log(2n² (2))) 4),, and 71, 72 = (1) be constants related with the data distribution defined in (10) in Appendix A. Then, with probability at least 5 2me 2d in T = 44 iterations, loss will descend at least 0. 0.193(m1 - 72 / log(n2 Log(n2/8)/m)) 0.0111 = 2(1 - Mlg(nn/8/mm)
Conclusions in Theorem 4.2 show the meaning of "mildly parameterized" in our title. The network does not need to be pver-parameterizee, i.e. the number of parameters may be smaller than the number of training data. Hence, we pick the term "mildly parameterized" instead of the more widely use "mildly over- wer-parameterriedd.. The former is weaker than the latter.
4.2 Multi-class classification with general loss and one-hot labels
In this subsection, we extend the early stage convergence results above to more practical settings. Specifically, we consider classification problem with one-hot labels and the general loss functions that satisfy the following conditions.
Assumption 4.4. The loss function ((:,)) be expressed f(y 22) ElyT y2) such that: (i) ē(-) is twice differentiable in IR; (ii) ε(-) is non-negative and non-increasing in R; (iii) there exist 20 E (0, 1), Jmiin Imax > 0, Amax 2 such that Jmin S - (2) S ymax and 0 S (=) S h hold for any [0, 20).

---

1 Imax 1, It is easy to verify that exponential loss fyy) y2) = e v2 (=0 = 1, Jmin
mmax = 1). logistic loss E y22 = logg(1 + e vi 1/2 (20 = 1, Imin = c+1 Imnx
1 hmax = 11) 2
and hinge loss ( 122) = max{0, 1 v リ2) (=0 1, 9min = 1, Imax 1, Amax
0) all satisfy
the conditions in Assumption 4.4.
On the training data, we impose the following assumption, which is milder than Assumption 4.1. Assumption 4.5. There exists $ > -1 -1 (z;, 23) 2 for any i.j E [m]]
Note that now we do not need any separability of training data, but only require there is no pair of data going into opposite directions. (We can even normalize the data and let ||>||| < 11-6 for some e > 0, rather than | 1. Then, the above assumption naturally holds.) Notably, Assumption 4.5 is more widely applicable than Assumption 4.1 above. It holds for normalized image datasets such as MNIST (LeCun et al., 1998) and CIFAR-10 10 (Krizhevsky al., 2009).
For the model, we use the two-layer ReLU neural network model with bias as our prediction model:
fa:; 0)
ago (b x + ck),
(6)
where (2) (x)U(x), as E RC and 0 (a Ltd. N(0, m(d+lj Ia), (0) = Vm(d+1) R(4+C+1)m We consider random initialization by (0)
T b
D c1. cm) E
am
and ag (0) = (1//mm...,1// m
1/mm)T for k E [m]] where is is again a constant that controls the initial scale of the first layer.
We show the following convergence result:
Theorem 4.6. Under Assumption 4.4 and 4.5, let ((t) be the parameters of model (6) trained by Stochastic Gradient Descent (3). If the width m = (1)., the input dimension d = 2(log m), the batch size B Mloo m.) the learning rate nn = 0.01 and the initialization scale k. = (n/ B). then with probability at least 1 - 8 0 0(m0.17B), the loss will descend (1(1) in = ? iterations.
By our analysis in Appendix B, we have the following corollary which gives specific numbers about 7. K. instead of progressive expression.
Corollary 4.7. Under the same assumptions as Theorem 4.6, let ((t) be the parameters of the model (6) trained by SGD (3). Let 17- = 0.01, K S (n0nnn/1,,10 n/B m 2 6; 20 = 1, Jmin 2'
Imax = 1, hmax 1 (in Assumption 4.4) and s 0 (in Assumption 4.5). Then, with probability at
least 1- 8 4me - m0.172 in T 34 iterations, loss will descend at least 0.262533 S2(1).
We remark here that the weaker Assumption 4.5 in this subsection compared with the Assumption 4.1 before is made possible by the one-hot labels. For one-hot labels, each label component is non-negative. Hence, neural networks can quickly Iearn a positive bias, allowing the loss to drop significantly.
4.3 Discussion
A large number of classical convergence analysis focus on the late stage of training. For neural net- works' optimization, many works focus the period of training after the loss is smaller enough (Lyu and l., 2019; Ji and Telgarsky, 2020; Chatterji et al., b; Zhou et al., 2021). However, the early stage is at least as important as the late stage, especially for non-convex problems.
The theoretical results we present in this section demonstrate the fast convergence during the early stage of optimization, i.e. the (short) period of time after the initialization. The two main results, Theorem 4.2 and 4.6, imply that if we train two-layer neural networks by GD or SGD with learning rate 17 < 0.01, then the loss will descend significantly (?((1) in the first T @(1/n) iterations.
Our results work under realistic settings. For models, our results hold for mildly over-parameterized and inder-parameterized neural networks, because we only need the width yn (((lg(nn/8)). For loss functions, our results apply for practical losses, such as quadratic loss and cross entropy loss. On the algorithm side, the results work for both GD (2) and SGD (3).
5

---

Though both concerning convergence of GD and SGD, our results are essentially different from the NTK theory, especially on the requirement of puramearamtiiiniaaonnn Our results only need while in the NTK analysis usually assumes that m is polynomial in n, 1/k etc, excluding the practical settings. Moreover, from the analysis side, we establish fine-grained analysis of the training dynamics of each neuron, which may be of independent interested. See Section 6 for a summary our techniques, and Appendix A and B for the detailed proof. Our analysis helps understand how the convergence in the early stage happens. When GD or SGD is used with small initialization, the initial network is close to the saddle point at O. However, neurons will adjust directions rapidly, and the iterator will enter good region which contains neither spurious local minima nor saddle points. Then, neurons will keep going towards the right directions for a period of time, during which process the loss will descend fast and significantly.
m =[(log(n/
5
Global Convergence
When stronger conditions are imposed on the data distribution and the loss function, we can show global convergence of GD, i.e. starting from any initialization, the GD descends the loss to O.
For loss functions, we consider the following exponential-type classification loss:
Assumption 5.1. The loss function ((:-) can be expressed as E(y1 y22 = {(y) y2) such that: (i) ((-) is twice continuously differentialle in R; (ii) E(-) is positive and non-increasing in IR; (iii) there
exist 9b > 0, 2 s.t. (=)) S 9b and S [=) S for any z E R; (iv) there exists ga > 0, s.t. E(x)
{(x)
[()= for any 2 2 O. ((=)
9a S
The similar assumption exponential-type loss has been used in prior works (Lyu and Li, 2019). It easy to verify that both exponential loss ( H2) e-vi
vi v/a (g. = gb = h 1) and logistic loss ( 82) = log(1 + e-! 1/) (ga 2 9b = 1, = 1) satisfy the conditions Assumption 5.1.
Let V be constant defined as
V 11111 (
1
8 8log(n2/0) m
{2 n
Y, Amin (X.Xx) A Amin x (7)
max
n E Rdx X_ == (1g+1 .In) R 9, and = min nTTx; æT 2 O. We have the following convergence theorems in which V controls the j.jin the some class
16
where X + = (x1 2g
convergence speed.
Theorem 5.2. Under Assumption 5.1 and Assumption 4.1, let O(t) be the parameters of model
(4) trained by Gradient Descent (2) starting from random initialization (5). Let the width m = (log(n/5)) the initialization scale Ompuo n) in (5), V be defined in (7), c be a constant in (0, 6(1+2n0)) 442 the constant > 0 be sufficiently small, the hitting time To = [n log 2) Ga 1, the parameter E (o)) and the learning rate satisfy
1/0 S 1/2/2.
for = 0 for 1 < < To
7) T
7: -(0()))---
fort 2 To
Then, with probability at least
8 - 2me 2d, GD will converge at polynomial rate:
((((t)) < ((0(1))
1st<
Ve
((O(t)) = O()). t2
Theorem 5.3. Under Assumption 5.1 and Assumption 4.1, let parameters of model (4), and only train this layer by Gradient Descent (2) starting from random initialization (5). Let the width m = ((log(n/5)) the initialization scale k = O(mppo/ ni) in (5), the constant V be defined in (7), the constant S 1 and the learning rate satisfy
be the imput layer
{ No S 1/2v2, for = 0 1/2 Z[ for t 2

---

Then with probability at least 1 - 2mm GD will converge at exponential rate:
t-1 ((((1)), 12 ((((1)) - (1 (1-
Theorem 5.2 and 5.3 describe the whole training process starting from random initialization to convergence. If all parameters in the network are trained, we obtain the global convergence with arbitrary polynomial rate. On the other hand, if only the input layer parameters {bx Jkemm are trained, we can obtain exponential convergence rate. Fixing some layers of the neural network is a common practice for theoretical studies (Chatterji et al., 2021b). The results show that adaptively increasing learning rate helps GD achieve faster convergence. Similar idea has been explored in previous works such (Lyu and Li, 2019; Chatterji et al., 2021b).
similar convergence result was proven in (Phuong and Lampert, 2020). where the authors showed global convergence result of two-layer neural networks trained by Gradient Flow (GF) with cross entropy loss and orthogonal separable data. The main difference between our results and analysis with that in (Phuong and Lampert, 2020) is that we study GD instead of GF, Due to the discretization, GD is more complicated than GF, For example, the balanced property of two layers (kEjj) lag (t)l Ekejm) [t]]( = O, 2 0) holds for GF throughout the training, but it does not hold for GD. Dealing with GD requires new analysis techniques and finer analysis, such as the neuron partition results derive. Another related work is (Lakshminarayanan and Singh, 2020). The neural partition approach in our article shares some similarity with the view of gating networks. In their work, the authors use gating networks to understand the role of depth in training deep ReLUU networks. By comparison, we focus more on the characterization of training dynamics of two-layer ReLU networks.
6 Proof Sketch and Techniques
6.1 Main techniques in the proof of Theorem 4.2
Inspired by the empirical work (Ma et al., 2020), we study the nonlinear behavior of each neuron in the early stage of training. First, we define the hitting time T. and we will study the convergence in the early stage 0 s T:
T sup {t E N: max (fa;; 0())) S 1, on ())) 0, vk E [m]] VO s0t+1))) *E[n]
6.1.1 Neuron partition: fine-grained dynamical analysis of each neuron and each sample
During our proof, we conduct fine-grained analysis to the interaction of neurons and samples. Specifically, we characterize the impact of each sample on each neuron, and for each sample classify neurons into four categories according to their effect. We call this analysis "neuron partition".
For the i-th training data (æ;, Yi), we divide all neurons into four categories during training 0. t < T and study them separately. We define the living neurons TL, the true-dead neurons (D) the false-living neurons (L, and the false-dead neurons (t), at time t as:
TC,(t) {K E [m] yo (t) > (,b, > 0).
TD,(t) == [k E [m] y:a > 0, bx)) 0).
FC,(t) = {K E [m]] (t) < 0, bt) x, <.>00 >,)0,
FD.(() = {K E [m] man (1) < 0, (() a. 5]. It is easy to verify [m] (TC.
(see Lemma A.
Under a weak assumption on the width of the network, we have the following results for the neuron
partition at initialization:
Lemma 6.1 (Informal statement of Lemma A.2). If m = 2(log(n/8). then with high probability, if data and data j are the same class, we have cTrd n (0) arecoos xx) m.
4m
The next important lemma characterizes the evolution of the neuron partition under GD dynamics.
7

---

Lemma 6.2 (Informal Lemma A.4). With high probability, for any i E [n] and S T we have:
(SI) True-living neurons remain true-living rrue-iiing: (L,(t) (L +1).
(S2) False-dead neurons remain false-dead: FD((() C FD((++1)
(S3) True-dead neurons turn true-living in the firt step (t = 0): TD;(0) CTC(().
(S4) False-living neurons uurn false-dead in the firt step (t = 0): FL, FD,(1).
(S5) For any [n]. 1 S T. k
[m]] and b (t(bbbx(t + 1), we have sgn(b x,)
sgn(b, (1)x.)= * O.
To interpret Lemma 6.2, we display the first-step dynamics in Figure 1.
neuron (0g <
(0)
neuron 1 by (a: 0)
sample i mn; (3) 1)
4 by (a4 <
sample
(Mi
sample j my (M/ 1)
sample J (2)
neuron 2 by (ag 0)
newron 2 b2 (#2 0)
neuron 3 by (as 0)
neuron
(eg 0)
(a) == O.
(b) = 1.
Figure 1: The first-step dynamics of four neurons which belong different neuron partitions / for sample (with label 1) and sample j (with label -1). Fig (a) and Fig ((b) depict the directions of each neuron at and 1, respectively. Results Lemma 6.2 can visualized: (S1) neuron E (0) and 1 E (1). (S2) neuron 4 E (D) and E FD,(1). (S3) neuron 2 TC, (0) and (1). (S4) neuron E (0) and E FD,(1). (Counterexample) By our Lemma, the first-step dynamics of neuron 3 cannot happen.
6.1.2 Estimate of hitting time and parameters
Before the analysis of loss descent, need to estimate the hitting time T and the speed of change for each age by in the early stage. Our tool is comparing the hitting time T with an exponentially growing hitting time T. (see (12) in Appendix A).
Te = sup { e N: (1 + 2t++1) - (1- 1-22)) S 1; (1 +1+ (1+2n) 21) < $1).
where the order of magnitude of the two absolute constants hidden by S is 10 Generally speaking, we can prove the following facts.
Lemma 6.3 (Informal Lemma A.7 and A.8). For any sample i [n]] and 1 < Te, have: (PI) For any k E T (t), au (t)ssg((aa))) (0)) is non-decreasing, and au (t), by(t) have an exponential
upper bound.
(P2) For any E [m]. a& (t) has an exponential lower bound.
(P3) f [ (t)) has an exponential upper bound.
(P4) T 2 T. 2 T = 2
6.1.3 Gram matrix and gradient lower bound
The gradient lower bound is usually an important technical component in the analysis of loss descent. Our analysis establishes novel lower bound for the gradient for mildly parameterized neural networks based on the neuron partition.
If define the error for sample i at as et(t) f(z;; (()) Vr. and the Gram matrix at t G(t) (Vf(z;; (t)) Vf(z,; ((t)) For the analysis of GD, loss descent is

---

Ej= (tt)) Qualitatively, bigger gradient leads to the faster descent of the loss. To derive a tight lower bound for the gradient, we first derive the following bound for the entries of the Gram matrix:
determined by \VC(0(t))
Lemma 6.4 (Informal Lemma A.9). Under the condition of Theorem 4.2, with high probability, the
(t)
0gx
Gram matrix (tt = 0g x
has the following form for any S t T and i, E [
GL(
G+(t),,. (t) 2 x. æ]] of the absolute constant hidden by 2 10
a2rrosss m2)
O(V
log(n/61) m
0. where the order of magnitude
4m
Combining the estimate above as well as estimates on parameters and the prediction function, we establish our gradient lower bound.
Lemma 6.5 (Informal Lemma A.10). For any 1 s T | = 2(1 - (1 2t))
Besides, since we consider GD, the following per-step estimate of loss function (Bubeck et al., 2015) well the Hessian upper bound is also useful.
Lemma 6.6 (Informal Lemma A.11 and A.12). For < t < T - 1, we have H == [ = O(1) and (( + 1)) S (() + (((())+(((((0)))))))) ((( 0(1) +
sup along GD rujjectory
H 100 + 1) 0
6.2
Proof outline of Theorem 4.2
First, we can estimate the initial neuron partition (Lemma 6.1) at the initialization. Then, we study the dynamics of neuron partition trained by GD (Lemma 6.2), which gives precise directional characterization about each neuron for each sample. Next, by comparing with an exponentially growing hitting time, we can estimate the hitting time T and the parameters (Lemma 6.3). Combining Lemma 1. 6.2 and 6.3, we derive a dynamical lower bound of each element of Gram matrix (Lemma 6.4), which induces our gradient lower bound (Lemma 6.5). Finally, by combining the gradient lower bound (Lemma 6.5) and the loss upper bound (Lemma 6.6), we complete our proof. For more details, please refer Appendix A.
The proof of Theorem 4.6 is similar to Theorem 4.2 and put into Appendix B.
6.3 Proof outline of Theorem 5.2 and 5.3
The proofs of Theorem 5.2 and 5.3 still rely on the dynamical analysis of neuron partition for each sample. First, the estimate of initial neuron partition (Lemma 6. 6.1) also holds, and the dynamical analysis of neuron partition in Lemma 6.2 holds for any t > 1 (Lemma C.5 and C.6). Second, can prove that the structure and lower bound of Gram matrix in Lemma 6.4 hold for any t 2 1 (Lemma C.9), which induces the gradient lower bound below. Finally, combining the estimate of gradient lower bound, Hessian upper bound and loss upper bound, we complete our proof. For more details, please refer to Appendix C and D.
Lemma 6.7 (Informal Lemma C.10). Under the condition of Theorem 5.3, with high probability, we have the following lower bound for any t Z 1: [VL(((()) 2 L(O(t)), where the absolute constant hidden by Z is vV, defined (7).
In Lyu and Li (2019); Chatterji et al. (2021b), gradient lower bounds are derived for the late stage of training when L(O(t)) < 1/n, and take the form (((((())) 2 (()(t)) log (1/C(0(t)) [[O())]]- Our lower bound, instead, works on the whole training process. With this stronger gradient lower bound, we can show the exponential convergence in Theorem 5.3.
Another important technical detail is the stability of GD, i.e., the learning rate should be upper bounded by the quantity related with the top eigenvalue of the Hessian. Our analysis also considered this factor, but implicitly. For global convergence results, the loss function we consider is exponential- type, whose Hessian can be controlled by the loss value. For example, for Theorem 5.2, there exists (L)((( for any /. 2 1, please refer to Lemma C.15 in the appendix. Therefore, we can transform the Hessian-controlleed learning rate condition into the loss- learning rate condition in Theorem 5.2.
an absolute constant C >>> 0, s.t. || V ((O)((

---

7 Experiments
MNIST and CIFAR-10 experiments. As we mentioned in Section 4.2, our theory about early stage convergence applies to a wide range of dataset (Assumption 4.5) such as MNIST and CIFAR-10. We the two datasets (with normalization) compare the experimental result with Theorem 4.6. Specifically, we use the first 1000 data in MNIST dataset and the first 1000 data in CIFAR-I0 dataset, separately. (The two datasets with normalization |
both satisfy Assumption 4.5.) And we use the two-layer ReLUU network with the logistic loss { y2) log(1 + exp(- yy y2))- m. Ki 1] choosen by Theorem 4.6. We study the change of our bounds under different network width m and learning rate v). All experiments are conducted on MacBook pro 13" only using CPU. See the code at https: / / thub. com/ hwmm//wm//////// Early SaalyySSaaaeeee wet.eeoeeeeeeeeeeeeeeeeee MtwpineeeeerrrPPPPPOP IPS2022.
Table 1: Results of MNIST and CIFAR-10 experiments. In the first table, we fix 1 0.01 and change m, in the second one, we fix m 200 and change 1).
m Our hitting iteration
100
200
500
1000
34
Realistic loss descent (CIFAR-10)
3.45 x 10-1
5.34 x 10
6.11 x 10-1
6.53 10
Realistic loss descent (MINST)
3.89 X 10
4.78 x 10 -1
5.97 x 10-1
6.45 x 10"
Our loss descent bound
2.63 x 10
0,01
0.005
0.002
0.001
Our hitting iteration
34
69
173
346
Realistic loss descent (CIFAR-10)
4.89 x 10-1
5.34 X 10
5.43 x 10-1
5.37
10
Realistic loss descent (MINST) Our loss descent bound
4.78 X 10-1
4.98 x 10-1
4.95 x 10-1
4.92
10
2.63 x 10-1
The results in Table 7 show that our loss descent bound (5-th row in each table) is relatively tight, basically in the same order of magnitude as the realistic loss descent (3-rd row and 4-th row each table). And our loss descent bound does not change with the choice of different datasets.
8 Conclusion and Future Work
In this paper, we study the convergence of GD and SGD when training mildly parameterized neural networks. On one hand, we show early stage convergence for wide range of loss functions, optimization algorithms, and training data distributions. On the other hand, under some assumptions on the loss function and data distribution, we show global convergence of GD. Our analysis can be extended to the minimization of population risk (See Appendix E).
The theoretical understanding of the training of neural networks, especially for neural networks with practical sizes, still has a long way to go. For instance, although the late stage training for exponential loss function is simple and clear, that for other losses is much more complicated. Phenomena like unstable convergence (Wu al., 2018; Cohen al., 2021; Ahn al., 2022; Ma et al., 2022) happen. Better understanding of these phenomena during training is an important direction of future work.
Acknowledgments and Disclosure of Funding
We thank anonymous reviewers for helpful suggestions. Mingze Wang is supported in part by the National Key Basic Research Program of China: 2015CB856000.
References
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. arXiv preprint 2:X.iv2220.00000 2022.
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances neural information processing systems, 2019a.
10

---

Zeyuan Allen-Zhu Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pages 242-252, PMLR, 2019b.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. International Conference on Machine Leurning, pages 322-332. PMLR, 2019a.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances Neural Information Processing Systems, 32, 2019b.
Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 8(3-4):231-357 2015.
Niladri Chatterji, Philip M Long, and Peter Bartlett. When does gradient descent with logistic loss interpolate using deep networks with smoothed relu activations? In Conference on Learning Theory, pages 927-1027. PMLR, 2021a.
Niladri Chatterji. Philip M Long, and Peter L Bartlett, When does gradient descent with logistic loss find interpolating two-layer networks? Journal of Machine Learning Research, (2(159):1445 2021b.
Patrick Cheridito, Arnulf Jentzen, Adrian Riekert, and Florian Rossmannek. proof of convergence for gradient descent in the training of artificial neural networks for constant target functions. Journal of Complexity, page 101646, 2022.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over werrparameterized models using optimal transport. Advances in neural information processing systems, 31, 2018.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009.
Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge stability, arXiv preprint arXiv:2103.0000 00066, 2021.
Amit Daniely. Sgd learns the conjugate kernel class the network. Advances Neural Information Processing Systems, 30, 2017.
Tian Ding, Dawei Li, and Ruoyu Sun. Sub-optimal local minima exist for neural networks with almost all non-linear activations. arXiv preprint urXiv: 1.01413,,, 2019.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675- 1685. PMLR, 2019.
Simon Du, Xiyu Zhai, Barnabas Poezos, and Aarti Singh. Gradient descent provably optimizes 2018.
over-parameterized neural networks. arXiv preprint
Weinan E, Chao Ma, Qingean Wang, and Lei Wu. Analysis of the gradient descent algorithm for deep neural network model with skip-connections. arXiv preprinl urXiv: 052663 2019.
Weinan E, Chao Ma, and Lei Wu. A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics. Science China Mathematics, 63(7): 1235, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1806.0777 075778076007... 2018.
Arnulf Jentzen and Adrian Riekert. A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with relu activation for piccewise linear target functions. arXiv preprint 2rXiv:2108.04622 2021.
11

---

Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. arXiv preprint orXiv:1810.020333 2018.
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 11517776--71888 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Chandrashekar Lakshninarayanan and Amit Vikram Singh. Deep gated networks: A framework to understand training and generalisation in deep learning. arXiv preprint arXir:2002.03996, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( (6(11):2278-2324 1998.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. Advances Neural Information Processing Systems, 31, 2018.
Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with relu activation. Advances in neural information processing systems, 30, 2017.
Yuanzhi Li, Tengyu Ma, and Hongyang R Zhang. Learning over-parametrized two-layer neural networks beyond ntk. In Conference on learning theory, pages 2613-2682. PMLR, 2020.
Roi Livni, Shai Shalev-Shwart and Ohad Shamir. On the computational efficiency of training neural networks. Advances in neural information processing systems, 27, 2014.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogencous neural networks. arXiv preprint arXiv: / 199660059000 2019.
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems, 34, 2021.
Chao Ma, Lei Wu, and Weinan E. The quenching-activation behavior of the gradient descent dynamics for two-layer neural network models. arXiv preprint arXiv:2006. 14450, 2020.
Chao Ma, Lei Wu, and Lexing Ying. The multiscale structure of neural network loss functions: The effect on optimization and origin. arXiv preprint arXiv:2204.1 1326, 2022.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115633):77665-7777 2018.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on Learning Theory, pages 2264. PMLR, 2019.
Mary Phuong and Christoph H Lampert. The inductive bias of relu networks on orthogonally separable data. In International Conference on Learning Representations, 2020.
Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International conference on machine learning, pages 4433-4441. PMLR, 2018.
Itay M Safran, Gilad Yehudai, and Ohad Shamir. The effects of mild over pwe-paaramttriization on the optimization landscape of shallow relu neural networks. In Conference on Learning Theory, pages 3889-3934. PMLR, 2021.
Ruoyu Sun, Dawei Li, Shiyu Liang, Tian Ding, and Rayadurgam Srikant. The global landscape of neural networks: An overview. IEEE Signal Processing Magazine, 37(5):95-108 2020.
Yuandong Tian. An analytical formula of population gradient for two-layered relu network and its applications convergence and critical point analysis. In International conference on machine learning, pages 3404-3413. PMLR, 2017.
12

---

Lei Wu. Learning a single neuron for non-monotonic activation functions. NDw:22022080644, 2022.
arXiv preprint
Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over over-parameterizae learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31: 8279 8288, 2018.
Gilad Yehudai and Shamir Ohad. Learning a single neuron with gradient methods. In Conference on Learning Theory, pages 3756-3786. PMLR, 2020.
Gilad Yehudai and Ohad Shamir. On the power and limitations random features for understanding neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization (2016). arXiv preprint arXiv: 161 I.03530, 2017.
Kai Zhong. Zhao Song, Prateek Jain, Peter I. Bartlett, and Inderjit S Dhillon. Recovery guarantees for lae-hidden-llyer neural networks. International conference on machine learning, pages 4149. PMLR, 2017.
Mo Zhou, Rong Ge, and Chi Jin. A local convergence theory for mildly over- wer-parameterized two-layer neural network. In Conference on Learning Theory, pages 4577-4632. PMLR, 2021.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks. arXiv preprint arXiv:1811.088 08888, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] The global convergence results in
Section 5 are built on restrictive assumptions of the loss function and training data.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In Section 3, and 5.
(b) Did you include complete proofs of all theoretical results? [Yes] In Appendix A. B, C,
D, and G.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] As a URL in Section 7.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Section 7.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Section
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "tQqCVCCYT-M", "reviewer": "Reviewer_uRDi", "summary": "This paper studies the early stage of convergence for moderate width networks optimized via the squared loss and logistic-style loss functions.  More precisely, they demonstrate that the loss will descend by an $\\Omega(1)$ quantity in the early stage of training.  If the data are very well-separated, for logistic-style loss functions they can demonstrate global convergence.  A contribution is that they only require logarithmic width for the network.  As a matter of technique the novelty is that rather than prove NTK deviation bounds they control the NTK gram matrix via fine grained analysis of the neurons.  While previous works such as [1] and [2] use overparameterization to demonstrate that the activation patterns do not change much, the work under consideration does not rely on overparameterization and performs a more refined analysis of neurons based on true/false-alive/dead pairings.\n\nReferences:\n\n[1] Gradient Descent Provably Optimizes Over-parameterized Neural Networks by Du et al.\n\n[2] Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks by Arora et al.", "questions": "Most of my questions were outlined in the “Strengths and Weaknesses” section, however I will reiterate these here as well as introduce some new questions.\n\nQuestion 1: Could the authors provide a more detailed discussion of how their analysis of the early stage of training is able to improve over what more standard NTK analysis would give.\n\nQuestion 2: 2nd inequality of line 570 could be explained better.\n\nQuestion 3: In Lemma A.13 the authors upper bound the gradient deviations using the loss Hessian.  However, for ReLU the loss Hessian does not exist in the proper sense.  I believe this can be overcome using a different argument however this discrepancy must be addressed or at least explained.\n\nQuestion 4: Could the authors provide a more detailed discussion of the scale of the initialization.  If $\\kappa$ is small then the gradients of the inner layer weights will dominate over the gradients of the outer layer weights (due to properties of ReLU).  This is not bad necessarily however this should be explained to the reader.  I believe this is essential to the proofs.  For example the authors study the stopping time $T$ up to which the signs of the outer layer weights haven’t changed.  The size of $\\kappa$ will be quite relevant for determining how quickly the outer layer weights change in sign.\n\nQuestion 5: Could the authors provide a more detailed comparison of their work and [3] which addresses some of my concerns outlined in the “Strengths and Weaknesses” section.\n\nReferences:\n\n\n[3] Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks by Ji and Telgarsky https://arxiv.org/pdf/1909.12292.pdf", "limitations": "The main contribution of this work as I see it is to provide a fine grained analysis of the neurons for the shallow ReLU network.  This is indeed interesting, although it will be difficult to generalize to multiple layers or to activation functions other than ReLU.\n\nBased on my concerns in the \"Strengths and Weaknesses\" section, I do not view the present version of this manuscript to be ready for publication in this conference.  If the authors provide strong responses to my concerns to show that I am mistaken or missed crucial points I will consider raising my recommendation.\n\n**Edit after rebuttal: I upgraded my recommendation from a 3 to a 5 based on the edits and responses from the author.  Please see my comments below for my reasoning.**\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe strength of the paper is that it only requires logarithmically wide networks and it provides a fine-grained analysis of the neurons that is novel as far as I can tell.  Furthermore the writing and presentation is easy to follow.  However, I have some concerns about this work.  I would like the authors to provide a more detailed explanation about how their analysis is able to prove results that cannot be proven using existing tools from NTK theory. If one is only interested in analyzing the early stage of training one does not need overparameterization to perform standard NTK analysis.  Up to a $O(1)$ stopping time, parameter deviations are $O(1)$ and thus NTK deviations are $O(1/\\sqrt{m})$.  Thus if one is only interested in the early-time dynamics the width of the network can be taken independent of $n$.  As far as the statements for global convergence, under Assumption 4.1 the data are linearly separable.  When the data are linearly separable with positive margin, the target function is separable with positive margin using the NTK model see [3].  For this case [3] demonstrated that polylogarithmic width suffices for the logistic loss to achieve global convergence.  To be fair in [3] they only train the hidden layer, however due to $\\kappa$ being small in the present work the gradients of the hidden layer will typically be larger than the gradients for the outer layer.  In this case the changes in the hidden layer should be more significant than the changes in the outer layer.  Thus I do not see this being too essential here.  Also in Theorem 5.3 the authors only train the hidden layer which allows a more direct comparison.\n\nReferences:\n\n[3] Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks by Ji and Telgarsky https://arxiv.org/pdf/1909.12292.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qgtt6tsBAbE", "reviewer": "Reviewer_8f7C", "summary": "This paper studies the convergence of GD and SGD when training mildly parameterized neural networks starting from random initialization. The authors prove an \"early stage convergence\" that the loss is decreased by a significant amount in the early stage of the training quickly. ", "questions": "Assumption 4.1 assumes that close data points have same output, this also implies the training data has significant low-frequency components. According the frequency principle (https://arxiv.org/abs/1901.06523), low frequency would converge at the beginning and fast, Theorem 4.2 can be intuitively understood. For Assumption 4.5, it constraints the input but not the output. That means Theorem 4.6 applies for any training data. However, in practical training, if the training data is dominated by very high frequency components, the loss may not decay or decay extremely slow. Does this example show that the \\Omega(1) bound in theorem 4.6 may be loose?\n\nThe model in Theorem 4.2 lacks of universal approximation ability (without bias), how does this result help understand a model with bias. \n\nA loss that satisfies Assumption 4.4 may not a common case in deep learning, such as MSE and cross-entropy. Can authors explain more about how Theorem 4.6 depends on the Assumption 4.4? \n\nI am still confusing about how Theorem 4.2 and 4.6 help understanding the training process. For example, can these two theorems tell us in which situation there is an early-stage convergence and there is not, or to achieve a fast convergence at the initial stage, how should we design the network.\n\nRegarding the global convergence part, the results heavily relay on the Assumption 4.1 and model (4), which seems not very realistic. Can authors discuss more on how these results can help understand the practical neural network?\n", "limitations": "The theoretical results in this paper are nice and sound solid, however, I suggest the author should make the limitation clear in abstract and introduction, which are now quite misleading. The informal statements of theorem ignore very important assumptions, which will mislead readers by regarding the results as very general results. The title and the abstract should explicitly point out the paper studies two-layer neural network at least. These limitations at least include the model, loss and data. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written, technique sound, and studies important problems of neural network training process, one is the fast convergence at the early training stage, the other is the global convergence of mildly-overparameterized neural networks. However, some assumptions may be not so realistic, which makes reader hard to connect the results and practical training process.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "L5e6lKiigg", "reviewer": "Reviewer_nRXr", "summary": "This paper investigates the dynamics of ReLU neural networks in the initial portion of training. By using the discontinuity of the ReLU activation function on the hidden layer as well as a structured dataset a partitioning of the hidden layer neurons is defined for each datapoint. Specifically hidden neurons are grouped into one of four partitions based on whether their input-to-hidden (hidden-to-output) weights have positive/negative dot product with the input (output). From this structured initial setup it is possible to partition the initial hidden neurons based on the initial weight distributions. This then allows for bounds on the network error to be defined since the dynamics of the partitioned subnetworks are more deterministic and have discernible properties (such as most neurons remaining in their original partition). Thus, the primary results of this work are theoretical statements on the lower bound which the error will decrease by in the early stages of learning and a global convergence theorem for the structured dataset.  ", "questions": "## Main Text\nI have the following questions or minor points which could be clarified (in addition to the questions which fit in with the Strengths/Weaknesses above):\n* Line 55: two full-stops after \"neuron\"\n* Line 58: It is mentioned that the loss landscape will be complicated at initialization, however, with small initial weights this does not appear to be true as the model will be sitting very close to a smooth saddle point at 0.\n* Line 95: It says capital letters are used to denote vectors but the example shows a lower case bold letter for a vector (which is used in the rest of the work)\n* Line 131 to 134: As a theoretical tool using the labels to preprocess data is fine (and I acknowledge that this assumption is not new to this work), but it limits the scope of the theory as it then cannot apply to unlabelled data or test data. Again I think this could be mentioned.\n* Assumption 4.5: it never becomes apparent to me why this assumption is necessary or different from Assumption 4.1. I would just like to understand that better\n* Line 169: Can remove \"the\"\n* Line 216: I assume the $\\lambda$'s are the singular values but we should be told what they are.\n* For Theorem 5.3: Are only the input-to-hidden weights being trained?\n\n## Appendix (optional)\n* Line 519: $0 \\leq s \\leq t+1$ is stated twice\n* Line 520: Where does the $\\gamma_1$ variable come from or how is it determined?\n* Line 539: what is $P^\\infty$, if it is the infinite width limit for the hidden layer then does this fit in with the notion of this theory being for mildly over-parametrized models?\n* Line 541: How does the following line apply from the union bound? Is it the union bound over the hidden neuron distributions?\n* Line 547: Is this mean to say \"union bound\"?\n* Line 560: Is the final piece of this line guaranteed since from the line before the $-1-f(x,\\theta)$ make it seem like the entire second part of the expression could be negative?\n\nAs a guide, if points 2 and 3 from the points under weaknesses for Quality are addressed I would be inclined to increase my score by 1. If point 1 is addressed and the results are compelling I would increase my score by 1.", "limitations": "There appear to be a couple limitations which could be acknowledged that I have pointed out above. If I am not mistaken then I think they should be mentioned. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n## Originality\nTo my knowledge this is the first work which has theoretically used neuron partitioning, however, as the paper acknowledges, this has previously been done empirically. Focusing on the early stages of learning as opposed to the more prominent approach of investigating the late stages of learning also adds to the originality. \n\n## Quality\nThe paper is of a high quality and well structures. The research itself is focused in its goal and the methodology appears appropriate. The only figure in the paper is clear and helpful.\n\n## Clarity\nThe paper is clear and well written. Notation is intuitive, used sparingly and used consistently which aids in the readability. Figure 1 adds to the clarity of the work and displays a helpful setting that demonstrates how Lemma 6.2 can be consistent for different datapoints.\n\n## Significance\nThis appears to be the most difficult part to assess and I do have some reservations (see below). However, a tighter bound on the gradient norm is given and the theoretical technique of partitioning the hidden neurons could potentially yield future insight. For me the rigor of this work is what adds to the significance substantially. Having gone through Appendix A quite carefully and Appendix C (it seems the proof technique is similar for all Appendices) the proofs seem correct and rigorous.\n\n# Weaknesses\n## Originaity\nThis work does bear some resemblance to prior theoretical work on gated neural networks, see [1] for example. I think this should be acknowledged.\n\n## Quality\nI have 3 primary concern from the main text of this work\n1. This work could benefit greatly from a simulation of the derived equations. While I think purely theoretical work is certainly acceptable, the theory should still be checked against even a toy example. The XOR dataset [2] for example seems like it would be appropriate and provide a sense of how tight the derived bounds really are. Adding something like this is my main recommendation and its absence my main reservation.\n2. On line 98 and 99 the \"less than or equal to some multiple\" operator $\\lesssim$ is defined. This operator appears very loose since $a \\lesssim b$ translates to $a \\leq cb$ for some positive c value. This appears to be the same as saying $a \\leq \\infty$ assuming b is positive. As far as I can see it is used sparingly. However, it is used early on in the main text to define the learning rate which is obviously an important variable and impacts other variables such as $\\kappa$. It is also used for Lemma 6.4 which is used for the global convergence Theorem. What is the effect of this on the tightness of the bounds and perhaps it is necessary to address this.\n3. The learning rate is another point of concern in that some learning rates presented should be unstable, yet no mention is made of this in the text. Specifically if the learning rate must be less than a particular ratio of the top eigenvalue of the hessian to avoid divergence [3]. Such a limitation is not place when defining the learning rate (as mentioned for the second point above) or for Theorem 5.2 where the learning rate is inversely proportional to the loss (and will tend to infinty as a result). I don't think ignoring stability in this case is an issue, however, I again think it should be mentioned.\n\n## Significance\nThe three points under quality above also limit the  significance, or rather limit my ability to determine this work's significance. The main issue again being that it is unclear how tight the bounds really are or how applicable they are without some form of simulation being run. If not such simulation fits all of the assumptions then this would need to be acknowledged. If this is the case I do not view it as grounds for rejection but it is important for a reader to know. That said, this work seems to be a good addition to the theoretical literature and I think the strength outweigh the weaknesses for its significance.\n\n[1] Lakshminarayanan, Chandrashekar, and Amit Vikram Singh. \"Deep Gated Networks: A framework to understand training and generalisation in deep learning.\" arXiv preprint arXiv:2002.03996 (2020).\n\n[2] Brutzkus, Alon, and Amir Globerson. \"Why do larger models generalize better? A theoretical perspective via the XOR problem.\" International Conference on Machine Learning. PMLR, 2019.\n\n[3] Cohen, Jeremy M., et al. \"Gradient descent on neural networks typically occurs at the edge of stability.\" arXiv preprint arXiv:2103.00065 (2021).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H3T9cRPeIZ", "reviewer": "Reviewer_rRrN", "summary": "This paper studies the training dynamics of GD and SGD in mildly over-parameterized neural networks. Specifically, the major result focuses on the early-stage training dynamics and proves that the loss decreases by a significant amount in the early iterations. Global convergence result is also provided.", "questions": "Line 57: Is it \"mildly overparameterized\" instead of \"mildly parameterized\"?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is technically solid. I think the most interesting result is the early-stage result, as it presents a concrete hitting time when the loss descends to a relatively low value. In particular, the hitting time is proved to be small, which matches the empirical observation that the training loss descends rapidly in the first few epochs. Therefore, this early-stage result might be able to explain practical training. The partition technique also looks interesting and may help in understanding the training dynamics of GD in neural nets.\n\nThere are still some weak points in this paper.\n\nI skim the appendix and believe the proof is correct. However, the author seems to assume specific values of $\\eta$, $\\kappa$ and $m$ in the proof. It is better to keep $\\eta$, $\\kappa$ and $m$ in the complete proof to better observe the relationship between the loss bound and these parameters.\n\nThe assumption is still a bit strong. Specifically, the initialization is assumed to be close to 0. This assumption may be not desirable in the following two aspects: 1. It may not characterize the training dynamics in the global region; 2. It may not match practice.\n\nThe settings are not unified in binary and multi-class classification. Specifically, for binary classification, the paper uses the ReLU network without bias; whereas for multi-class classification, the paper uses the ReLU network with bias. It would be better if the settings are changed to the same.\n\nThe comparison with the results in late-stage training (Line 50, 185) looks a bit strange. In my humble point of view, these results focus on the generalization performance of GD in neural networks and are orthogonal to the topic of this paper.\n\nThe writing of this paper could be improved. The partition names in Lemma 6.2 look confusing as “false-dead” neurons are truly dead in the sense that they remain dead, and “true-dead” neurons are not dead as they can turn into true-living. There are also some grammar mistakes in the paper, especially in the singular/plural form and the usage of articles (Line 41, 42, 55, 74, etc.).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept