The Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
Authors: Chao Ma
Lei Wu
Weinan E
Venue:  (2021)
Keywords: Gradient descent
neural networks
implicit regularization
quenching-activation

Abstract: A  numerical and phenomenological study of the gradient descent (GD) algorithm for training two-layer neural network models is carried out for different parameter regimes. It is found that there are two distinctive phases in the GD dynamics in the under-parameterized regime: An early phase in which the GD dynamics follows closely that of the corresponding random feature model, followed by a late phase in which the neurons are divided into two groups: a group of a few (maybe none) “activated” neurons that dominate the dynamics and a group of ``quenched” neurons that support the continued activation and deactivation process.  In particular, when the target function can be accurately approximated by a relatively small number of neurons, this quenching-activation process biases GD to picking sparse solutions.  This neural network-like behavior is continued into the mildly over-parameterized regime, in which it undergoes a transition to a random feature-like behavior where the inner-layer parameters are effectively frozen during the training process.  The quenching process seems to provide a clear mechanism for  ``implicit regularization''. This is qualitatively different from the GD dynamics associated with the ``mean-field'' scaling where all neurons participate equally.

Under review as a conference paper at ICLR 2021
THE QUENCHING-ACTIVATIOO BEHAVIOR OF THE GRADIENT DESCENT DYNAMICS FOR TWO-LAYER NEURAL NETWORK MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
A numerical and phenomenological study of the gradient descent (GD) algorithm for training two-layer neural network models is carried out for different parameter regimes. It is found that there are two distinctive phases in the GD dynamics in the under-parametenn pareeerraaaete regime: An early phase which the GD dynamics fol- lows closely that of the corresponding random feature model, followed by late phase which the neurons are divided into two groups: a group of a few (maybe none) "activated" neurons that dominate the dynamics and group of "quenched" neurons that support the continued activation and deactivation process. In par- ticular, when the target function can be accurately approximated by a relatively small number of neurons, this seanching-activatiin process biases GD to picking sparse solutions. This neural network-like behavior is continued into the mildly over- peee-parameterieed regime, in which it undergoes a transition to a random feature- like behavior where the inner-layer parameters training process. The quenching process seems to provide a clear mechanism for "implicit regularization". This qualitatively different from the GD dynamics associated with the "mean-fied" faendffedd" scaling where all neurons participate equally.
effectively frozen during the
INTRODUCTION
In the past few years, much effort has been devoted to the understanding of the theoretical foun- dation behind the spectacular success of neural network (NN)-based machine learning. The main theoretical questions concern the training process and the generalization property of solutions found. For two-layer neural network (2LNN) and deep residual neural network models, it has been proved that solutions with "good" generalization performance do exist. Specifically, it has been shown that for the appropriate classes of target functions, the generalization error associated with the global minimizers of some properly regularized 2LNN and deep residual neural networks obey Monte Carlo-like estimates: ((1/m) for the approximation error and ((1/vn) for the estimation error, where m and n are the number of parameters and the size of the training set, respectively (Barron, 1994; Bach, 2017; E al., 2019b;a). The fact that these estimates do not suffer from the curse of di- mensionality (CoD) is one of the fundamental reasons behind the success of neural network models in high dimensions.
An important open question is: Do standard optimization algorithms used in practice find good solu- tions? NN-based models often work the over-parameterizes regime where the models can easily fit all the training data, and some of solutions may give rise to large test errors (Wu et al., 2017). However, has been observed in practice that small test error can often be achieved with appropriate choice of the hyper-parameters, even without the need of explicit regularization (Neyshabur et al., 2014; Zhang et al., 2017). This means that there are some "implicit regularization" mechanisms at work for the optimization algorithm with the particular choices of papre--prraeters..
A rather complete picture has been established for highly paeeterraeeeriieed NN models. Unfor- tunately the overall result is somewhat disappointing: While one can prove that GD converges to a global minimizer of the empirical risk (Du et al., 2019b;a), the generalization properties of this global minimizer is no better than that of associated random feature model (RFM) (Jacot et al., 2018; E et al., 2020; Arora et al., 2019). In fact, E et al. (2020) and Arora et al. (2019) proved that the entire GD paths for the NN model and the associated RFM stay uniformly close for all time.

---

Under review as a conference paper at ICLR 2021
A natural question is then: Can there be implicit regularization when the network is less over- parameterized? What would be the mechanism of the implicit regularization? More generally, what is the qualitative behavior of the GD dynamics in different regimes including the under- parameterized regimes? In this paper, we provide a systematic investigation for two-layer neural networks by well-designed experiments. Our objective is to get some insight from this kind of ex- perimental studies, which we hope will be helpful for subsequent theoretical work. Specifically, our findings are summarized as follows.
It is observed that when the network less pver-parameterized. the GD dynamics exhibit two phases. During the first phase, GD follows closely that of the corresponding RFM. Afterwards, GD enters the second phase where neurons form two groups (the first group might be empty): a group of activated neurons and group of quenched neurons. Depend- ing on the target functions, the quenched neurons can exhibit continued quenching and sparse activation processes. In partieular, if the target function can be well approximated by a small number of neurons, GD is biased to picking sparse solutions.
Based on these observations, we then investigate how the extent of over-parameterization affects the generalization properties of GD solutions. We find that the test error shows sharp transition within the mildly over-parameterized regime. This transition suggests that implicit regularization is quite sensitive to the change of the network width.
Lastly, we study 2LNNs under mean-field scaling (Chizat & Bach, 2018; Mei et al., 2018; Rotskoff & Vanden -ijnden,, 2018; Sirignano & Spiliopoulos, 2020), i.e. with an extra 1 1/m factor added to the expression of the function, where m denotes the number of neurons. We observe that in this case all the neurons contribute pretty much equally and the test performance is much more robust to the change of network width.
2 PRELIMINARIES
2.1 TWO-LAYER NEURAL NETWORKS
Under conventional scaling, a two-layer neural network model is given by:
Sm (x;; a, B) = Σ (3 x)) = a" a (Ba),,
(1)
where R" (bI, b2 ..... )T Rmxd and ott) max(0, t) is the ReLU activation function. Later we will consider the mean--fill scaling where the expression above is replaced by
1 Σ
1 a. (b x) _ (Bw),
fm( a, B)
(2)
but we will focus on the conventional scaling unless indicated otherwise. As a comparison, the random feature model is given by fm (zx; a. Bn), where only the coefficient can be varied; Bo is randomly sampled and is fixed during training.
S {(x, V4 f* denote the training set. f* is the target function. We assume that {æx} } are drawn independently from To, the uniform distribution over sd = fo Rdd |||||| 1}.
The empirical risk and the population risk are defined by R, (a, B) 1 " Ei (2 a, B)
(xx))) and R(a, B) Eg (( (x;; a, B) (=))))), respectively.
Following the study of the function space for two-layer neural networks (E et al.. 2019e; Bach, 2017), we will focus on target functions of the form f*) = Eo [a (b) x)] with " being a probability distribution over 1d-1 The population risk can be written as:
R(a, B) = En
ajo(b, x) - -Eb-- (aa(a) x1
j=1
Σ
an haa 1 b,,)) - Σ
Bow (b)k:bb,,
J1.
1=1
EArT BB (a (b)a² a (a'"((b''))) (b')k(b, b'),,
(3)
2

---

Under review as a conference paper at ICLR 2021
where k(b, B') |]]|||6||| (sin 0 + (T 0) cos 0) with & arccos((b, (Cho & Saul, 2009).
Since our primary interest is to gain some insight about the behavior of the training dynamics in different parameter and scaling regimes, we choose to work with idcalized setting with simple synthetic target functions instead real data sets. Real data sets have extra complicating factors that prevent from focusing on partieular aspects of the training process. In fact most of our efforts will be devoted to target functions of finite neurons or that can be accurately approximated by a relatively small number of neurons, similar the widely used teacher-student setting (Saad & Solla, 1995; Tian, 2017; Safran & Shamir, 2018; Goldt et al., 2019). However, we will also discuss one example for which this is not the case.
2.2
GD DYNAMICS FOR TWO-LAYER NEURAL NETWORKS
We consider the most popular initialization (LeCun al., 2012; He et al., 2015),
a,(0) ~ (0 3", (0) ~ 1//1
(4)
For the original LeCun initialization (LeCun et al., 2012), 1/ym. However, we have found consistently that for 2LNNs the behavior of the GD dynamics is qualitatively very close to the case when 3 0. We refer Appendix A for some numerical results along this line. For simplicity we will focus on the case when 3 = 0.
Du al. (2019b) proved that GD converges exponentially fast to a global minimizer in the highly over-parameterrzze regime. Subsequently it was shown in (E et al., 2020) that these GD solutions are uniformly close that of the associated RFM with Bg- = (0) as the features:
Theorem (Informal), Lef @(t) denote the GD solution of associated RFM. For any 8 (0,1), if
m 2 (oly(m,,og(1/))) 3 = O, with probability 1 1 ó we have (x; a(t), Bo)l poly(n, log(1/6))
sup mggd-11rg (0,00)
al;t), a
(5)
Vm
The key observations for this case is the time scale separation: (t)
O(1b, () = O(1) and
(t) O((ay)) O (poly(n)/m) /m). In the highly wer-parameterizee regime, the dynamics of b, is effectively frozen. GD for the 2LNN degenerates to the GD for the corresponding RFM.
There are three important large parameters: m,n, d. They are the number of neurons, the number of training samples, the input dimension, respectively. There are two obvious extreme situations that are interest. One when m >> n. This was described above in Theorem and relatively well understood. The other is when >>> which will be investigated by considering n co. The regime when m ~ n or m ~ n//d+ + 1) are also of interest since these are regimes where the "resonance" (or the closely related "double descent") phenomena might occur, as we learned from previous work on RFM (Advani & Saxe, 2017; Belkin et al., 2019; Ma al., 2020).
GD DYNAMICS FOR THE CASE WITH INFINITE SAMPLES
As a starting point, we first investigate the GD dynamics for the population risk, i.e. n = oc. We will see later that the phenomena revealed here are indicative of the neural network-like (NN-like) behavior for GD dynamics.
3.1 SINGLE NEURON TARGET FUNCTION: NEURON ACTIVATION AND DEACTIVATION
First we look the case where the target function a single neuron:
(xx = (bb x.),
where = Ej. The results are presented Figure 1. One can observe several interesting features about the GD dynamics.
(1) Initially, the GD dynamics for the 2LNN is close to that of the corresponding RFM. (2) The GD dynamics for the 2LNN and RFM depart from each other around the time when the loss function for the RFM starts to saturate. (3)The converged solution for the 2LNN is very sparse. In fact only one neuron contributes significantly to the model in this experiment.
3

---

Under review as a conference paper at ICLR 2021
NN
10
T-20000
RF
8
NN
0.
2011
1000
1006 10:00 2000
10000
2000
Number Iterations
Index
Namber of Iterationa
6.
(a)
(b)
(C)
(d)
Figure 1: The dynamic behavior of learning the single neuron target function. The results of RFM are shown comparison. Here. 200, 100 and the learning rate is 0.001. (a) The dynamics of the population
risk. (b) The a coefficients of the converged solution. (c) The dynamies of the magnitude of each neuron, where the magnitude is defined bll. (d) The projection to the first two coordinates of b. The green and orange ones correspond to the initialization and GD solution, respectively.
We also see that there are two phases in the GD dynamics. In the first phase, the dynamics follows closely the GD dynamics for the RFM. In the second phase, the outer layer coefficients a small except for one neuron. In the transition from the first to the second phase, except for one neuron, all the other neurons are "quenched" in the sense that their outer layer coefficients a keep decreasing. Thus, the dynamics of their inner layer weights b become very slow. Consequently, as shown in Figure ld, only one neuron is "specialized" to the teacher b" (Saad & Solla, 1995), and the inner layer weights of most neurons have hardly changed. The same behavior was observed with other realizations of the initial data, except that the number of "activated" neurons can be different. But in all cases, we always observe few "activated" neurons in the second phase.
3.1.1 CIRCLE NEURON TARGET FUNCTION: MULTI-STEP PHENOMENON
Next we consider a more sophisticated target function: (x) En-ao lo(b" x)), where To is the uniform distribution over the unit circle | (b E Rd of + by = and by = vi > 2}.
10
10
Namberr (enmroTe
Hutfoer of aamoone
Index neurons (C)
(a)
(b)
Figure 2: The dynamic behavior of learning the circle neuron target function. Here 100, d 100 and the learning rate is 0.005. (a) The dynamics of the population risk. (b) The dynamics of the magnitude of cach neuron; the inset is the zoom-in of the first 30000 iterations. (c) The a coefficients of the solutions selected to represent the three steps.
typical dynamic behavior of the population risk is shown in Figure 2a. We see that there are still two phases. But in the second phase, the population risk decreases in a "step-like" fashion. To see what happens, we plot the dynamies of the a coefficient of each neuron in Figure 2b. We see that after the first phase, most of the neurons start to die out slowly (see the inset of Figure 2b). As the GD dynamics proceeds, few new neurons are activated from the "quenched neurons". This activation process can be very slow, and the loss function is almost constant before activation actually happens. The activation process is relatively fast and causes a fast decay of the loss function. Figure 2c shows three representative solutions for the three steps. We see that from the first and second step, two more neurons are activated. From the second to the third step, one more neuron pops out.
3.1.2 FINITE NEURON TARGET FUNCTIONS
The observation that GD dynamics picks out sparse solutions and the associated quenching- activation behavior happens in more general setting: learning finite neurons. Consider the target function (x) = Eyn ob) x.).
4

---

Under review as a conference paper at ICLR 2021
We are interested in the case: m > m.. The single-neuron target func- tion is special case with m² 1. Figure 3 shows the dynamic behav- 50, m 40. In this
no ml
ior for case m is the same order of m², but we still see the GD dynamics tends to find solutions with the number of ac- tive neurons close to m". The learn- ing process is qualitatively similar to the case of circle neuron target func- tion except that the activation process proceeds in a more continuous fash- ion and therefore the step-like behav-
-116
-0.0
10
20
60
nouron
iterations
Figure 3: The dynamic behavior of learning finite-neuron target function. Here, of = 1/m, (b} } are uniformly drawn from gd-1 with d = 100 and the learning rate is 0.001. Left: The magnitude of cach neuron of the convergent solution. Right: The dynamics of the magnitude of each neuron.
ior is less pronounced. This trend becomes clearer m is much larger than m See Appendix B for more experiments.
3.1.3 SURFACE NEURON TARGET FUNCTION
The target functions studied above are all functions that can be accu- rately approximated by small set of neurons. Next we consider an exam- ple in the opposite direction: the "sur- face neuron target function"
16P
RF
1371
0.00
0.88
(x) = Eb (a x]]]
415
11.
2.11 1.5
6.5
L.
Number Ranations
Number ituratioon
where T3 is the uniform distribu-
tion over Ω = {b Σ 4/2
-i Dy
Figure 4: The dynamic behavior of learning the surface neuron target function. Here m 100, 100 and learning rate 1) = 0.005. Left: The dynamics of the population risk; Right: The dynamics of the magnitude for each neuron. The inset is the zoom- in of the first 500, 000 iterations.
and = 0, Vi > d/2). This func- tion is represented by very large set of neurons, each which contributes an equal small amount. Shown in Figure 4 are the numerical results. One can see that in this case all the neurons quenched and there no activated neurons. The activation process is replaced by smooth changes of all the neurons.
3.2 UNDERSTANDING THE SECOND PHASES BY USING EFFECTIVE DYNAMICS
In the second phase, the neurons be divided into groups (the first group might be empty): I1 and 112, where 11, 1l2 are the set of indices of active and quenched neurons, respectively, We have aj = (1) for j In and a, (1) for 1g. Denote by fI, (æ; t) EJer, (b) (1)z)) with i i= 1,2 the functions represented by the active and quenched neurons, respectively.
The dynamics of neurons for the two groups are very different. For j E Tr, both aj and by changes significantly. However, for lll,, as evolves much faster than by since â, ((I
0(1),b; ~ O(Ia,)) & As result {az (aibier, is effectively slaved to the optimal solutions with (bybee,, held fixed, which is given by
argmin Ex(( Σ (ajobb (1)z) + (z; (z))
(6)
jeL, In this way, one obtains an effective dynamics governing the second phase: For active neurons, i.e. j E I,
(t EEl(f(zzt) f (z) (t)x)]
b,(t) (a (f)=)=);
for quenched neurons, i.e. Tz.
ay(t) (t)
(8)
6,(t) = -Eol(f(x;t) (6
5

---

Under review as a conference paper at ICLR 2021
where (t) is the solution of (6), and ((x;t)=] a is the functions repre- sented by the solutions of the effective dynamics at the time t.
We numerically test how accurate the effective dynamics is for two target functions: single neuron and surface neuron. For surface neuron, I1 is empty. Shown in Figure 5 are the results. We see that indeed the effective dynamics is able to capture the original dynamics very well, even at relatively long-time scales (see Figure 13 in the appendix).
100
t-20
42
500
t-4020
04
a(()
t-00
t- t-999990
(x)
000
www WV
Figure 5: Comparison between a(t) and a aff (t) at different times, where (t)), a (t) are the solutions of the original and effective dynamics, respectively, Left:: single neuron target function fi. Right: The surface neuron target function få-
GD DYNAMICS FOR THE CASE OF FINITE TRAINING SAMPLES
We now turn to the more realistic situation where the size of the training set is finite.
4.1 THE DYNAMIC BEHAVIOR
We refer the reader to Appendix D for some numerical results of the highly over-parameterized regime well the under-parameteriize regime, i.e. n/(d + 1). We find that in the under- parameterized regime, the dynamic behavior of GD is qualitatively similar to the case co.
We call the regime between the highly parepaaraeteri and under-parameterized regimes the mildly over-aarameterized regime. The key question of interest is how the mildly over- parameterized regime bridges the highly over-parameterized regime and the ander-parameterizee regime. We have already seen that these two regimes differ in several aspects. One is that in the highly pverpparameterized regime, the inner layer coefficients barely change. the under- parameterized regime, small number of neurons experience large changes for their inner-layer coefficients.
First, we investigate the GD dynamics for two interesting scalings: = cn/(d + 1) and m Cn where e and C are constants. Shown in Figures 6 are two examples for m 3n/(d + 1) and 0.75n with 200, d = 19 respectively. More examples with the same scalings but different values of n can be found in Appendix D.
Derations
teralions
terations
(a) 3n/((
(b) m 0.75m
Figure 6: The dynamic behavior of learning single-neuron target function for mildly over- wve-parameterized cases. Here n 200, d 19 and learning rate is 0.001. For each case, Left: The dynamics the training and test losses (results from the corresponding random feature model is also plotted for comparison); Right: The dynamics of the magnitude of cach neuron.
One can see that the behavior shown in Figures 6a resembles the ones shown for infinite samples, whereas the behavior shown in Figures 6b does not. In the first case, the test accuracy improves substantially after the GD dynamics departs from that of the RFM, and there is a notable presence of the activation phenomena. In the second case, the test error saturates after the GD dynamics

---

Under review as a conference paper at ICLR 2021
departs from that of the RFM, and there clear presence of the activation phenomena. We will call the first case "NN-like" and the second case "RF-like".
4.2 GENERALIZATION ERROR AND THE PATH NORM
Next we examine the generalization error of the GD so- lution. Shown in Figure 7 are the test errors of GD so- lutions with varying m's 's and n's. The target function is the single neuron target function ff. We see that the test errors exhibit a sharp transition somewhere between m = n/(d + 1) m n. This is consistent with the observation in Figure 6 that when m ~ n/(d 1), GD shows a NN-like behavior while like behavior. Moreover, the transition seems to become sharper with increasing values of and m.
Test errors
-0.6
2.6
-1.8
-2
logho(n)
n, GD shows RF-
-3.6
2.0
4.2
-5.4
In the following, we take a closer look at how the in- crease of affects the test performance by consider- ing the circle neuron target function. The results for the single-neuron target function are similar, and can be found in Appendix E. For 2LNNs, the generalization gap can be controlled by the path norm of the parame- ters (Neyshabur et al., 2015; E et al., 2019b), defined by ||@||p = ΣØ= ررا |, ||2- Therefore we will compute the path norm of the parameters selected by GD.
6.0
2.0
log1a(m)
Figure 7: Heatmap test errors of GD so- lutions with varying m''s and n's. The target function is the single-neuron target function fi with d 40. GD is stopped when the training loss smaller than 10 The two dashed lines corresponds to m = n/(d +) and m = n, respectively.
Figure 8a examines the test error and the path norm as changes. One can see that as m becomes larger, the test error of the NN model eventually becomes close that of the RFM. One thing to notice is that these changes seem to behave smoothly across the points: m = n/(dd + 1) and m = n, where the NN and the RFM change from an pnder-pprameetrrized situation to over- over-varamenn wer-parameterized situation, respectively. We do not observe the peak of test errors around m = n/(d + +) as suggested in (Belkin et al., 2019). We suspect that the peak observed in (Belkin al., 2019) is the result of the special training method used there. Interestingly, Figure 8 and Figure 7 show that there does exist peak around m n, the same place for the RFM. This should be the result of the close proximity of the GD dynamics for 2LNN and RFM during the first phase. The latter performs extremely badly when m n due resonance shown in (Ma et al., 2020). Thus it is not surprising to see that NÑ also performs the worst around
In addition, the path norm seems to serve as a good indicator of the generalization performance. For example, one can see dramatic increase of the path norm from m n/ d 1) to m n, and the path norm peaks around m n.
mr 2nvid+1)
m=1.5n
-0.5
lest arorg
-1.0
path norm
oe (tat emor)
-2
logui enrr))
-1.5
-1.5
d-10
-20
-3
-200
d+100
MN RF
-3.0
-3.5
2 l0920/
Iogrs(wath))
2.0
log
20
lognti))
255
3.0
(b)
Figure 8: Test performance of GD solutions for the circle neuron fz. (a) The path norm and test error as function of m. Here n 200, d 20 and the learning rate is 0.001. GD is stopped after the training error smaller than 10 6, (b) The test error as function n three different input dimensions.
To further explore the difference between the two scalings n2/dd + 1) and Figure 8b the test error as a function of the size of training set. The result suggests that under the scaling m n, the test error may suffer from curse of dimensionality (CoD), while for the scaling
show in
7

---

Under review as a conference paper at ICLR 2021
n/(d++ 1). it does not to be the case. Let the test error decrease O(1/no(d), CoD means that ad)) decreases (to 0) as d - oo. Thus, it is difficult to learn the target function when large if the model suffers from CoD.
GD DYNAMICS WITH
SCALING
For simplicity, we call the GD dynamics of the scaled 2LNN (2) GD-MF and the GD dynamics of the unscaled 2LNN (1) GD-conventional. Under the mean-field scaling, 0j and by are of the same order. Thus áj and b, are also of the same order, which means that there is no time-scale separation.
First Iet us look at the case when n Figure 9 shows the dynamic behavior of GD-MF for single neuron target func- tion f. Different from GD-conventional shown in Figure I, we see that almost all the neurons move significantly and con- tribute to the model.
20
GD MF
T-35000
-10
Figure shows the error for the case with finite sample. We see that the test er- ror GD-MF varies more smoothly with the increase of network width. There is al- the perfor-
Index of neurons
6y100
Figure 9: GD-MF dynamics for learning the single neuron target function fi. Here m 50, d 100, learning rate is 0.001. Left: The magnitude ron for the converged solution. Right: The projection to the first two coordinates correspond to the random initialization; the orange ones cor- respond to the solutions found by GD-MF.
and neu-
most no clear deterioration mance even when we increase the network width to the highly over wver-parameterized regime. This is clearly different from the case for GDDconnentional. Moreover, we observe that the GD-MF solutions do not seem to suffer from CoD at the regime m ~ n, which is not the case for GD-conventional (see Figure 8b).
for each neuron. The green ones
Test errors
1.44
-1.0
-18
MF Conventional
-224
-1.5
(tes mmo
d=100
-2.
-25
Iesy/test (testt
-3
-384
-3.0
424
18
-3.5
4.
3
2.
2.5
3.0
logao(m)
logig(m)
lognf(n)
Single neuron fr.
(b) Circle neuron fz-
Figure 10: (a) The heatmap of test errors GD-MF solutions for learning the single neuron fi (b) The test performance of learning the circle neuron f2 The left shows the test error as a function of m. The right shows the test error as function of for the scaling 1.5m.
CONCLUSION
The experimental results shown in this paper suggest the following: (1) Under the conventional scal- ing, both the training process and the generalization performance are quite sensitive to the network parameters. With the mean-field scaling, both more stable, though not always better. (2) In the NN-like regime under conventional scaling, the quenching process provides the mechanism of im- plicit regularization, since it does not allow the the path norm grow out of control. Consequently the generalization gap is controlled.
Obviously, many questions remain open. For example, how do the results discussed in this paper manifest for practical datasets and deep neural networks? Can we substantiate some of the findings by rigorous results? These are all questions that left for future work.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXiv:1710.03660 2017.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang- On exact computation with infinitely wide neural net. In Advances in Neural Information Processing Systems, pp. 1139-8148, 2019.
Francis Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Ma- chine Learning Research, 18(19)11-53 2017.
Andrew R Barron. Approximation and estimation bounds for artificial neural networks. Machine Learning, 14()):115-333, 1994.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical biri-cari trade-off. Proceedings of the National Academy of Sciences, 116(32):15449-1588 2019.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In Advances in neural information processing systems, pp. 3036-3046, 2018.
Youngmin Cho and Lawrence K Saul. Kernel methods for deep learning. In Advances in neural information processing systems, pp. 342-342-243 350, 2009.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pp- 1675- 1685, 2019a.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference Learning Representations, 2019b.
Weinan E, Chao Ma, and Qingcan Wang. A priori estimates of the population risk for residual networks. arXiv preprint arXiv: 090..02554, 2019a.
Weinan E, Chao Ma, and Lei Wu. A priori estimates of the population risk for two-layer neural networks. Communications in Mathematical Sciences, 1755):447-1145, 2019.
Weinan E. Chao Ma, and Lei Wu. Barron spaces and the compositional function spaces for neural network models. arXiv preprint 1xXiv:1906.08039 2019c.
Weinan E, Chao Ma, and Lei Wu. A comparative analysis of optimization and generalization prop- erties of two-layer neural network and random feature models under gradient descent dynamics. Science China Mathematics, pp. 1-24, 2020.
Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dy- namics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. In Advances Neural Information Processing Systems, pp. 6981-6991, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1036-1034, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Advances in neural information processing systems, pp. 8580- 8589, 2018.
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Chao Ma, Lei Wu, and Weinan E. The slow deterioration of the generalization error the random feature mode In Mathematical and Scientific Machine Learning Conference, 2020.

---

Under review as a conference paper at ICLR 2021
Song Mei, A Montanari, and P Nguyen. A mean field view of the landscape of two-layers neural networks. In Proceedings of the National Academy of Sciences, volume 115, pp. E7665-E7671, 2018.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprini arXiv: 1412.6614, 2014.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.
Grant Rotskoff and Eric Vanden-Eijnden Parameters as interacting particles: long time convergence scaling of neural networks. In Advances in neural information processing systems, pp. 7146-7155, 2018.
and asymptotic
David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52 (4):4225, 1995.
Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer ReLU neural net- works. In International Conference on Machine Learning, pp. 4433-4441, 2018. 2018
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 130(3):1800-1855 2020.
Yuandong Tian. An analytical formula of population gradient for two-layered ReLU network and its applications in convergence and critical point analysis. In International Conference Machine Learning, pp. 3404-3413, 2017.
Lei Wu, Zhanxing Zhu, and Weinan E. Towards understanding generalization of deep learning: Perspective loss landscapes, arXiv preprint arXiv:1706. 10239, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
10

---

Under review as a conference paper at ICLR 2021
A THE INFLUENCE OF THE INITIAL MAGNITUDE OF THE OUTER COEFFICIENTS
First we consider the case / B= 1/11/m 1/ym. m. Figure 11 shows the numerical results with the same setting as in Section 3.1. We that the dynamical behavior is qualitatively the same as the case of B 0.
For 3 = 1 1/m² with 7 2 1/2, the time-scale separation between the inner and outer layers always hold initially as long as is large enough. In these cases we basically the same kind of behavior as shown in the main text. To simplify the experiments and the presentation, we set 3 = 0, i.e.
y 00, in which case do not need to take very large values of
8 044
13
300
100
1800
1000
Namber horatoon
Vodes neasss
(h)
0.1
30
-12
2003 2000
300
2.00
1300
3000
2010
080 7000
Namber itaraciona
Number of Hteraoona
(c)
(d)
Figure 11: The dynamic behavior of GD with و = 1/ym. Here the target function is the single neuron fi. m = 200, d 100 and learning rate is 0.001. We also plot the results of RFM comparison. (a) The dynamic behavior of the population risk; (b) The magnitude of each neuron for the converged solution. (c) The dynamics of the coefficient of each neuron. (d) The dynamics of [8,[0]], the projection of by to b" = ei.
B ADDITIONAL RESULTS FOR THE FINITE NEURON TARGET FUNCTION
Figure 12 shows additional results for learning finite neurons when is much larger than m". We see that the behavior is basically the same as Figure 3 except that more neurons concentrate around
100, m 40
100,
0.10
0.04
0.06
0.022
0.00
-0.05
0.00
-0.10
20
60
100
0.0
0.5
1.0
1.5
2.0
Index of neuron
Number of iterations
40 and learning rate 0.001. Left: The magnitude of each neuron of the final solution; Right: The dynamics of the magnitude for each neuron.
Figure 12: Learning finite neurons with a; 1/m, b ~ no.
Here m = 100, m
11

---

Under review as a conference paper at ICLR 2021
C ADDITIONAL RESULTS FOR THE EFFECTIVE DYNAMICS
In Figure 13, we plot the difference between the effective dynamics and original dynamics as function of number of iterations. We see that the two dynamics are pretty close in the second phase even in long time scale.
70
1011
(alf) /la///l
10-2
B
10-2
10-3
10-4
[BC
S0llle/ (BB])
10200
20000
3000
2001
200000
000000
Figure 13: The difference between the effective dynamics and the original dynamics. Left:: The single neuron target function fí with m 50; Right: The surface neuron target function fi with m = 50.
D ADDITIONAL EXPERIMENTAL RESULTS FOR GD DYNAMICS WITH FINITE SAMPLES
D.I THE PNDEE-PARAMETEERIZE REGIME
We first look at the regime where the network is under-paree paader-paramttrizedd, i.c. m < mn/(d+1) A typical result for the single neuron target function is shown in Figure 14. We see that overall the qualitative behavior of GD is similar to that of the case when n oc studied in Section 3. There are still two phases, and the convergent solution is sparse. Figure 14b shows that the quenching process becomes slower compared to the case with infinite data.
m-200-22
meo. n-000 d-20
mein
n1
NV, sest
RI trai
LO55
02
Numbber bendiiote
Indox cinnerons
(a)
(b)
(c)
Figure 14: GD dynamics for learning the single neuron target function in the under--arramtterized regime. Here m = 9, df = 20, n = 200 and the learning rate л) = 0.005. (a) Time history of training and test errors. (b) Time history of the magnitude for each neuron. (c) The magnitude of each neuron of the convergent solution.
Figure 15 shows the result for the circle neuron target function. We see again that there are still two phases. But the multi-step phenomenon becomes less pronounced compared the infinite data case. From Figure 15b, the neurons can still roughly be divided into two groups: active and quenched ncurons. Also we still observe some activation during the second phase, although this process is much more smooth compared to the case with infinite data.
D.2 THE MILDLY OVER PEEE-PARAMTTERIEED REGIME
Here provide more experimental results under the two scalings n/(d 1) and m n. The results are shown in Figure 16 and 17. We see that the dynamic behavior similar to the ones in Figure 6.
D.3 THE HIGHLY OVER PARAMETERIZED REGIME
We first look at the simple situation where the network is highly wer-parameteriize. Figure 18 shows some typical results in this regime. Clearly in this regime, the GD dynamics for the NN model stay uniformly close that of the RFM for all time, as proved in E et al. (2020). We
12

---

Under review as a conference paper at ICLR 2021
-01
34
05
28
20
00
93
26
31
Figure 15: GD dynamics for the circle neuron target function in the under- auder-parameterizer case. Here m 50, n = 500, d = 5, (a) The dynamics of training and test loss. (b) The dynamics of the outer coefficients.
1-400
mu50. nw402 dx20
Iain
KN IcSt
RE, train
RF
2000
Number of barallore
21000
500
(b)
Figure 16: The dynamic behavior of the GD solutions for 3n/(dd + 1). Here and learning rate = 0.001.
60, n 400, 19
m-300 1-400 d=20
8:00
NN ta
Lo05
NN ses1
RF trah
RF
Number of baralions
Number of ootrriaoons
(a)
(b)
(c)
Figure 17: The dynamic behavior of the GD solutions for 0.75n. Here 3000 400, d 19 and learning rate 17 0.001.
see that the training error goes to exponentially fast, but the testing error quickly saturates. Note that Theorem 1 suggests that the network width should satisfy m 2 n² Xx In(n" (0 8-1,, but in this experiment m 0.5m² is enough.
Ti-200, n-200
70-0000 n-200
16-1
m
tnsesst rt tra
rioos:
001
10
1000
1000
2000
1050
2300
Iltrations
27 Dourtee
Figure 18: Comparison between the GD solutions for 2LNN and RFM for the single neuron target function ft The learning rate 0.001 and m 2000, n 200, = 20. Left: The time history the training and test error. Right: The magnitude of the converged solutions.
13

---

Under review as a conference paper at ICLR 2021
E ADDITIONAL RESULTS ON THE TEST PERFORMANCE
Figure 19 shows how the test error is affected by the number of samples n and the number of neurons m for the single neuron target function.
m-2n(d-1)
m=1.5n
12
10
4-10
llg ero)
10
d-40
pan n0m7
10
d-100
Teat artor
10"
13
10
10"
RF
2
3
logth/Withh)
logue(widhh)
Number
Number
solutions for single neuron fi (a) path norm and test error as function of m. Here n 200, = 20 and the learning rate is 0.001. GD is stopped after the training error is smaller than 10t (b) The test errors as function of n for three different input dimensions.
Figure 19: Test performance
14


Reviews:
[{"review_id": "yOeMIj_-Ld6", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**SUMMARY** The authors perform an empirical study of the dynamics of gradient decent for single-hidden-layer NNs with ReLU activations on some synthetic datasets under L2 loss. They identify two distinct phases: an early phase matching RF models and a late phase where the neurons split into activated and quenched groups. When the target function admits a sparse representation, the learned function can have a sparse activation pattern, which is interpreted as a form of implicit regularization.\n\n**PROS** The authors identify a mechanism of quenching and sparse activation patterns that should be of interest to researchers studying the generalization properties of NNs. The mechanism seems sensitive to the data distribution and the network width \n\n**CONS** My main concern is about the robustness of the results. Several simplifying or arbitrary choices are made, which while necessary for theoretical analysis, should be analyzed more critically in an empirical paper. Specifically, the results are restricted to a simple single-hidden-layer architecture, ReLU activation functions, and a simplistic, synthetic data distribution. \n\n **RECOMMENDATION** In its current form, I vote for rejecting the paper. While there are some interesting observations, the uncertainty about their robustness and the absence of a theoretical explanation weaken the paper. Addressing either of these shortcomings would improve the paper.\n\n  **ADDITIONAL QUESTIONS** \n- The scaling for the first-layer weights b seems odd and nonstandard. Since the data are points on the unit sphere, the scaling I/d means the pre/post-activations are little-o 1. Is this intentional? What happens if the b_ij are order 1?\n- How are \\mathbb{I}_1 and \\mathbb{I}_2 determined in Sec. 3.2 and Fig. 5?\n- How correlated are the test error and the path norm? In Fig. 8 they look related, but it’s difficult to tell how much.\n- Is the step behavior in Fig. 2 related to Ghorbani, Mei et al. 2020?\n\n**MINOR COMMENTS**\n- Some additional motivation for f* when pi* is uniform on the unit circle would be helpful.\n- Consider including some additional citations for the random feature model, particularly in Sec. 4.1: Mei and Montanari 2019; d'Ascoli, Refinetti et al. 2020; Adlam and Pennington 2020. They study the effects of under/overparameterization on the generalization error.\n- I also think the overall structure of the paper could be improved to support the main claim the paper is making. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lp_RdmZQb4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper experimentally identifies a phenomenon in which the participation among the neurons in a two-layer neural net trained with gradient descent (GD) changes with time. In particular: \n- The paper shows for certain artificial target functions with low-dimensional structures, the neurons start out from a random-feature behavior and then a few are selectively activated, distinct from the rest of the neurons which are “suppressed”. \n- This behavior is shown to be pronounced when the network is less (or mildly) over-parameterized and suffers less from the curse of dimensionality (CoD). It is almost unobservable when the network has width m exceeding the training size n and suffers more from CoD. \n- Lastly the paper argues that this behavior is specific to the choice of scaling, by showing that under the mean field scaling, the network displays much less sensitivity to the scale of the width.\n\nLarge-width neural networks have been a central object of studies for the past few years. With certain scalings (typically collectively referred to as NTK scalings), as width goes to infinity, the network becomes “linearized”. This is probably among the few works that studies how the network under NTK scaling can behave beyond the linearized regime. In particular, while most of these few works concern with the mathematical question of “how much” the network can learn better than linearized methods, this work identifies a novel mechanism on “how different” the network’s behavior is from linearized methods. I’m unaware if the quenching/activation mechanism has been reported before. In this sense, the result of the paper has an interesting position in this literature.\n\nThe measurement of a neuron’s participation via the product of the two weights is interesting and elucidative of the use of the path norm. The presentation is also clear and easy to follow.\n\nThe paper however has several shortcomings. In no particular order:\n\n1) The target functions are idealized, and no experiments are reported on more realistic settings. Is the quenching/activation mechanism a common phenomenon with real life datasets? Are the transition boundaries n/(d+1) and n universal? We observe from Fig. 4 that no clear quenching/activation process takes place, so it’s questionable whether it can be observed for a difficult and highly diverse real life dataset. A suggestion is to perform experiments on MNIST, CIFAR-10 and CIFAR-100 (which might resemble the single neuron, circle neuron and surface neuron examples).\n\n2) The paper may consider having a discussion on recent works (e.g. [1, 2]) that consider linearized behaviors in the regime where m depends mildly on n/d or is almost independent of n, and whether they may conflict with the result in the paper.\n\n3) The paper gives no explanation or heuristic to understand how the quenching/activation phenomenon arises. In particular, can we know in advance which neuron is going to be activated, even in the simple case of single neuron target function? The paper mentions “The same behavior was observed with other realizations of the initial data, except that the number of activated neurons can be different.” Does it mean that it is entirely possible, although unlikely, that all neurons are activated? If the size of the activated neuron set is highly unpredictable, it would limit usefulness of the effective dynamics.\n\n4) It is unclear how to make the effective dynamics a useful tool. It looks like the switching time (at which we switch from the first phase to the second phase) and the sets I1 and I2 have to be observed a priori from the experiment with the full dynamics. Is there an understanding on simple properties (e.g. how large) of these quantities?\n\n5) In the derivation of the effective dynamics, it is unclear why the second layer’s weights of the quenched neurons move faster than the first layer’s weights. We only know from Theorem 1 that such timescale separation occurs when the network is random-feature-like, but we do not know if it is so during the quenching process in the mildly over-parameterized and under-parameterized regimes.  One can actually observe from Fig. 1 that most neurons are still not quenched when the network departs from the random-feature behavior.\n\n6) The paper suggests that due to quenching of most neurons, the path norm remains small since it is essentially a sum over only activated neurons. However the circle neuron target example shows that the population risk continues to decrease once a neuron emerges from the quenched state and becomes activated. Does this somehow violate the path norm intuition? How does the path norm look like through time?\n\nUltimately I find the finding interesting, but the paper leaves a lot of questions unanswered. Still I would encourage the authors substantiate the paper with more analyses and findings, empirical or theoretical, since this is quite a novel finding.\n\n[1] Polylogarithmic width suffices for gradient descent to achieve arbitrary small test error with shallow relu networks, Ji and Telgarsky, ICLR 2020.\n\n[2] Neural networks Learning and Memorization with (almost) no Over-Parameterization, Daniely, 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7qY6jOeyfcT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the gradient dynamics of two-layers neural networks. It is empirically shown that, for a specific type of initialization, for less over-parameterized neural networks, the gradient dynamics follows two phases: a phase that follows the random features model where all the neurons are \"quenched\", and another phase in which there are a few \"activated\" neurons. This type of behavior of gradient dynamics is different from that of the mean-field regime. \n\nThis paper is well-written. The quenching and active phenomena is interesting. Existing theory characterized the behavior of neural networks in the mean-field/neural tangent regime, and this paper studied a regime that is different from the existing ones. The authors used the effective dynamics to explain the second phase of the dynamics, which gives some good intuitions. \n\nThere are the following issues of this paper. \n1. This paper did not distinguish between \"random features model\" and \"neural tangent model\", although they are closely related. In particular, Theorem 1 presented in this paper is connected to the \"random features model\" as in [E, et.al. 2020], which does not really give the \"neural tangent kernel\" in [Jacot, et.al. 2018] and [Du, et.al., 2019b]. The difference is that, \"random features model\" only the linearization of the top layer, while the \"neural tangent kernel\" also contains the linearization of the bottom layer. Although they are closely connected, it is important to clearly point out the distinction. Moreover, the \"neural tangent\" scaling is a different scaling (different than both MF and RF scaling), where there is a 1/\\sqrt{m} coefficient in front of the network. I think at least the authors should discuss what happens for the \"neural tangent\" scaling when it is less over-parameterized. \n2. This is a pure empirical paper and the phenomenon has no justification (even heuristic mathematical justification). It would be good if the authors can (heuristically) explain which neuron will become activated (any random neuron, or a neuron that satisfies some initial condition?), and how long it will take for those neurons to become activated (perhaps this depends on the initial weights of the neurons). Without these justifications, I feel that the paper is less complete. (In comparison, the MF and NT results are rigorously established. )\n\nOverall, I think the phenomenon discovered in this paper is interesting. However, it didn't comment upon the neural tangent scaling, and it lacks mathematical support. I feel that this paper is on the borderline. I will consider raising my score if the authors can resolve these issues. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0sDvv8iVbBE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies gradient descent dynamics of two-layer neural networks with ReLU activation function. In a controlled setting, a list of experiments investigates the dynamics and compares them for different regimes where the relation of number of hidden neurons to number of training samples is changed and for several specific target functions. The paper proposes that training happens in two steps: The first phase resembles the learning of a random feature model (RFM) where only the output layer coefficients are changed. The second phase can activate or deactivate neurons and decreases coefficients to favor sparse solutions if the target function admits a sparse solution. The dynamics are further compared the setting of mean-field scaling, which shows a different learning dynamic.\n\nPros:\n+ The paper presents experiments on dynamics of gradient descent with the goal to give new insight on the important and unsolved phenomenon of implicit bias observed in deep neural networks. \n+ By starting in a (simplified) controlled setting, a large scope of experiments is conducted that studies different aspects and different training regimes. The setting and the experiments are all well-described and clearly presented. Interesting behavior of the learning dynamics can therefore be shown. \n+ With the largely overparameterized setting fairly well-understood, the current paper approaches the medium-overparameterized regime where the learning dynamics are more complicated and more close to neural networks in practice. Therefore, the paper targets an important knowledge gap.\n\nWhile some interesting observations can be made, I consider the presented results and gained knowledge as limited in scope and hence slightly tend to suggest to reject.\n\nCons:\n- In particular, no clear conclusion can be made from the results. In fact, there is no conjecture of how the results may manifest in practical neural networks. (In case I missed such a conjecture, then I wonder why the conjecture was not tested in a practical setting.) \n- The paper is purely experimental. Since it considers largely simplified settings (specific target functions, data sampled uniformly from the unit sphere, in most experiments the true solution can be found with a sub-network consisting of only one hidden neuron) and since the networks and optimization method (gradient descent) are rather simple, one could expect that at least some theoretical contribution or explanation could be given, which the paper lacks entirely. \n- While there is a large scope of experiments conducted, some results are only partial and draw conclusions without further investigations. It is hard to tell from the presented results how characteristic they are for other settings or behave under small changes such as the input dimension, target function, loss function, etc. \nFurther more detailed remarks on the specific contributions and the presentation are below. \n\nFor the author’s reply, I would appreciate a clarification of the exact contributions summarizing the findings and its possible implications for the training of practical neural networks.\n\n\nComments on the specific contributions:\n- The first contribution points out the existence two phases. In the first phase, the weights of the first layer do not change and the model behaves like a random feature model. This phenomenon seems to be limited to a few iterations where it is an almost trivial observation considering that the second-layer weights are initialized at zero (which implies a vanishing gradient for first layer weights). \n- The third mentioned contribution is peculiar since the only change is a factor of 1/m to the network function, which is equivalent (for the considered squared loss) to a scaling of the target function. The stated contribution is therefore that this scaling changes the observed dynamics significantly, which casts doubt when the observations generalize to other settings.\n- The observed behavior aims to explain the implicit bias of neural networks. However, this behavior can also be observed in overparamterized networks and the observed dynamics differ in the considered regimes of under-and overparameterized networks. This poses the question in which way the implicit bias is explained by the observations.\n - The paper suffers from its presentation. The list of contributions mentions an observed transition (what kind of transition?) and consider undefined terminology of quenched neurons. The terminology of quenched neurons is only loosely explained much later on page 4 as neurons that are “quenched” in the sense that their outer layer coefficients eventually start decreasing and then keep decreasing. (Shall quenching neurons have coefficients that converge to zero? If neurons can still pop up at a later stage, are they still considered quenched neurons? Are neurons with constant output layer coefficients but norm-decreasing inner layer coefficients also quenching (this difference cannot be distinguished in the plots)? It would be good to make precise what the authors mean by quenched neurons.\n\n\nMore detailed remarks:\n- First line page 3: citation seems misplaced, as I was unable to find the result in the paper. Please update reference or show a proof.\n- How many iterations does the first phase consist of where the model neural network behaves like a RFM? \n- Is it correct that the two phases in the loss development can not consistently be matched with changes in the parameters? \n- Is it correct that in the most realistic setting 3.1.2, there is barely a quenching behavior visible?\n- I was wondering about the notion of overparameterization. Why would a network be underparameterized just because the width is small even if the target function can be learned with a single neuron?It sounds reasonable to consider the setting of training polynomials of maximal degree 30 to learn a linear function as an overparameterized problem, no matter how many samples are considered for training. \n- The consideration to compare m being proportional to n against m being proportional to n/(d+1) has little meaning without fixing or comparing the respective constants (m=10^9*n/(d+1) vs m=10^-9*n also satisfies the stated proportionality), and most importantly without experimenting with different dimensions d. Since it is not investigated how the behavior changes with changes of d, there cannot be made any conclusion about the different regimes. Also the consistency claim in 4.2 is meaningless when changing the constants. It would be consistent if in 4.1 one would consider m=n and m=n/(d+1) instead of these terms scaled by constants (For a single d, suitable constants can always be found.) If no experiments for changing input dimension are carried out, then all we need to care about is the quotient m/n. \n\n\nTypos:\nFigure 6: legend not entirely visible \nConclusion: „the the“", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper empirically investigates the gradient dynamic of two-layer network nets with ReLU activations on synthetic datasets under $L^2$ loss. The empirical results show that for a specific type of initialization and less overparametrized neural nets, the gradient dynamics experience two phases: a phase that follows the random features model where all the neurons are *quenched* and another phase where there are a few *activated* neurons. As pointed out by Reviewer 1, this paper lacks mathematical support and did not distinguish between *random features model* and *neural tangent model*. Reviewer 3 and Reviewer 4 also complained that the paper is purely experimental. Therefore, this paper may benefit from proposing an at least heuristic or high-level conjecture/interpretation/argument that tries to explain the empirical results.  
Decision: Reject