Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Authors: Hanxu Zhou
Qixu Zhou
Zhenyuan Jin
Tao Luo
Yaoyu Zhang
Zhi-Qin John Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: training dynamics
neural networks
phase diagram
initialization

Abstract: Substantial work indicates that the dynamics of neural networks (NNs) is closely related to their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs with infinite width (Luo et al., 2021), we make a step towards drawing a phase diagram for three-layer ReLU NNs with infinite width. First, we derive a normalized gradient flow for three-layer ReLU NNs and obtain two key independent quantities to distinguish different dynamical regimes for common initialization methods. With carefully designed experiments and a large computation cost, for both synthetic datasets and real datasets, we find that the dynamics of each layer also could be divided into a linear regime and a condensed regime, separated by a critical regime. The criteria is the relative change of input weights (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) as the width approaches infinity during the training, which tends to $0$, $+\infty$ and $O(1)$, respectively. In addition, we also demonstrate that different layers can lie in different dynamical regimes in a training process within a deep NN. In the condensed regime, we also observe the condensation of weights in isolated orientations with low complexity. Through experiments under three-layer condition, our phase diagram suggests a complicated dynamical regimes consisting of three possible regimes, together with their mixture, for deep NNs and provides a guidance for studying deep NNs in different initialization regimes, which reveals the possibility of completely different dynamics emerging within a deep NN for its different layers.

Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Hanxu Zhou', Qixuan Zhou', Zhenyuan Jin', Tao Luo Yaoyu Zhang'., Zhi-Qin John Xu'; School of Mathematical Sciences, Institute of Natural Sciences, MOE-LSC and Qing Yuan Research Institute, Shanghai Jiao Tong University
CMA-Shanghai, Shanghai Artificial Intelligence Laboratory Shanghai Center for Brain Science and Brain-Inspires Technology
Abstract
Substantial work indicates that the dynamics of neural networks (NNs) is closely related to their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs with infinite width (Luo all (02II,, we make step to- wards drawing phase diagram for three-layer ReLU NNs with infinite width. First, we derive a normalized gradient flow for three-layer ReLU NNs and ob- tain two key independent quantities distinguish different dynamical regimes for common initialization methods. With carefully designed experiments and a large computation cost, for both synthetic datasets and real datasets, we find that the dynamics each layer also could be divided into linear regime and a condensed regime, separated by critical regime. The criteria is the relative change of input weights (the input weight of a hidden neuron consists of the weight from its input layer the hidden neuron and its bias term) the width approaches infinity dur- ing the training. which tends to 0, +00 and (1), respectively. In addition, we also demonstrate that different layers training process within a deep NN. In the condensed regime, we also observe the condensation of weights in isolated orientations with low complexity. Through experiments under three-layer condition, our phase diagram suggests a compli- cated dynamical regimes consisting of three possible regimes, together with their mixture, for deep NNs and provides guidance for studying deep NNs in dif- ferent initialization regimes, which reveals the possibility of completely different dynamics emerging within a deep NN for its different layers.
lie in different dynamical regimes in
Introduction
As neural networks (NNs) have been employed to extensive practical and scientific problems, such as computer vision, natural language processing, and automatic driving, etc. Studying the rationale behind them help comprehend the principles behind NNs and better NNs. For example, the initialization of neural network parameters plays an important role in the training dynamics and generalization of NN fitting.
Several types of initialization schemes are often used practical trainings, such as He initialization (He all, 2015), LeCun initialization (LeCCun et all, 2012), Xavier initialization (Glorot and Bengid, 2010), etc. Theoretical study further explores how initialization affects the training dynamics of NNs. For example, the NTK scaling (lacot aD, 2018: Chizat and Bach, 2019: Arora ct all 2019: Zhang cf all 2021) leads to a training where parameters stay at a small neighborhood of the initialization. therefore, the NN after training can be well approximated by the first-order Taylor expansion at the
"Corresponding author: xuzhiqin @ @jtu.edu.cc
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

initial point, i.e., linear dynamics, The mean-field scaling (Mei et all 2019: Rotskoff and Vanden Eynden, 2018: Chizat and Bach, 2018 Singnano and Spiloooulos 2020 leads a training where parameters significantly deviates from the initialization, therefore, the NN dynamics exhibits clearly non-linear behavior. Luo et et all (202T) systematically study the effect of initialization for two-layer ReLU NN with infinite width by establishing phase diagram, which shows three distinct regimes, i.., linear regime, critical regime and condensed regime, based on the relative change of input weights (the input weight of hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) as the width approaches infinity during the training, which tends to 0, (1) and +oo, respectively. The condensed regime is named so, because empirical study finds that orientations of the input weights condense in several isolated orientations, which is a strong feature learning behavior, an important characteristic of deep learning.
The study in Luo et all (202TI) is limited in two-layer NNs, however, empirical studies show that deep networks can learn faster and generalize better than shallow networks in both real data and synthetic data (He et all, 2016; Arora et all, 2018; Xu and Zhou, 202I). Because of multi-layer structure and non-linearity, the study of multi-layer nonlinear NNs is often challenging. For example, the mean-field theory is established on two-layer NNs and difficult to be extended to multi-layer NNs (Mei et all 201% Rotskoff and Vanden-Lijnden 2018 Chizat and Bacli, 2018: Sirignano and Spiliopoulos, 2021). Even for empirical study, excessive ayper-parameted often set an obstacle to obtain clear phenomena as well, where multi-layer NNs could have distinct characteristics compared with shallow two-layer NNs. For example, due to the existence of other layers, how is the dynamical regime of the weights of a layer in multi-layer NNs different from two-layer ones and can weights of two different layers experience distinct dynamical behaviors in one training process? Answering these questions is important understand how the dynamics of multi-layer NNs depend on the initial hyper-parane pprrmeaarmeeers, thus, leading to a deeper understanding of deep NNs and providing guidance for tuning parameters in practical training tool for the theory community. For instance, if we want to study the nonlinear effect of a hidden layer, we can set this interested hidden layer in the non-linear regime and all other hidden layers in the linear regime. Further, we can continue to study how the non-linear effect of a hidden layer depends on its depth.
Three-layer network is good starting point study multi-layer NNs as it only has two hidden layers for studying the interaction between layers. Besides, to have a clear boundary of different focus three-layer ReLU NNs with large-width limit. A difficulty to
dynamical regimes, overcome is to find appropriate quantities required for describing a phase diagram, which should be able to identify distinct different regimes, and can also account for the same dynamical behavior when papammpprretters are different but those quantities are the same. To this end, we derive a normalized gradient flow for a three-layer ReLU NN. Since we only care about the training trajectory up to a time scaling, we obtain three independent quantities that meet our requirements. And then, we further simplify the setting by assuming weights of the two outer layers use a same initialization scheme as the commonly used in practice, and we obtain two key independent quantities. With carefully designed experiments and a large computation cost, for both synthetic datasets and real datasets, we find that weights of each layer also have a linear regime and a condensed regime, separated by a critical regime. The criteria is the relative change of input weights as the width approaches infinity during the training, which tends to 0, +oo and O(1), respectively. In addition, we also find that different layers can lie different dynamical regimes in a training process. In the condensed regime, we similarly observe the condensation of weights in isolated orientations.
This work empirically draws a phase diagram for three-layer ReLU NNs with large width, as shown in Fig. II For deeper networks, this work suggests that the dynamical regime for the weights of each layer is similar to that of two-layer NN and different layers can lie in distinct regimes in a training process. For NNs with not too large width, the boundary between regimes is not sharp but this work still sheds lights on how the initialization affects the dynamics of weights in different layers. Our study on phase diagram of three-layer ReLU NNs thus lays an important foundation for future work on the dynamical behavior of multi-layer NNs and corresponding implicit regularizations at each of the identified regimes and guides the parameter initialization tuning in the practice.
2 Related works
series of works have shown that global minima can be found for NNs with several specific initial- izations in linear regime (Zou ctal 20s Allen et all 2010: Durcaal al, 2019: E all 2020) or

---

CR for Z.
CR wil, OR for Will
Wi, CR W!l
LR for W., A.R. Will
Ensingie:
(1,00
(5R 199
Xovicr
NTK
Leran, He
(iN.i)) W. (0, wi (0,89)
Jagm
Condenand Relgree,
Regine
Figure 1: Phase diagram of three-layer ReLU NNs at infinite-width limit. The four regimes are empirically partitioned, and the marked examples are studied in existing literature (see Table 1 for details).
for the two-layer ReLU NNs in the whole linear regime eL 2021). Gerger et al. (2020) probe the dynamic crossover between the NTK limit and the mean-feild limit, and Yang (2020); Yang and Luttwn (24124)) show that tangent kernel of a randomly initialized neural network with 'arbitrary architecture' converges to deterministic limit, the network width goes to infinity. Chen et al (2021) prove that for wide shallow NNs under the mean-fichl field scaling and with a general class of activation functions, when the input dimension is at least the size of the training set, the training loss converges to zero at a linear rate under gradient flow.
In the condensation regime, large network with condensation is effectively a network of only few effective neurons, leading to an output function with low complexity (Bartlett and Mendelson. 2002), thus, may provide possible explanation for good generalization performance of large NNs (Breiman, 1995; Zhang al,, 2021). The condensation suggests a bias towards simple function, which is consistent with the frequency principle that NNs tend to learn data by low-frequency func- tion (X all 2UTE, Rahaman el all 201% Xn el all 2022 21122). NNs have significant feature learning in the condensation regimes. To actively learn features, Lyu et all (2021)) small initial- ization to study how gradient descent can lead a NN to a solution 'maxx -mrginn solution.
How the condensation happens during the initial training stage is explored for ReLU activation func- tion (Maennel et all 2018; [Pellegrini and Birol, 2020) and general differential activation functions (Zhou et all 2021). The embedding principle (Zhang et all 2021abb shows that the loss landscape of NN contains all critical points/functions of all the narrower NNs, thus providing basis for why large network can condense during the training.
3 Preliminary
For convenience, we consider three-layer NN with m hidden neurons for each layer,
fo(x) aT (W
(1)
where x E Rd, is the scaling factor, is the width of both layers, 0 = vec(a, W Will with E R W E Rmxm, W E RRmx is the set of parameters initialized by a (00 (8) !
Ek (O, (2). will kR ~O, (8)) i.i.d. The bias term 6(") can be incorporated by expanding
and will to (x 1)T and (wil), 6 dfl)T, while 6(² can be incorporated by expanding æ (WW)) [2] to (TT.11 T)T and (W12), 1.
and 2
At the infinite-width limit NNs can be approximated by linear dynamics neural tangent kernel (NTK) (Jacot et (2018)),
00, given 31, 82, 83 O(1), for v Mim,, the gradient flow of
whereas given 31 V mi mittai 32 V mi mm Gz 1 for a O(1) gradient flow of NN
exhibits highly nonlinear dynamics of Xavier Glorot and Bengio (201))..

---

Assumption 3.1. For convenience, we assume that the number of neurons in each hidden layer is equal, i.c. my m2 m.
Since the output is linear with respect a, is not the main concern, and the initialization methods used for neural network in practice often has 32 = B3g, where B is a constant in dependent of m. So, take the following assumption.
Assumption 3.2. 82 Bß3, where B is constant independent of m.
Cosine similarity: The cosine similarity for two vectors u and is defined as
ute
D(u, v) (uTT))/2(oTo)11>
(2)
4 Rescaling and the normalized model of the three layer neural network
To identify sharply distinctive regimes/states, we take the infinite-wddth limit m - our starting point. Identification of the coordinates is important for drawing the phase diagram. There are some guiding principles for finding the coordinates of a phase diagram etl 2021): (i) They should be effectively independent; (ii) Given a specific coordinate in the phase diagram, the learning dynamics of all the corresponding NNs statistically should be similar up to time scaling; (iii) They should well differentiate dynamical differences except for the time scaling.
Guided by above principles, in this section, we perform the following rescaling procedure for fair comparison between different choices of hyperparameters and obtain normalized model with two independent quantities irrespective of the time scaling of the gradient flow dynamics.
The empirical risk is
(s(
Σ (Jo(a:) - yz)))
2n
Then the training dynamics based on gradient descent (GD) the continuous limit obeys the follow- ing gradient flow of 0,
d0
(oRs(0).
dt
More precisely, we obtain that:
da
(
o
(2] ((
d/
71
dw [2]
( (M
o'
(5)
dw(]
(a o' (w (wi))))) @ 2 o'(il") m W
dt
where 6 = (: a setting
- vi). the operation @ is the Hadamard product.
Then,
33 a, W
wP
w
[1]
Wl), 3
(6)
B2
so that all the parameters are picked from standard normal distribution. By the chain rule of deriva- tive and the homogeneity of ReLU, i.e. a(nu) = an and o'(au) = a' o'(u), where 0s scalar and is vector, obtain that:

---

Table Common initialization methods with their scaling parameters
Name
a
W
W
K2
K3
72
73
NTK
m1
1
m2g
Lecun
1
d mg
TP2
He
2
2
d m2
122
myydg
Xavier
2
d+-m ma+
8/(m; mm
mg
d mm
(mg
m)
da
Kgd W (W a.:)
ez.
dt
d dw
nij
n
Kza C
W
W 21) (W x,)"² eis
x,)"
(
dt
dW
-n²
a
21
W x;1)) @ 0' ewllll x2)= ei
db
where, we introduce four scaling parameters
$3
B3
318388
K1
K2
K3
7 == (aII) 8
(8)
32
B1
Note that K1, K2 and Ky do not follow principle (ii) and (iii) above infinite-width limit. They are in general functions of which attains O, O(1), too m - co. To account for such dynamical difference under different widely considered power-law scalings of ax, B1, 32 and Bg shown in Table 1 and based on our assumption [ and 8.2, we arrive at
log K2 log m
log K3
222 lim +00
% lim limm-toc
log m
where our assumptions and 3 mean K
B, leading to 71 limmy-os K1. So we (0, 72, 73).
log
discuss the phase diagram of three layer network under the coordinates: (71, 72, 73) and focus on (72, (3).
Remark 4.1. Here we list some commonly-used initialization methods and/or related works with their scaling parameters shown in Table 1.
5 Empirical phase diagram
5.1
Experimental setup
Throughout this section, we use three-layer fully- connectdd neural networks with size, d-m-m-dout- The input dimension determined by the training data, i.e., = 1 for synthetic data and d 28 x 28 for MNIST. The output dimension is dout 1 for synthetic data and dous 10 for MNIST. The number of hidden neurons m is specified in each experiment. All parameters are initialized by Gaussian distribution N(0, var), where var depends on B1, 3r and 83. The total data size is n. The training method gradient descent with full batch, learning rate Ir and MSE loss. For synthetic data, we use simple 1-d problem of training points.
5

---

5.2 Intuitive experiments of synthetic data
With V and 7/2 as coordinates, in this subsection, we illustrate through experiments the behavior of diversity of typical cases over the phase diagram regarding W and W respectively. We use simple 1-d problem of 4 training points, which allows an easy visualization.
23-
31-
02
40124
73
1.0
(b) 73 1.5
(c) 73 2.0
-350
-25
(d) 73 1.0
(e) 73
1.5
(1) 733 2.0
Figure 2: Learning four data points by three-layer ReLU NNs with different y 's are shown in the first row. The corresponding scatter plots of initial (red) and final (green) {W [1] pm shown in the second row. 7/2 hidden neuron number m 10000.
jk=1 E R² are
The first row in Fig. shows typical learning results over different %3's. from a relatively jagged interpolation (NTK scaling) to smooth interpolation (mean-field scaling) and further to a linear spline interpolation. With input weight and bias, w is two-dimensional for d = 1. We display the (((W()) & ), (wi. )2)] of the trained network Fig. 22 For 7/3 = 1.0, the initial scatter plot very close the one after training. However, for 7s 2.0, active neurons (i.e., neurons with significant amplitude and away from the origin) are condensed at few orientations, which strongly deviates from the initial scatter plot. For 73 1.5, the scatter points present an intermediate state.
&
Inspired by W we found that the directions between the input weights is very important group feature, which can also reflect the characteristies between different 7/3 and 722 Considering that w is m + dimensional, We cannot directly and intuitively show {W/? [2] === in figure. Here, w [2] represents the input weight of the k-th neuron in the second hidden layer. Therefore, to characterize the distance between different vectors in {W(+)
k 1k=1- we introduce D(u, v) to denote the inner
[2] 1m
product of two [W( [2) } as Eq. (2). shown Fig. B. for Y
2.2, w
[2] hardly has a strong
relationship with each other, while for 7s 2.8, most of the w!!l very close to each other, showing strong condensation.
Phase diagram for three-layer neural networks
In this section, with 73 and 72 coordinates, characterize at - the dynamical regimes of NNs and identify their boundaries in the phase diagram through experimental approach. How to characterize and classify different types of training behaviors of NNs is important open question. However, there is little work on the three-layer phase diagram or the multi-layer one. For the three- layer infinitely wide ReLU neural networks, shown in Fig. 1, several works have proved specific points in the lhyee-layer phase diagram belong to the linear regime of both W1ll, and W!! However,

---

cos distancer: relo
COn dadance: relp
coe distance: IBU
635
633
120
435
625
525
605
100
000
(a) 73 2.2
(b) 73 2.5
(c) 73 2.8
Figure 3: Learning four data points by three-layer ReLU NNs with different Yt's are shown in the first row, with 7/2 0.5 and hidden neuron number 10000. The color indicates Duu, u) of two hidden neurons' input weights initialized by Gaussian distribution, whose indexes are indicated by the abscissa and the ordinate, respectively. If neurons are in the same navy-blue block, D(u, v) se 1 (beige block, (u,,v) Rs O, and red block, D(u, v) -1), their input weights have the same (per- pendicular, opposite) direction. Considering that not all neurons are activated for ReLU activation function, we only select half of the neurons with the largest modulus length to draw D(u, v).
its exact range in the phase diagram remains unclear. On the other hand, NN training dynamics can also be highly nonlinear at m - 00, such as widely used initialization method, Glorot and Bengio (20TM), as a point shown in the phase diagram. However, whether there are other points in the phase diagram that has similar training behavior is not well understood in three-layer condition. In addition, is still not clear if there are other regimes the phase diagram that are nonlinear but behaves distinctively comparing to the Glorot and Bengid (20I0) initialization method. In the following, we will address these problems and draw a three-layer phase diagram.
5.3.1 Regime identification and separation
To characterize different dynamical regimes, we define the relative change of weights as
RD(W(l)) 100 0w: (0)ll2
10w. 9w: (0)lk
RD(W
(10)
[|0w; (0)l2
10w, (0)ll2
where W. and Ow, are the weights after training for the first and the second hidden layer, and Ow, (0) and Ow, (0) for initial weights.
10a
00.6000
slopo--0. 0216
slopc=0.2194 data
data
data
10
10-1
10a
10a
10a
10
107
10a
10a
10
m
m
(a) 73 0.9
(b)
1.5
(c) 73 2.1
Still learn four data points in Fig. D by three-layer ReLU NNs with different 73's 's and ?2 0. The slopes in (a)(() are fitted by RD(W(ll) w.r.t m = 100, 1000, 2000, 5000, 10000 in a log-log scale. As 73 grows from 0.9 to 2.1, the corre- sponding slopes grow well.
Figure 4: RD(W(Ll)
We quantify the growth of RD(W()), = 1,2 as m + co. As shown Fig. [(-c)), they approximately have power-law relation. Therefore, similar to Ct (2021), define
7

---

Table 2: Two groups of Sw; and Sw.- Within a group, and 's are the same, while a, a, W and W are different. These values are the average of eight experiments.
Group
a
W [2]
W
Y22
Ys
Sw,
Sw;
Std(Sw,)
Std(Sw,)
0.5
8/15
8/15
5/15
0
1.1
-0.4108
-0.9119
No. 1
1
11/30
11/30
11/30
0
1.1
0.4084
0.9364
6.44x10-3
1.05x10-1
m
m 0.2
m 0.2
m
0.2
0
1.1
0.4231
0.9310
0.3
m
7/6
m 7/6
m T/15
0.7
2.5
0.3130
0.0251
No. 2
m
16/15
m 16/15
m 11/30
0.7
2.5
0.3176
0.0263
4.64 x10
2.50x10-3
U.3
m 29/30
m
29/30
m
4/15
0.7
2.5
0.3243
0.0205
log RD(W
Swi = lim
(11)
log
which empirically obtained by estimating the slope in the log-log plot. As shown in Table NNs with the same pair of 72 and "/3 but different 31, $2 and 83, have very similar Sw, which validates the effectiveness of the normalized model. the following experiments, we only show result of one combination of a, 81, B2 and 33 for pair of 7/2 and 73-
Then, we explore the phase diagram by experimentally scanning Sw, over the phase space. The result for the same 1-d problem as in Fig. D is presented in Fig. B In the red zone, where Sw, is less than zero, RD(WI)) + O as + oc. indicating linear regime. In contrast, in the blue zone, where Sw; is greater than zero, RD(W!.) + cc m - cc, indicating highly nonlinear behavior. Their boundary are experimentally identified through interpolation indicated by stars in Fig. 55, where RD(Wle)) regime, by large amount of experiments In order to verify the rationality of the three-layer phase diagram, we also verify the results on the MNIST dataset, as shown in Fig. E. and obtain similar results.
O(1). The dashed lines are speculative boundary, i.e., critical
12 42
3,002AAUU ZUGTA GLODAL AEGEMA 20L
44-
-22
03
1>
(a) Sw:
(b) Sw:
(c) Swz
Figure 5: For synthetic data, Sw: estimated on three-layer ReLU NNs of 100, 1000, 2500, 5000 and 10000 hidden neurons of each layer over 73 (ordinate) and 72 (abscissa). The stars zero points obtained by the linear interpolation over different different 73 for each fixed 72 in (a) to (c), while the stars are zero points obtained by the linear interpolation over different dif- ferent 2/2 for each fixed 7/ in (b). Dashed lines are auxiliary lines indicating our conjecture.
5.4 Condensation
We then examine the condensation over the phase diagram. For w we display the (((W(()) ), (wi"l) ) for two different regions in Fig. M As is shown in Fig. M on the left side of the blue squares, the parameters after training and the initialization parameters are very

---

03
04
41
47-
(a) Sw, of mnist
(b) Sw; of mnist
(c) Sw, of mnist
Figure 6: For mnist data, Sw: estimated on three-layer ReLU NNs of m 100, 1000, 2500, 5000 and 10000 hidden neurons of each layer over "ts (ordinate) and "22 (abscissa). The stars are zero points obtained while the stars are zero points obtained by the linear interpolation over different different 72 for each fixed 73 in (b). Dashed lines auxiliary lines indicating jecture.
the linear interpolation over different different " for each fixed "/2 in (a) and (c),
close, while on the right side of the blue squares, weights with not too small magnitude are con- densed at a few orientations, which strongly deviates from the initial scatter plot. The graphs in the blue boxes present intermediate state process state, making the aoooe-mee change continuous process. Next, we need to define a quantity to describe this phenomenon.
For the second hidden layer, the weights is very high-dimensional. We define quantity to qualita- tively measure the degree condensation. We define < (m/2)² Eny² (uu,, u.)l, which the average inner product distance between different W and subscript [A] indicates that the row of w/2l which has a magnitude with ranking among all rows. More condensation yields larger As is shown in Fig. M. we could clearly observe that the condensation tendency is consistent with phase distribution Fig.
03
-1
-32
0.2
235
2.75
Figure 7: Regime characterization of three-layer Neural Network for { (1] pm
width 10000.
The network structure and the four points of the objective function are consistent with Fig. 0 73 and 72 indicated by the abscissa and the ordinate, respectively. The red dots the small figure corre- spond to the initialization parameters, and the green dots correspond to the post-training parameters.
6 Conclusion
In this paper, empirically characterized the linear, critical, and condensed regimes with distinctive features and draw phase diagram for the three-layer ReLU NN the infinite-width limit. The homogeneity of ReLU is is an important basis for deriving the coordinate of the phase diagram, so could not be replaced by any non-linearity activation function. Through experiments, we further identify the condensation as the signature behavior in the condensed regime of strong non-linearity.

---

2.20
2,65 2.20 2:55 2,00
0.8
0.7
0.6
0.5
V2
0.5
0.4
0.3
0.2
0.1
Y3
Figure 8: < of three-layer Neural Network for { 22] Im and the four points of the objective function are consistent with Fig. B. 7 and 722 indicated by the abscissa and the ordinate, respectively. For the color of the squares in the picture, dark blue means that the w condense in one direction, while dark red means that W s are irrelevant with each other. Since the experimental results will change with the initialization by Gaussian distribution, in order to ensure the stability and repeatability of the experimental results, we repeated the experiment for 8 times, averaged the results, and finally obtained the figure.
Jk-i at width 10000. The network structure
Our phase diagram figures out the relation between the training dynamics of multi-layer neural networks and their initialization, reveals the possibility of having different training dynamics for different layers within neural network, and serves as a map that provide guidance for the future study of deep NNs, such how different dynamical regimes of different hidden layers affect the generalization.
Acknowledgments and Disclosure of Funding
This work is sponsored by the National Key R&D Program of China Grant No. 2022YFA 1002000
(Z. X., T. L., Y. Z.), the Shanghai Sailing Program, the Natural Science Foundation of Shanghai Grant No. 20ZR1429000 (Z. X.), the National Natural Science Foundation of China Grant No.
62002221 (Z. X.). the National Natural Science Foundation of China Grant No. 12101401 (T. L.), Shanghai Municipal Science and Technology Key Project No. 22IC1401500 (T. L.), the Na- tional Natural Science Foundation of China Grant No. 12101402 (Y. Z.), Shanghai Municipal of Science and Technology Project Grant No. 20JC1419500 (Y.Z.), the Lingang Laboratory Grant NoLLG-QS-20220--00 (Y.Z.), Shanghai Municipal of Seience and Technology Major Project No. 2021SHZDZX0102, and the HPC of School of Mathematical Sciences and the Student Innovation Center, and the Siyuan-l cluster supported by the Center for High Performance Computing at Shang- hai Jiao Tong University, Key Laboratory of Marine Intelligent Equipment and System, Ministry of Education, P.R. China.
References
T. Luo, Z.Q.. J. Xu, Z. Ma, Y. Zhang, Phase diagram for two-layer relu neural networks at infinite- width limit, Journal of Machine Learning Research 22 (2021) 1-47.
K. He, X. Zhang, Ren, Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE international conference computer vision, 2015, pp. 1026-1034.
Y. LeCun, I.. Bottou, G. Orr, K.-R. Müller, Efficient backprop, in: Neural networks: Tricks of the trade, Springer, 2012. pp. 9-48.
10

---

X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the thirteenth international conference on artificial intelligence and statisties, 2010, pp. 246-256.
A. Jacot, F. Gabriel, C. Hongler, Neural tangent kernel: convergence and generalization in neural networks, in: Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018, pp. 8580-8589.
L. Chizat, F. Bach, A note on lazy training in supervised differentiable programming, in: 32nd Conf. Neural Information Processing Systems (NeurIPS 2018), 2019.
Arora, S. Du, Hu, Li, R. R. Salakhutdinov, R. Wang, exact computation with an infinitely wide neural net, Advances in Neural Information Processing Systems 32 (2019) 8141- 8150.
J....Xu Xu, T. Luo, Z. Ma, A type of generalization error induced by initialization in deep Y. Zhang, Z. neural networks, in: Mathematical and Scientific Machine Learning, PMLR, 2020, pp. 144-164.
S. Mei, Misiakiewicz, A. Montanari, Mean-field theory of two-layers neural networks: dimension- free bounds and kernel limit, in: Conference Learning Theory, PMLR, 2019, pp. 2388-2464.
G. M. Rotskoff, E. Vanden-Eijnden, Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks, in: Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018. pp. 7146-7155.
L. Chizat, F. Bach, On the global convergence of gradient descent for over over-parane parameterized models using optimal transport, in: Proceedings the 32nd International Conference on Neural Informa- tion Processing Systems, 2018, pp. 3040-3050.
Sirignano, K. Spiliopoulos, Mean field analysis of neural networks: A central limit theorem, Stochastic Processes and their Applications 130 (2020) 1820-1852.
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
S. Arora, N. Cohen, E. Hazan, On the optimization of deep networks: Implicit acceleration by overparameterizzaionn, in: G. Dy, Krause (Eds.), Proceedings of the 35th International Con- ference Machine Learning, ICML 2018, Stockholmsmässan. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, PMLR, 2018, pp. 244-253. URL: http::/proceee.nneeeee/
m press wwtp////a/araaaaaaa html]
Z.-Q. J. Xu, H. Zhou, Deep frequency principle towards understanding why deeper learning is faster, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
D. Zou, Y. Cao, D. Zhou, Q. Gu, Stochastic gradient descent optimizes over-parameterizee deep relu networks, arXiv preprint arXiv: TDW::8111088880 1.08888 (2018).
Z. Allen-Zhu, Y. Li, Z. Song, A convergence theory for deep learning via parametrriizin,nnnnn, in: International Conference on Machine Learning, PMLR, 2019, pp. 242-252.
S. S. Du, X. Zhai, B. Péezos, A. Singh, Gradient descent provably optimizes wre-paaameterizee neural networks, in: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019.
W. E, C. Ma. L. Wu, A comparative analysis of optimization and generalization properties of two- layer neural network and random feature models under gradient descent dynamics, Science China Mathematics (2020) 1-24.
M. Geiger, S. Spigler, A. Jacot, M. Wyart, Disentangling feature and lazy training in deep neural networks, Journal of Statistical Mechanics: Theory and Experiment 2020 (2020) 113301.
G. Yang, Tensor programs i:: Neural tangent kernel for any architecture, arXiv preprint arXiv:2006. 14548 (2020).

---

G. Yang, E. Littwin. Tensor programs iib: Architectural universality of neural tangent kemee training dynamics, in: International Conference on Machine Learning, PMLR, 2021, pp. 11762-11772.
Z. Chen, E. Fanden-Eijnden J. Bruna, On feature learning in shallow and multi-layer neural net- works with global convergence guarantees, in: International Conference on Learning Representa- tions, 2021.
L. Bartlett, S. Mendelson, Rademacher and gaussian complexities: Risk bounds and structural results, Journal of Machine Learning Research 3 (2002) 463-482.
L. Breiman, Reflections after referceing papers for nips. The Mathematics of Generalization XX (1995) 11-15.
C. Zhang, Bengio, M. Hardt, B. Recht, O. Vinyals, Understanding deep learning (still) requires rethinking generalization, Communications of the ACM 64 (2021) 107-115.
Z.-Q. Xu, Y. Zhang, Y. Xiao, Training behavior of deep neural network in frequency domain, International Conference on Neural Information Processing (2019) 264-274.
N. Rahaman, D. Arpit, A. Baratin, F. Draxler, M. Lin, F. A. Hamprecht, Y. Bengio, A. Courville, On the spectral bias of deep neural networks, International Conference on Machine Learning (2019).
Z.-Q.J. J. Xu, Y. Zhang, T. Luo, Xiao, Z. Ma, Frequency principle: Fourier analysis sheds light deep neural networks, Communications in Computational Physics 28 (2020) 1746-1767.
Z.-Q.J. J. Xu, Y. Zhang, T. Luo, Overview frequency principle/spectral bias in deep learning, arXiv preprint arXiv:2201.00
(2022).
K. Lyu, Z. Li, R. Wang, Arora, Gradient descent on two-layer nets: Margin maximization and simplicity bias, Advances in Neural Information Processing Systems 34 (2021).
H. Maennel, O. Bousquet, S. Gelly, Gradient descent quantizes relu network features, arXiv preprint #xXiv:1803.08367 (2018).
Pellegrini, G. Biroli, An analytic theory of shallow networks dynamics for hinge loss classifica- tion, Advances in Neural Information Processing Systems 33 (2020).
H. Zhou, Q. Zhou, T. Luo, Y. Zhang, Z.-Q. J. Xu, Towards Understanding the Conden- sation Neural Networks at Initial Training, arXiv e-prints (2021) arXiv:2105.1 1686.
arXiv:2106 1666
Y. Zhang, Z. Zhang, T. Luo, Z. Xu, Embedding principle of loss landscape of deep neural networks, Advances in Neural Information Processing Systems 34 (2021a).
Y. Zhang, Y. Li, Z. Zhang, T. Luo, Z.-Q. J. Xu, Embedding principle: a hierarchical structure of loss landscape of deep neural networks, arXiv preprint arXiv:211 1. (2021b).
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A]. You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
12

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] In supplemen- tary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A]
(e) Did you diseuss whether the data you using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "u74ABJKGy0F", "reviewer": "Reviewer_PDY9", "summary": "This paper builds upon the work of Luo et al., 2021 and characterizes the phase diagram for different dynamical regimes in the case of a three-layer neural network at infinite width. Based on appropriately chosen quantities, the paper shows that different layers can be simultaneously in the linear, critical or condensed regime. The proposed phase diagram is backed with several experiments that depict the nature of the learned function and dynamics in these regimes.  ", "questions": "*Firstly, check out the first point under weaknesses. I would be very interested in looking at additional results related to the mentioned questions therein.*\n\nBesides, there are also some other questions and suggestions that I have related to the experiments:\n\n- Figure 3: Can you make the plot for the similar setting as in Figure 2? It is better to have results in comparable settings. Also, do you see more condensation in the initial layers or towards the output? \n\n- How do the results in Figure 2 and 3 with increasing number of samples? Do we see that some initialization regimes coalesce when number of samples is large?\n\n- Can you share a similar plot as in Figure 4 for RD(W^[2])?", "limitations": "NA. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- I personally find this line of work of characterizing the phase diagram to be quite insightful. It provides a useful perspective to think about neural networks. \n\n## Weaknesses: \n- The phase diagram partitions the space into four possibilities based on the choice of regime for each layer. However,  more or less all the experiments and results are in the setting when the second hidden layer is in the linear regime. As a result, most of their findings are qualitatively very similar to that observed in Luo et al., 2021. Given their richer characterization of the phase diagram as compared to prior work, the authors nevertheless do not discuss these potential implications: e.g., it would have been understanding to compare what happens in the condensed regime for both hidden layers. As well as, how does condensed regime in a deeper layer while being linear in the lower layer compare to the converse scenario? Further, quantitatively exploring the advantages/disadvantages of being in these different regimes on generalization/optimization could have been worthwhile too. So, I think there are several missed opportunities here and the paper is not properly fleshed out, and overall lacks in contribution.\n- On this note, this work is heavily inspired and based on the Luo et al., 2021 -- which in itself is not wrong, but their manner of presentation is misleading. For example, the quantities gamma_3, gamma_2, kappa_3 are essentially identical to the gamma, gamma^\\prime, kappa in Luo et al 2021 (gamma_3 is just 1/2 more than gamma since here we have one additional layer). Likewise most of the considered experiments are modeled based on this prior work, and even most of the presented results/figures are quite similar. Again to emphasize, it is absolutely alright to build up on results and experiments from past work, but there should be appropriate acknowledgement --- which are missing here. To give another example, the authors write in section 5.3.1 of regime identification and separation, that \"we define the relative change of weights... \" but it is the same as Luo et al 2021 and even the results in the figure below are basically the same as in that work. \n\n### Minor Remarks:\n- For Xavier initialization, the values of the gamma_2 and gamma_3 should be interchanged. Also, in the same row for Xavier init, the the value of kappa_3 seems wrong. \n- Writing is a bit sloppy and redundant at times: some lines are repeated multiple times, in the abstract, at the beginning of introduction, and towards the end of introduction. And often there are many grammatical mistakes, so it would be a good idea to fix them as well.\n- The figures do not come out well in print.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c2aZ_KaLdGy", "reviewer": "Reviewer_iryP", "summary": "This work studies empirically the effect of initialization scalings on the dynamics of three layer neural networks. The main result is a phase diagram exhibiting quite clearly the presence of different dynamical regimes for the two different layers (condensed and linear).", "questions": "- Is the phase diagram of Fig. 1 plotted based on theoretical or empirical lines ? This should be made more clear.\n- « For instance, if we want to study the nonlinear effect of a hidden layer, we can set this interested hidden layer in the non-linear regime and all other hidden layers in the linear regime » what do the authors mean by « study the nonlinear effect » ? This is vague and does not match the description of « guidance for hyper parameter tuning » mentioned in the sentence before \n- «  the dynamics of a is not the main concern » : could the authors provide more justification on this ?\n- I do not understand the following sentence (« which leads to the initialization parameters... »), and the parameter B in beta2 = B beta3 does not seem to be defined or used anywhere\n- Figures 5 and 6 : the caption mentions that the width m is varied across 5 different values but I am confused about how that reflects in the figures : are the results averaged over the different values of m ?\n- It would be nice to have a discussion on the impact of the nonlinearity. For example including a phase diagram for Tanh ?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- Despite a few typos the paper is well presented and easy to read.\n- The phase diagram of figure 1 is neat and a valuable contribution in my opinion. It is well illustrated by the empirical figures of Section 5.\n- The empirical investigation is thorough.\n\n## Weaknesses\n\n- The most important part of the paper, where the parameters gamma2 and gamma3 are derived, is rushed and difficult to understand. The authors should clarify their choice as it looks a bit like dark magic right now\n- Some important works on phase diagrams are missing in related works, for example the work on disentangling feature and lazy regime by Mario Geiger and colleagues and the series of papers on tensor programs and muparametrization by Greg Yang and colleagues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7a-bwebVKWi", "reviewer": "Reviewer_1ShU", "summary": "\nBuilding on Luo et al. 2021, the current submission presents a detailed numerical study of gradient flow of feedforward neural networks with different scaling parameters used for their initialization.\n\nThe experiments reveal that different layers of the three-layer architecture exhibit  changes to their parameters that are:\n\n1. negligible, aka *linear* regime\n\n2. constant wrt. to network size aka *critical* regime\n\n3. increasing with network size, aka *condensed* regime.\n\nAnalogously to Luo et al. 2021, the above regimes are revealed to be codimension two bifurcations, controlled by quantities $\\gamma_2, \\, \\gamma_3$ defined in Eq. 9.\n\nBy carefully varying $\\gamma_2, \\, \\gamma_3$ for each hidden layer, the authors demonstrate that the weights can evolve separately and in tandem in either of the above three regimes.", "questions": "Use of cosine similarity:\nAm I correct to assume that the cosine similarity is specific to ReLU is positive homogeneous?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To the best of my knowledge, this submission gives new insight into training dynamics of MLPs with more than one hidden layer.\nThe numerical nature of the study is well complemented by its clever analysis of the bifurcation parameters.\nThis in turn allows for an elegant and insightful analysis of what I believe is considered a dauntingly difficult problem.\n\nI have not kept up to date with the most recent analyses of the gradient dynamics of shallow NNs, and hence I will defer to other reviewers for estimates of the impact of this work.\nNotwithstanding that, I greatly value the elegance and insights that this submission presents.\n\nMinor comments:\n\n1.  The manuscript would benefit from proofreading, especially in regards to small grammatical and stylistic errors.\n\n2. Figure 3: Perhaps a diverging colormap would be better suited for $D(u,v)$?\n\n3. Table 2 and Figure 4: I would like to encourage the authors to report some measure of spread for the experimentally collected quantities.\n\n4. Figures 4 & 5: please label the axis with $\\gamma_2, \\gamma_3$. I believe the captions indicating the ordinate and abscissa are inconsistent with numerical ranges explored elsewhere, e.g. in figure 6.\n\n5. Figure 3 and 8: mention or indicate graphical the expectation for $D(u,v)$ and $\\zeta$ under a Gaussian distribution over the weights", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept