Towards Understanding the Condensation of Neural Networks at Initial Training
Authors: Zhiqin Xu
Hanxu Zhou
Tao Luo
Yaoyu Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: neural networks
training
condensation dynamics
implicit regularization

Abstract: Implicit regularization is important for understanding the learning of neural networks (NNs). Empirical works show that input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term)  condense on isolated orientations with a small initialization. The condensation dynamics implies that the training implicitly regularizes a NN towards one with much smaller effective size. In this work, we utilize multilayer networks to show that the maximal number of condensed orientations in the initial training stage is twice the multiplicity of the activation function, where ``multiplicity'' is multiple roots of activation function at origin.  Our theoretical analysis confirms experiments for two cases, one is for the activation function of multiplicity one, which contains many common activation functions, and the other is for the layer with one-dimensional input. This work makes a step towards understanding how small initialization implicitly leads NNs to condensation at initial training stage, which lays a foundation for the future study of the nonlinear dynamics of NNs and its implicit regularization effect at a later stage of training.

Under review as a conference paper at ICLR 2022
TOWARDS UNDERSTANDING THE CONDENSATION OF NEURAL NETWORKS AT INITIAL TRAINING
Anonymous authors
Paper under double-blind review
ABSTRACT
Implicit regularization is important for understanding the learning of neural net- works (NNs). Empirical works show that input weights of hidden neurons (the input weight of hidden neuron consists of the weight from its input layer the hidden neuron and its bias term) condense on isolated orientations with a small initialization. The condensation dynamics implies that the training implicitly reg- ularizes a NN towards one with much smaller effective size. In this work, we utilize multilayer networks to show that the maximal number of condensed orien- tations in the initial training stage is twice the multiplicity of the activation func- tion, where "multiplicityy" is multiple roots of activation function at origin. Our theoretical analysis confirms experiments for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions, and the other is for the layer with one- one-dimensional input and arbitrary multiplicity. This work makes a step towards understanding how small initialization implicitly leads NNs to condensation at initial training stage, which lays a foundation for the future study of the nonlinear dynamics of NNs and its implicit regularization effect at later stage of training.
INTRODUCTION
Over preepparameterized neural networks often show good generalization performance on real-world problems by minimizing loss functions without explicit regularization (Breiman [9955] Zhang al. 2017). For pver-parameterized NNs, there are infinite possible sets of training parameters that can reach a satisfying training loss. However, their generalization performances can very different. It is important to study what implicit regularization is imposed aside to the loss function during the training that leads the NN to a specific type of solutions.
Empirical works suggest that NNs may learn the data from simple to complex patterns (Arpit al. 2017 Xu et al.) 2019] Rahaman et al., 2019)] Xu et al., 2020: Jin et al.. [020 Kalimeris et al, 2019 For example, an implicit bias of frequency principle widely observed that NNs often learn
the target function from low to high frequency (Xu et al. 2019 Rahaman et al. 2019: Xu et al.)
2020 which has been utilized to understand various phenomena (Ma et al.J 2020 Xu & Zhou,
2 and inspiring algorithm design Liu et al. (2020). The NN output, either simple or complex,
collective result of all neurons. The study how neuron weights evolve during the training
central to understanding the collective behavior, including the complexity, of the NN output.
Luo al. (20TT) establish a phase diagram to study the effect of initialization on weight evolution for two-layer ReLU NNs at the infinite-width limit and find three distinct regimes in the phase diagram, i.e., linear regime, critical regime and condensed regime. The non-lineat regime, a largely unexplored non-linear regime, is named as condensed regime because the input weights of hidden neurons (the input weight or the feature of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) condense on isolated orientations during the training (Luo et al., (202)). The three regimes are identified based on the relative change of input weights as the width approaches infinity, which tends to 0, (11) and +o0, respectively.
The condensation is a feature learning process, which is important to the learning of DNNs. Note that in the following, condensation accompanied by default assumption of small initializa- tion or large relative change of input weights during training. For practical networks, such as resnet 8-like (He et al. 2016) in learning CIFAR10 10, as shown in Fig. Ta) and Table T we find that

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison of common (Glorot & Bengio 2010) and condensed Gaussian initializations
on resnet 8. m (Min + mout) 1/2. Min in-layer width. Mout: out-layer width.
common
condensed
Glorot-unifome Glorot (O, )
(O,
N (0, m
N N(0 (#) )²)
Test
0.8807
0.8777
0.8816
0.8847
0.8824
0.8826
Test
0.8857
0.8849
0.8806
0.8785 0.8824
0.8813
0.8807
Test
0,8809
0.8860
0.8761
0.8861
0,8800
the performance of networks with initialization in the condensed regime is very similar to the com- mon initialization methods. However, the condensation phenomenon provides an intuitive explana- tion of the good performance as follows, which may lead to a quantitative theoretical explanation in future work. The condensation transforms a large network to a network of only a few effective neurons, leading to an output function with low complexity. Since the complexity bounds the gen- eralization error (Bartlett & Mendelson 2002), the study of condensation could provide insight to how NNs are implicitly regularized to achieve good generalization performance in practice.
For two-layer ReLU NN, Maennel et al. (2018) prove that, as the initialization of parameters goes to zero, the features of hidden neurons condense at finite number of orientations depending on the input data; when performing linearly separable classification task with infinite data, [Pellegrini & Biroli (020) show that at mean-field limit, a two-layer infinite-widit width ReLU NN is effectively equal to a NN of one hidden neuron, i.e., condensation
single orientation. Both works (Maennel et al.) 2018 Pellegrini & Biroli] 2020) study the condensation behavior for ReLU-NNs at an initial training stage which the magnitudes of NN parameters far smaller from well-fitting an O(1) target function. However, it still remains unclear that for NNs of more general activation functions, how the condensation emerges at the initial training stage.
201
30C
0.7
339
(a) test accuracy
(b) ReLU, epoch 8
(c) Laanh epoch 61
Figure 1: The test accuracy in (a) and condensation in (b, c) of networks on CIFAR1O. Each network consists of the convolution part of resnet18 and colnecco (FC) layers with size 1024- 1024-10 10 and softmax. The color in (b, e) indicates the inner product of normalized input weights of two neurons in the first FC layer, whose indexes are indicated by the abscissa and the ordinate. We discard about 55% of the hidden neurons, in which the Lg-norm of each input weight is smaller than 0.001, while remaining ones bigger than 0.05 in (b). The convolution part is equipped with ReLU activation and initialized by Glorot normal distribution (Glorot & Bengio 2010). For FC layers in (a), the activation is ReLU and they are initialized by three common methods (red) and three condensed ones (green) as indicated Table [ The learning rate 10-3 for epoch 1-60 and 10-4 for epoch 61-100. For (b, c), the learning rate is 5 x 10-6 for visualization and FC layers are initialized by (O, me out
and equipped with ReLU in (b) and x tanh(x) in (c) as activation
functions. Adam optimizer with cross-entropy loss and batch size 128 are used for all experiments.
In this work, we show that the condensation at the initial stage is closely related to the multiplicity p at x = O, which means the derivative of activation at æ 0 is zero up to the (p - 1)th-order and is non-zero for the p-th order. To verify their relation, we use the common activation func- tion sigmoid(x), softplus(z)) tanh(r), which are multiplicity one, and variants of tanh(x).ie. tanh(x) and x² tanh(r) with multiplicity two and three, for our experiments. For comparison, also show the initial condensation of Re),(z), which is studied previously (Maennel et al. 2018) and has totally different properties at origin compared with tanh(r). Our experiments suggest that
2

---

Under review as a conference paper at ICLR 2022
the maximal number of condensed orientations is twice the multiplicity of the activation function used in general NNs. For fnite-width two-layer NNs with small initialization at the initial training stage, each hidden neuron's output in a finite domain around 0 can be approximated by a p-th order polynomial and so is the NN output function. Based on the y-th order approximation, we show preliminary theoretical support for condensation by theoretical analysis for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions, and the other is for the layer with one me-dimensional input and arbitrary multiplicity. Therefore, small initialization imposes an implicit regularization that restricts the NN be effectively much narrower neural network the initial training stage.As commonly used acti- vation functions, such as (anh(v), sigmoid(#)) softplus(z)) (ar), etc., are all multiplicity one, our study of initial training behavior lays an important basis for the further study of implicit regularization throughout the training.
RELATED WORKS
research line studies how initialization affects the weight evolution of NNs with a sufficiently large or infinite width. For example, with an initialization in the neural tangent kernel (NTK) regime or lazy training regime (weights change slightly during the training), the gradient flow of infinite-wide NN, can be approximated by linear dynamics of random feature model (Jacot et al. 2018: Arora eet al. 2019; ZZann et al., 2020 E et al.) 2020) Chizat & Bach, (019)). whereas for the initialization the mean-field regime (weights change significantly during the training). the gradient flow of indhh NN exhibits highly nonlinear dynamics (Mei et al. 9019 & Vanden- Eijnden
2018 Chizat Bach 2018 Sirignano & Spiliopoulos] 2020 Pellegrini & Biroli (2020) analyze how the dynamics of each parameter transforms from lazy regime (NTK initialization) to rich regime (mean-fiel) initialization) for an two-layer infinitewwidth ReLU NN to perform a linearly separable classification task with infinite data. Luo et al. (2021) systematically study the effect of initialization for two-layer ReLU NN with infinite width by establishing a phase diagram, which shows three distinct regimes, i.c., linear regime (similar to the lazy regime), critical regime and condensed regime (similar to the rich regime), based on the relative change of input weights as the width approaches infinity, which tends to O(1) and +oc, respectively. [uo et al. (202)) also empirically find that, in the condensed regime, the features of hidden neurons (orientation of the input weight) condense in several isolated orientations, which is strong feature learning behavior, an important characteristic of deep learning, however, in Luo et al. (2021). it is not clear how general of the condensation when other activation functions are used and why there is condensation.
3 PRELIMINARY: NEURAL NETWORKS AND INITIAL STAGE
A two-layer NN is
fo(x)) = Σ a x))
(1)
1=1
where (-)) is the activation function, wy (w, by) Rd+ is the neuron feature including the input weight and bias terms, and (z, 1) d is combination of the input sample and scalar
1 1. O is the set of all parameters, i.c., {ag, w; y- For simplicity, we call wy as input weight or weight and x as input sample.
A L-layer NN can be recursively defined by feeding the output of the previous layer as the input to the current hidden layer i.e.,
11), 1), for (,,3, L} ! = (x, 1), a!'l_ = (z(Wl")))))) 1), a = (a(Wl)))) 1-1),,
f(0. x) aT=!" (D) A fz)
where Wil- (WIl, bl) E [mm and mi represents the dimension of the 1-th hidden layer. For simplicity, we also call each row of W!ll as input weight or weight and af input
3

---

Under review as a conference paper at ICLR 2022
neurons. The target function is denoted as f*)).. The training loss function is squared error
1 n (1o(m))) Rs(0)= 2n
1 (.))))
1=1
Without loss of generality, assume that the output is one-dimensional for theoretical analysis, because for high-dimensional cases, we only need to sum the components directly. For summation, it does not affect the results of our theories. We consider the gradient flow training
0 - - (O)
(4)
For convenience, we characterize the activation function by the following definition.
Definition 1 (multiplicity p)- Suppose that (xx satisfies the following condition, there exists a p E N and p 2 1, such that the k-th order derivative ((k)(0) = 0 for k = 1,2,... 1, and ((P)) (0) o 0, then we say a has multiplicity p-
In the experiments, we study the condensation at the initial stage of training. For a fixed loss, the step need to achieve it is highly related to the size of learning rate. Therefore, definition of the initial stage of training by the size of loss in this article, that is the stage before the value of loss function decays to 70% of its initial value. Such definition is reasonable, for generally a loss could decay to 1% of its initial value or even lower. The loss of the all experiments in the article be found in Appendix A.3 and they do meet the definition of the initial stage here.
propose
4 INITIAL CONDENSATION OF INPUT WEIGHTS
is intuitively believed that NNs are powerful at learning data features, which should be an impor- tant reason behind the success of deep learning. A simple way define learned feature of a neuron is by the orientation of its input weights. Previous work in Luo et al. (202I) show that there is condensed regime, where the neuron features condense on isolated orientations during the training for two-layer ReLU NNs. The condensation implies that although there are many more neurons than samples, the number of effective neurons, i.e., the number of different used features in fitting, is often much smaller than the number of samples. Therefore, the condensation provides potential mechanism that helps reeetparameteiized NNs avoid overfitting. However, it still unclear how the condensation, for general NNs with small initialization, emerges during the training. In this section, we would empirically show how the condensation differs among NNs with activation functions of different multiplicities, followed by theoretical analysis in the next section.
4.1 EXPERIMENTAL SETUP
For Synthetic dataset and MNIST: Throughout this work, fully-connected neural network with size, d-m---. -m-dout- The input dimension is determined by the training data. The output dimension is dout rons m is specified in each experiment. All parameters are initialized by a Gaussian distribution (O,, var). The total data size is n. The training method is Adam with full batch, learning rate ir and MSE loss. For synthetic data, we sample the training data uniformly from sub-domain of Rd
for synthetic data and dous 10 for MNIST. The number of hidden neu-
For CIFARIO and CIFARIOU dataset: We use Resnet 8-ikee neural network, which has been de- scribe in Fig. thoroughly. The input dimension is determined by the training data. The output dimension dout 10 for CIFARIO and dout 100 for CIFARI00 All parameters are initialized by Gaussian distribution ((0, var). The total data size is n. The training method is Adam with batch size 128, learning rate Ir and Cross-entropy loss.
4.2 MULTIDIMENSIONAL DATA
We first show the condensation at initial training stage in fitting multidimensional dataset. Since the input is multidimensional vector, the direction is also multidimensionall To characterize the condensation, we use D(u, v) to denote the inner product of the normalized vectors of two input weights, i.e., D(u, v) uTv.
4

---

Under review as a conference paper at ICLR 2022
(a) (a)
(b) tanh(a)
(c) tanh(x)
(d) (eLU(()
(c) sigmoid(a)
(f) softplus(z)
Figure 2: Condensation of two-layer NNs. The color indicates D(u, v) of two hidden neurons' input weights at epoch 100, whose indexes are indicated by the abscissa and the ordinate, respectively. neurons are in the same beige block, D(u, v) input weights have the same (opposite) direction. The activation functions are indicated by the sub-captions. The training data is 80 points sampled from Σl 3.5 sin(xi + 1), where each an is uniformly sampled from 24 n = 80, d = 5, m 50, dowt var 0.005 Ir= = 10-3000.0000 3,-8 10 2.5 x 10 for (a-d), (e) and (f), respectively.
(navy-blue block, Duu, v) ~ -1), their
We two-layer fully-connected NN with size 5-50-1 fit n 80 training data sampled from 5-dimensional function Σt=13 3.5 sin( + 1), where x (x1, I2, ... IT5) TR. and each oy uniformly sampled from [-4,2] 2]. As shown in Fig. aa) for activation function tanh(z), the color indicates D(u,2) 2)) of two hidden neurons' weights epoch 100, whose indexes are indicated by the abscissa and the ordinate. If the neurons in the same beige block, D(u, v) (navy-blue block, D(u, v) ~ -),, their input weights have the same (opposite) direction. Obviously, input weights of hidden neurons condense two opposite directions, i.e., line. As the multiplicity increasing, NNs with z tanh(x) (Fig. (()) and x² tanh x (Fig. c) condense at two and three different lines, respectively. For activation function sigmoid (a:) in Fig- d) and softplus(1) (I) in Fig. e).. which frequently used and have multiplicity one, NNs also condense at two opposite directions. For ReLU in Fig. f for which the multiplicity definition cannot apply, the NN condenses at three directions, in which two are opposite. Through these experiments, we conjecture that the maximal number of condensed orientations is twice the multiplicity of the activation function used at initial training.
1m0 MIN
(a) layer
(b) layer
(c) layer
(d) layer 4
(e) layer 5
Figure 3: Condensation of six-layer NNs with residual connections. The actívation functions for hidden layer to hidden layer 5 are x² tanh(x), tanh(r), sigmoid(x)), tanh(x) and softplus(r), respectively. The numbers steps selected in the sub-pictures are epoch 1000, epoch 900, epoch 900, epoch 1400 and epoch 1400, respectively, while the NN is only trained once. The color in- dicates D(u, v) of two hidden neurons' input weights, whose indexes are indicated by the abscissa and the ordinate, respectively. The training data is 80 points sampled from a 3-dimensional function
Σ 4 ssin + 1), where each zg is uniformly sampled from [-4, 2]. 80, d 3, 18, dant = var 0.012. Ir 4 10-5
For multilayer NNs with different activation functions, we show that the condensation for all hidden layers is similar the two-layer NNs. In deep networks, residual connection is often introduced to overcome the vanishing of gradient. To show the generality of condensation, we perform an experiment of six-layer NNs with residual connections. To show the difference of various activa- tion functions, we set the activation functions for hidden layer 1 to hidden layer 5 as x² tanh(x:), x tanh(z), sigmoid(x)) tanh(x) and softplus(z). respectively. The structure of the residual is h4+1(x) o (x) + bi) (x) where hı(x)) is the output of the /-th layer. As shown in Fig. 3. input weights condense at three, two, one, one and one lines for hidden layer to hidden not necessary. We show experiment of B but without residual connections Appendix A.6 To show the
layer 5, respectively. Note that residual connections
the same structure as in Fig. universality of condensation, train resnet18- neural networks learn CIFAR10. We study

---

Under review as a conference paper at ICLR 2022
the condensation of the first fully connected layer of the network, using ReLU and tanh (z) the activation functions and initialization distribution N [O, (,t)))). As shown in Fig. (bb) and (c), the condensations for activation (eLU(x) and tanh (x) are consistent with Fig. and our conjucture.
More experiments on dataset CIFAR 10 and CIFARIOO can be found in Appendix A.5
We also find that when the training data is less oscillated, the NN may condense at fewer directions. For example, as shown in Fig. a), compared with the high frequency function in Fig. 2 we only change the target function to be a lower-frequency function, i.e., Σ 3.5 sin + 1). In this case, the NN with x²2 tanh(z) only condenses at three directions, in which two are opposite. For MNIST data in Fig. bb,, we find that, the NN with x² tanh(x) condenses at one line, which may suggest that the function for fitting MNIST dataset is low-frequency function. For CIFARI00 data in Fig. 4c.. we find that input weights of the first FC layer with tanh(x) condense at only one line, which implies that features extracted by the convolution part of the NN may own low complexity.
Ott. vl
20
DIU
1023
No. mabo
1023
(a) Σ k=1 3.5 (2n: 1)
(b) MNIST
(c) CIFARI00
Figure 4: Condensation of now-frequency functions with two-layer NNs in (a.b) and condensation of the first FC layer of the Resnet18 8-like network on CIFAR 100 in (c). The color indicates D(u, v) of two hidden neurons' input weights, whose indexes are indicated by the abscissa and the ordinate, For (a.b), two-layer NN at epoch: 100 with activation function: x² tanh(x). For (a), we discard about 15% of hidden neurons, in which the I--norm of each input weight is smaller than 0.04, while remaining those bigger than 0.4. The mean magnitude here for each parameter is (0.44/785)0.5 ~0.01, which should also be quite small. All settings in (a) are the same as Fig. 2 except for the lower frequency target function. Parameters for (b)
60000, d 784, m 30, dout 10, var = 0.0012. Ir = 5 x 10-5. The structure and parameters of the Resnet1 8- neural network for (c) the same Fig. 1 bb), except for the data CIFARI00 and learning rate ir 1 10-6
To understand the mechanism of the initial condensation, we turn to experiments of I-d input and two-layer NN, which can be clearly visualized in the next subsection.
4.3 1-D INPUT AND TWO-LAYER NN
For I-d data, visualize the evolution of the two-layer NN output and each weight, which confirms the connection between the condensation and the multiplicity of the activation function.
-10
(a) tanh(x)
(b) tanh(()
(c) x² tanh(())
(d) ReLU(())
Figure 5: The outputs of two-layer NNs at epoch 1000 with activation functions lanh(x), a tanh(2r), x² tanh(r), and ReL.(xx) are displayed, respectively. The training data is 40 points uniformly sampled from sin(3x) + sin(6x)/2 with x E [-1, 1.5). illustrated by green dots. The blue solid lines are the NN outputs test points, while the red dashed auxiliary lines the first, second, third and first order polynomial fittings of the test points for (a, b. c, d), respectively. Parameters are 40, 1, m 100, dout var 0.0052 Ir 10

---

Under review as a conference paper at ICLR 2022
We display the outputs at initial training, epoch 1000, in Fig. 5 Due the small magnitude of parameters, an activation function with multiplicity can be well approximated by a p-th order polynominal, thus, the NN output can also be approximated by p-th order polynominal. As shown in Fig. 5fa-c), the NN outputs with activation functions tanh(r), x tanh(z) and x² tanh(x) overlap well with the auxiliary of a linear, quadratic and a cubic polynominal curve, respectively. In Fig. djd), the NN output with ReLU activation function deviates from linear function (red auxiliary line). Particularly, the NN output has several sharp turning points. This experiment, although simple, but convincingly shows that NN does not always learn a linear function at the initial training stage and the complexity of such learning depends on the activation function.
(a) tanh(x)
(b) tanh(2)
(c) x tanb(x)
(d) ReLU(())
Figure 6: The direction field for input weight All settings are the same as Fig. S Around the original point, the field has one, two, three stables lines, on which input weight would keep its direction, for tanh(x), tanh(x)) and x² tanh(x), respectively. We also display the value of each weight by the green dots and the corresponding directions by the orange arrows.
(w, b) of the dynamics in 4.3 at epoch 200.
We visualize the direction field for input weight w := (wy, by). following the gradient flow,
aj n
n
(ww x.)
Since we only care about the direction w and aj is a scalar at each epoch, we can visualize uł by ni./ay. For simplicity, we do not distinguish wiz/a, and wi, if there is no ambiguity. When we compute w for different j's, e, for (i = 1,. ..... 12) is independent with Then, at each epoch, for set of {e,, x., Ji we can consider the following direction field
where e; = [o(::)
e, (w x2).
When w is set as Wj. we can obtain wjj. As shown in Fig. 6 around the original point, the field has one, two, three stables lines, on which neuron would keep its direction, for tanh(xr), x tanh(xr), and x² (z). respectively. We also display the input weight of each neuron on the field by the green dots and their corresponding velocity directions by the orange arrows. Similarly to the high-dimensional cases, NNs with multiplicity p activation functions condense reasonable to conjecture that the maximal number of condensed orientations is twice the multiplicity of the activation function used. As shown in Fig. d(), the field and the condensation for the NN with (x) is much more complex.
different lines
for p = 1, 3. Therefore,
Taken together, we have empirically shown that the multiplicity of the activation function is a key factor that determines the complexity of the initial output and condensation. In figures this see- tion. only show the results of the final step of condensation. To facilitate the understanding of evolution of condensation in the initial stage, we show several steps during the initial stage of each example in Appendix A.4 From the evolution, could directly observe how condensation occurs.
5 ANALYSIS OF THE INITIAL CONDENSATION OF INPUT WEIGHTS
In this section, we would present preliminary analysis to understand how the multiplicity of the activation function affects the initial condensation. At each training step, we consider the velocity field of weights in each hidden layer of neural networks.
Considering network with hidden layers,
vector W! [k] to represent the weight from the (k: layer to the j-th neuron in the k-th layer. For each is and j. w satisfies the following
7

---

Under review as a conference paper at ICLR 2022
dynamics, (see Appendix A.2
i = u w ú ii (eb u u)e
(5)
where w can be w/ for all k's and j's, r = w la is the amplitude, and u = w/r.
Suppose the activation function has multiplicity p. i.e., (0) 0 for k 1, ... P and
(P)) (0) * O. For convenience, define operator P satisfying Pw w ui ( u). Conden- sation refers to that the weight evolves towards direction that will not change the direction field and is defined follows,
-= + Pw = u(u 2) 0.
Since ui is scalar, un is parallel with u. u is unit vector, therefore, we have u u= / |||| In this work, we consider NNs with sufficiently small parameters. For example, suppose r = 2l/2 ~ O(e). is small quantity. Dynamics (5) shows that O(i) O(w) and O(ū) (fe) Therefore, the orientation u would moves much more quickly than the amplituder: r. In the following, study the case of (i) p 1 and (ii) my = 1.
5.1
CASE 1: p=1
Since we have (see Appendix A.2
wT w M
(f(0, x.) yi) [diaglo' [diaglo'(W/thlg [k- (E [h-1jT
where we use Ex W diag (o ( |)),, for E (2, .... L}. Eirrl E E".
For fixed step, we only consider the gradient w.r.t. Wh!h Suppose o'()) o 0 and parameters are small. Denote == (f(0, x.)) v:). By Taylor expansion,
Pw
leading order &
Qw
1
(E))
a)); Σ e,a [k-1]
( (diag (0)) (1+t:: a));
+ (
Σ
c.a0 [--1]
0,
where operator @ the leading-order approximation of operator P. and here 1 is indepen- dent with i because diag [W-') x diag(o' (0)). Since diag(o' I, and, WLOG, we assume &f 0, then
n1
n
C.a k-
[k
= +>
eya
2 u.
We have
Σn
C: [k-1]
Σn C
[k-1]
|> c,20 10
122
u
>> C 112
This calculation shows that for layer k, the input weights for any hidden neuron j have the same two stable directions. Therefore, when parameters are sufficiently small, which implies that the orientation u would moves much more quickly than the amplitude r, all input weights towards converging to the same direction the opposite direction, i.e., condensation on line.
5.2
CASE 2: my
By the definition of the multiplicity p. we have
((r) (0)
d' d(w x.) = (p 1)!
(w ); 11 + o((w x;)
8

---

Under review as a conference paper at ICLR 2022
where (.)-² and o'(.) operate component here. Then up to the leading order in terms of the magnitude of 0, we have (see Appendix A.2
Pu
leading order
a Qw
c:a
k- (l (w a
|e- (0-1)p-11)
[diag(
(p 1)! }(
o [p) (0)
(F)
n
Ik (w x;; -t- 1 Idiag| (0) } (Elkt2)L)a)) a)]; u)u.
+((,
e
(p 1)!
i=1
WLOG, also assume + O. And by definition, ru, have
1 Σ e [k 11 u1 x 10 1p11
Qw =
u
1 -=1 c,
k- 11 (uuzz To Mk- 11 1p- 112
1
E 1== C
[k (1 a - 11 1
x
or Ei= e,a,
[8-11 11
(utze [k 11 1--21111 1|2
Since d + 1 2, we denote u = (u1, u2) E R2² and a; [k-1] = ((xg) ( ) (2.
[ )2)
)2) E R?, then,
(u ( + u2 nt- 11 (
e, (z |k- [ 111
11 a a.
Em (1y (2) | )1 + 22 11) -2) (z) hh -
02
We obtain the equation for ú,
> [(a(z!--)) + (a!4-"l)>)) )2) - () = à > (1(z) (a(a!--1)) + (2:!t-1)) 2e (e/(t
Since it is an univariate p-th order equation, û = has at most complex roots. Because u is unit vector, u at most has pairs of values, in which each pair are opposite.
Taken together, our theoretical analysis is consistent with our experiments, that is, the maximal num- ber of condensed orientations is twice the multiplicity of the activation function used when parame- ters are small. Besides, because commonly used activation functions, such as tanh(x). sigmoid(zr), softplus(1)) (x), etc., are all multiplicity one, the theoretical analysis sheds light on practical training.
DISCUSSION
In this work, have shown that the characteristic of the activation function, i.e., multiplicity, is a key factor to understanding the complexity of NN output and the weight condensation at initial training. The condensation restricts the NN be effectively low-capacity the initial training stage, even for finite-width NNs. During the training, the NN increases its capacity to better fit the data, leading to potential explanation for their good generalization in practical problems. This work also serves as a starting point for further studying the condensation for multiple-layer neural networks throughout the training process.
As the scale of parameter initialization becomes larger, the condensation becomes weaker. The un- derstanding from a complete condensation would benefit our understanding the training process in common initialization, that is there is an effect, although not so strong, that can limit the complexity of the neural network in the initial training.
How small the initialization should be in order to a clear condensation is studied in Luo et al. (2021) for two-layer ReLU NNs with infinite width. For general activation functions, the regime of the initialization for condensation depends on the NN width. A further study of the phase diagram for finite width NNs would be important.
For general multiplicity with nigh-dimensional input data, the theoretical analysis for the initial condensation is a very difficult problem, which is equivalent to count the number of the roots of a high-order high-dimensional polynomial with special structure originated from NNs.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 8139 8148, 2019. URL htttp:::pppocceeeiiiiiee Abst html
www. wttpppppee/////aaaaaeeeepppep/ee
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look memorization in deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6- 11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 233-242, PMIR, 2017. URL httt:::/procceddngss. r.press/v00/ arpit Erpit170 7a html]
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
Leo Breiman. Reflections after refereeing papers for nips. The Mathematics of Generalization, XX: 11-15, 1995.
Lénaïc Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 3040-3050, 2018.
Lenaic Chizat and Francis Bach. A note on lazy training in supervised differentiable programming. In 32nd Conf. Neural Information Processing Systems (NcurIPS 2018), 2019.
Weinan E, Chao Ma, and Lei Wu. A comparative analysis of optimization and generalization prop- erties of two-layer neural network and random feature models under gradient descent dynamics. Science China Mathematics, pp. 1-24, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings the thirteenth international conference on artificial intelligence and statistics, pp. 249-256,, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianch and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018 December 3-8, 2018, Montréal, Canada, pp. 8580, 2018 URL
neurips cc cc//paper/201//has.. .htll
Pengzhan Jin, Lu Lu, Yifa Tang, and George Em Karniadakis. Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness. Neural Networks, 130:85 99, 2020.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin L. Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of in- creasing complexity. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3491 3501, 2019. URL httpp//ooo...tttt////oooo.ooooo. nenniips htt.a//wwppp..../ppp/////////aaa bol.
10

---

Under review as a conference paper at ICLR 2022
Ziqi Liu, Wei Cai, and Zhi- Chi-Qin John Xu. Multi-scale deep neural network (mscalednn) for solving poisson- bessoonbblltzman equation in complex domains. Communications in Computational Physics, (8(5):1970-2000 2020.
Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer relu neural networks at infthite-width limit. Journal of Machine Learning Research, (2(71):1-47 2021.
Chao Ma, Lei Wu, and E Weinan. The slow deterioration of the generalization error of the random feature model. In Mathematical and Scientific Machine Learning, pp. 373-389, PMLR, 2020.
Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly. Gradient descent quantizes relu network features. arXiv preprint arXiv: 180.0888000036.,, 2018
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension- -ree bounds and kernel limit. In Alina Beygelzimer and Daniel Hsu (eds.), Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA, volume 99 of Proceedings of Machine Learning Research, pp. 2388-2464. PMLR, 2019. URL http:// proceedings mlr. www.ppresssvv.e. /ttp//9999mm///// m/ 1itee111. html
Franco Pellegrini and Giulio Biroli. An analytic theory of shallow networks dynamics for hinge loss classification. Advances in Neural Information Processing Systems, 33, 2020.
Nasim Rahaman, Devansh Arpit, Aristide Baratin, Felix Draxler, Min Lin, Fred A Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of deep neural networks. International Conference Machine Learning, 2019.
Grant M. Rotskoff and Eric Vanden Vanden-Eijnden. Parameters as interacting particles: long time con- vergence and asymptotic error scaling of neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnetr (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7146-7155, 2018. URL httttss://ppoceeetttitt/ wwwpeeiiiinnnneeesseeeeeeee.eeee scc/paper/211/hhatt. Abss hhtm
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: central limit theorem. Stochastic Processes and their Applications, 12003):820-1852, 2020.
Zhi-Qu Qin Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. International Conference on Neural Information Processing, pp. 264-274, 2019.
Zhi-Qin John Xu and Hanxu Zhou. Deep frequency principle towards understanding why deeper learning is faster. In Proceedings of the AAÀI Conference on Artificial Intelligence, volume 35, 2021.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics, 18(5):174611767 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL /tttps://ppenzviiiie
net /wt//forroroeeee /oofforuunn Syd 9xx
Yaoyu Zhang, Zhi-Qin John Xu, Tao Luo, and Zheng Ma. type generalization error induced by initialization in deep neural networks. In Mathematical and Scientific Machine Learning, pp. 144-164. PMLR, 2020.
11

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 BASIC DEFINITIONS
In this study, we first consider the neural network with 2 hidden layers.
A two-layer NN is
m
fo(x) Σ a x),
(6)
where o--) is the activation function, Wi = (wj, bj) E RR+1 is the neuron feature including the input weight and bias terms, and x = (æ; 1) E IR++1 is combination of the input sample and scalar
1, 0 is the set of all parameters, i.e., {az. w/ ==1 For simplicity, call wy input weight
weight and a as input sample.
Then, consider the neural network with hidden layers,
x = (x, 1), ! (o(wi)) ),) 1), a = x 1), for (,,3, L} f(0,æ) -a (b) fo(x).
where W = (WIl, bl4) R(mxxm-))), and my represents the dimension of the 1-th hidden layer.
The initialization of W k,kr (1,2,3.... L} and a obey normal distribution w!l k,k for E {1, } and ay ~ NOO, (7.)).
NNO,, 88)
The loss function is mean squared error given below,
1 2n
(O)
Σ (Jo(x;) -1)2
(8)
For simplification, we denote (a( as f in following.
A.2 DERIVATIONS FOR CONCERNED QUANTITIES
A.2.1 NEURAL NETWORKS WITH THREE HIDDEN LAYERS
In order to better understand the gradient of the parameter matrix of the multi-layer neural network, we first consider the case of the three-layer neural network,
(x a aTo aTo(W
(9)
with the mean squared error the loss function,
1
(
Σ (fo(z)) - 3.)². 2n
(10)
1=1
df df and awni respectively, using differential form, We calculate awla
0f Jd)) dʃ = ir((
(11)
0x.
We consider df first, aW.
12

---

Under review as a conference paper at ICLR 2022
df = (
(W o x }
(12)
(W "
(a (a (w (w
tr((((a (Wilz(")) "aawwlll)
Tdwll))
where @ is Hadamard Product, and it is the multiplication of matrix elements of the same position.
Hence,
df = (a @ 2 dW 2]
(13)
= diagfo (W x ax
Then, we consider
df aWa.
df tri(a @ (w (W
(r((WP)) ( (W (w e d(w(l)))
(Wlll))
(14)
(e(((W))(aadd ) oo
tr(ll((W() (a
od ["T
Hence, we have,
df (w (a e (w ( o (Wll)))) dW
(15)
diag(o (will w 22 diag (w x Jax
Through the chain rule, we can get the evolution equation of W and W
dw
dR. (0)
dt
dWI[]
1
df
(f(0, x.. ) - y.) dwil
(16)
n
Σ (S(0, ..) yi vi)diag(o (wl'l,;;) w(2) ) ayy'Tdiaggo 'iag(o" (WIP)
Jao.
and
2 2]
(R)
dt
dW3
1 a
(f(0, xx)) yi) df
dw
(17)
n
(f(0 y1 vi)diagfo (wlllz""))azz)
A.2.2
L HIDDEN LAYERS CONDITION
And, consider the neural network with J. hidden layers,
13

---

Under review as a conference paper at ICLR 2022
df tr(da ((ddadd((W))))ll1)))
tr(((a o o (ww
=tr((wi AL) 2 2, z 1011-111 3}
-tr(l(nn Az) o |
8/1 (18)
( W -
Ir(AAdWlli x 1 =tr((A R- 1) ydd
where A, A (Wi++IJ A1+1)) o (Will for I k 1. and A - (Willlll-11)
Hence, we get,
df 1-
(19)
dWR
Through the chain rule, we can get the evolution equation of Wik)
dw [k]
dR2
dW
(f(0.x,) v.) df dw A
(20)
(f(0 11)A 1-11
Through o' (WW)) diaglo (Wæ)Ja Finally, the dynamic system can be obtained:
da
[4] (S(0, - v/))
di x
W14 dw(M
diag lo' (loooollll (للااا Jaz!'-1) (f(0,x;) - -y)),
dt T
dW [k]
WM
W
diaglo (wollwllll |t- 1 am! [k-1]1 (f(0,z;)) v.) vi L 1],
dt
(21)
where we use Fx))) = W diag (W And Elael E9 Ee++...FP E..
Let Tk.j
W.!k [k]
2-
We have
d w 1|2.
d di
11
(22)
di
Then we obtain
Pk. wilh| [k] wh!h!
(23)
Finally, we get
Fk.s = drk.j ww!t [+] w [k] /TkJ
dt w [k] ux.j.
(24)
14

---

Under review as a conference paper at ICLR 2022
W!ll rk.
where uk.
unit vector. Then have,
Ww!
duRj
d
Wk.j
dt
di
/ [M]
W
(25)
w/ Mk.
w (W!! uk.s)
W.l
uk (W,
[1] ukss))
Tk.j
To conclude, the quantities we concern are summarized as follows,
1
[ (f(0, yi)
à
(26)
iml
W114]
diag a Jax!"-11
|1- (f(0, xx;) yi).
(27)
72
WMl
1
diag (o (W 1/ jk- be batt |- (f(0, xx)) vi) Vi [1: [ (28)
Y1
jk.j = W!!!! uk.j
(29)
W(M M
uk . Wi(M [M] uKs)
(30)
Tk.j
where we use (xx) = W (ollfo((W!' And Elaol E9 g E.
A.2.3 PROVE FOR Pw IN 5.2
leading order &
We calculate Pw
Qw following,
15

---

Under review as a conference paper at ICLR 2022
Pw=
1
e
[--1] 11 [diag (o (W xx (E
1=1
[k 11 Idiag( 10 (W xo ) (E u)u
+
c
( (0)
R- (diag(
w (( (11)+
c:ai 71
(p - 1)!
[k-1] 11 diagf σ(0) (0)
( [4l -k- 1111-11111 (E a]]; u)u
12
1)!
o (0)
1) [diag ( (W x |-- 111-1 } diag
e
(v) (0)
[k- 11 Idiag/ (W|M) 1 p-1 I diag
[+1 ر]]; u)u
e
11
1}!
i=1
|--1] 1] Idiag ( 2x (t 11p- "I, diag (())(0)
| (+++))))))) a)
e 71
(p 1)!
(r)
|
1| (diag( (oogg((Wllk)) [1 [t 11 1 HB diag
[[3 uiu
e
(p 1)!
(p)
e:as Ik W [k] xp
11 1 (diag|
(E
1)1
1=1
+ ((
1
[k-1] 1|
(w!hi !b! -ل - ddaag
(
(p) (0)
(F [1 a]]; u)u,
n
(p- - 1)!
(31)
where (.)-1 and (.) operate on component here.
16

---

Under review as a conference paper at ICLR 2022
A.3 THE VERIFICATION OF THE INITIAL STAGE
We put the loss of the experiments in the main text here to show that they are indeed in the initial stage of training by the definition.
As is shown in Fig 2 and Fig 8 at the number of steps drew the graphs in the article, loss satisfies the definition of the initial stage, we consider that they are in the initial stage of training.
Only the epoch number could hardly reflect the initial stage, which also depends on the learning rate. Therefore, we use the loss to indicate the initial stage. Learning rate is the appearance of condensation. However, a small learning rate enables us to clearly observe the condensation process in the initial stage under a gradient flow training. For example, when the learning rate is relatively small, the initial stage of training may be relatively long, while when the learning rate relatively large, the initial stage of training may be relatively small.
sensitive to
We empirically find that to ensure the training process follows gradient follow, where the loss decays monotonically, we have select a smaller learning rate for large multiplicity p. Therefore, looks like we have a longer training in our experiments with large p. Note that for small learning rate in the experiments of small p. we can observe similar phenomena.
In all subsequent experiments in the Appendix, we will no longer show the loss graph of each experiment one by one, but we make sure that they are indeed in the initial stage of training.
17

---

Under review as a conference paper at ICLR 2022
1055
2.
$3..2
2.0
10.00
18
1634
1.8
1.6
10
20
40
epochs
epochs
epochs
(a) Fig lbb Step 8
(b) Fig c) Step 61
(c) Fig 2 a) Step 100
333.30
1017
A31-44
195
38x
epochs
epochs
epochs
(d) Fig bb Step 100
(c) Fig c) Step 100
(f) Fig 2 d) Step 100
Ioss
lss
kos
1140
12
$31.3
10
spachs
opochh
hrs
(g) Fig e) Step 100
(h) Fig 1 Step
Fig Step
1400
17917
8CC
4.-
4.4
150
250
130
280
epochs
Fig
Step 100
(k) Fig
Step 100
Fig
c) Step 250
Figure 7: Losses from Fig. to Fig The original pictures and the numbers of steps corresponding each sub-picture are written in the sub-captions.
18

---

Under review as a conference paper at ICLR 2022
kno
14410
l00s
52.10
27.12
200
450
600
1000
epochs
(a) Fig a) Step 1000
(b) Fig b) Step 1000
11..5
20.0
24:15
135
274.7
14.
epochs
epochs
(c) Fig 5 c) Step 1000
(d) Fig
d) Step 1000
5.m vgo
5350000
1055
5200 3on
100
200
epochs
epachs
(e) Fig 6(a) Step
Fig bb Step
Ioss
kno
2200117
22-1177
1055
5x12
100
(g) Fig 6 Step 200
(h) Fig d Step 200
Figure 8: Losses from Fig. to Fig 6 The original pictures and the numbers of steps corresponding each sub-picture are written in the sub-captions.
19

---

Under review as a conference paper at ICLR 2022
A.4 SEVERAL STEPS DURING THE EVOLUTION OF CONDENSATION THE INITIAL STAGE
In the article, we only give the results of the last step of each condense, while the details of the evolution of condensation show these details in Fig. Fig. [O] Fig. and Fig. [2] which also further illustrate the rationality of the experimental results and facilitate the understanding of the evolution of condensation in the initial stage.
lacking, which may provide better understanding. Therefore,
05.14
2AAM
(a) Step
(b) Step
(c) Step
(d) Step 6
(e) Step 8
100
f Step 30
gg) Step 51
(h) Step 54
i Step 58
Step 61
100
102
1000
(k) Step 150
(l) Step 215
(m) Step 220
(n) Step 225
(0) Step 250
(p) Step 5
<q) Step 10
(r) Step 15
(s) Step 50
(L) Step 100
(u) Step
(v) Step
(W) Step 15
(x) Step 50
yy) Step 100
Figure 9: Evolution of condensation from Fig. [b) to Fig. b) and Fig. D c). The evolution from
the first row to the fifth row are corresponding to the Fig. lb). Fig. [c) Fig. cc), Fig. a Fig.
b(.). The numbers evolutionary steps are shown in the sub-captions, where sub-figures in the last
row are the epochs in the article.
20

---

Under review as a conference paper at ICLR 2022
(a) Step 10
(b) Step 20
(c) Step 40
(d) Step 60
(c) Step 100
Nec
1m
Step 10
Step
(h) Step
Step 60
i Step 100
Heviddo
130
(k) Step
Step 20
(m) Step 40
(n) Step
(0) Step 100
New
km NRM
(p) Step
(g) Step
Step
Step
(L) Step
(u) Step
(v) Step
(w) Step 40
(x) Step
(у) Step 100
Figure 10: Evolution of condensation from Fig. to and Fig. a). The evolution from the
first row to the fifth row are corresponding to the Fig. c Fig. d), Fig. 2e.) Fig. 21., Fig. fa).
The numbers of evolutionary steps are shown in the sub-captions, where sub-figures the last row are the epochs in the article.
21

---

Under review as a conference paper at ICLR 2022
(a) Step 200
bb) Step 400
(c) Step 600
(d) Step 800
(c) Step 1000
Step 200
g Step 400
(h) Step 600
Step 800
i Step 900
Neu
130
(k) Step
(1) Step 200
(m) Step 600
(n) Step 800
(0) Step 900
km MRM
(p) Step 300
(q) Step
r Step 800
(S) Step 1000
(t) Step 1400
Step 100
(v) Step 500
(w) Step 900
(x) Step 1000
(y) Step 1400
Figure 11: Evolution of condensation from Fig. to e). The evolution from the first the
fifth row are corresponding to the Fig. aa), Fig. 3 b). Fig. 3). Fig. dd) Fig. 3(e). The numbers
of evolutionary steps are shown in the sub-captions, where sub-figures the last row are the epochs in the article.
22

---

Under review as a conference paper at ICLR 2022
(a) Step 20
(b) Step 40
(c) Step 60
(d) Step 80
(c) Step 100
(f) Step 40
gg) Step
(h) Step 120
ii) Step
i Step
(k) Step
Step
(m) Step 120
(n) Step 160
(0) Step 200
(p) Step
(q) Step
(1) Step
(s) Step 160
(1) Step
(u) Step
(v) Step
(w) Step 120
(x) Step 160
(y) Step 200
Figure 12: Evolution of condensation from Fig. to d) and 4 (b). The evolution from the first
row to the fifth row are corresponding to the Fig. b). Fig. 6 a)), Fig. (b. Fig. ((c), Fig. ((.). The numbers of evolutionary steps shown in the sub-captions, where sub-figures in the last row the epochs in the article.
23

---

Under review as a conference paper at ICLR 2022
A.5
THE EXPERIMENTS ON CIFAR 10 AND CIFAR 100 WITH RESNET I 8 LIKE NEURAL NETWORK
The condensation of the Resnet18-like neural network CIFARIO and CIFARIO0 is shown in Fig. 13 and Fig. 14 whose activation functions for fully-connected (FC) layers are lanh(x), (e)
sigmoid (z). softplus(x) and (anh(x). indicated by the corresponding sub-captions, respectively.
180
km
100
300
120
(a) (a))(()
(b) (e)U(()
(c) sigmoid (a)
(d) soft slflplusaz)
(e) Lanh(2))
Figure 13: Condensation of Resmetl8-like neural networks on CIFARIO. Each network consists of the convolution part of resnet18 and fully-connected (FC) layers with size 1024-1024-10 and softmax. The color in figures indicates the inner product of normalized input weights of two neurons in the first FC layer, whose indexes are indicated by the abscissa and the ordinate, respectively. We discard the hidden ncurons, in which the L2-norm of each input weight is smaller than 0,001, while remaining ones bigger than 0.05 in (b). The convolution part is equipped with ReLU activation and initialized by Glorot normal distribution (Glorot & Bengio 2010). The activation functions are (2)) ReLU((::, sigmoid(x)), softplus(2)) and tanh(x) for FC layers in (a), (b), (c), (d) and (e), separately. The numbers of steps selected in the sub-pictures are epoch 20, epoch 8, epoch 30, epoch 30 and epoch 61, respectively. The learning rate is 10 5 x 10-6, 1 x 10-8 1 10-8 and 5 x 10-6, separately .he FC layers are initialized by (O, mous
and Adam optimizer with
cross-entropy loss and batch size 128 are used for all experiments.
100
1:0
100
Ne: Indo 300
100
(a) (a))(x))
(b) ReLU(()
(c) sigmoid(a)
(d) softplus(x)
(e) tanh(z)
Figure 14: Condensation Resnet18-Iie 8-like neural networks on CIFAR100. Each network consists of the convolution part of resnet18 and fully-connected (FC) layers with size 1024-1024-10 and softmax. The color in figures indicates the inner product of normalized input weights of two neurons in the first FC layer, whose indexes are indicated by the abscissa and the ordinate, respectively. We discard about 15% the hidden neurons, in which the Lg-norm of each input weight is smaller than 0.0001, while remaining ones bigger than 0.0025 in (b). The convolution part is equipped with ReLU activation and initialized by Glorot normal distribution (Gilorot & Bengio 2010). The activation functions
tanh ReI.U(x), sigmoid(xr) softplus(I)) and x tanh(x)) for FC layers in (a), (b), (c), (d) and (c), separately. The numbers of steps selected in the sub-pictures are epoch 10, epoch 10, epoch 10, epoch 10 and epoch 250, respectively. The learning rate 1 10 1 10-7 3 x 10-8, 8 3 x 10-8 and 1 x 10-6 separately. The FC layers are initialized by N(0, m3 ), and
Adam optimizer with cross-entropy loss and batch size 128 used for all experiments.
24

---

Under review as a conference paper at ICLR 2022
A.6 MULTILAYER EXPERIMENTAL
The condensation of the six layer without residual connections is shown in 15 whose activation functions for hidden layer hidden layer 5 are x² (anh(x), (anh(x)) sigmoid tanh(x) and softplus(x)) respectively.
The condensation of the three layer without residual connections is shown functions are same for each layer indicated by the corresponding sub-captions.
16 whose activation
The condensation of the five layer without residual connections is shown in 17 whose activation functions are same for each layer indicated by the corresponding sub-captions.
The condensation of the five layer with residual connections is shown in tions are same for each layer indicated by the corresponding sub-captions.
18 whose activation func-
(a) layer
(b) layer 2
(c) layer 3
(d) layer 4
(c) layer 5
Figure 15: Condensation of six-layer NNs without residual connections. The activation functions for hidden layer to hidden layer 5 x2 lanh(x), lanh(x), sigmoid(xr), tanh(x) and softplus(r).. respectively. The numbers of steps selected in the sub-pictures are epoch 6800, epoch 6800, epoch 6800, epoch 6800 and epoch 6300, respectively, while the NN is only trained once. The color indicates Duu, v) of two hidden neurons' input weights, whose indexes are indicated by the abscissa and the ordinate, respectively. The training data 80 points sampled from a 3-dimensional function
Ex=1 4 (innn111i) + 1), where each va is uniformly sampled from [-4, 2]. n S0, d 3, m 18, dous var 0.008?, Ir 5 5
(a) tanh(z)
(b) tanh(x)
(c) x² tanh(z)
(d) ReLU(x)
(c) sigmoid(z)
(f)
softplus(1)
(g) tanh(x)
(h) tanh(x)
x² tanh(z)
(j) ReI.U(z))
(k) sigmoid(z)
(l) softplus(z)
Figure 16: Three-layer NN at epoch 700. (a-f) are for the input weights of the first hidden layer and (g-l) are for the input weights of the second hidden layer. The color indicates D(u, v) of two hidden neurons' imput weights, whose indexes The training data is 80 points sampled from 5-dimensional function Σ k- s (8n(82 + 1), where each 2₁ is uniformly sampled from [-4, 2].
indicated by the abscissa and the ordinate, respectively.
80, d 5, m Ir = 10 10-4,02 2 x 10 1.4 x 10 for (a-d), (c) and (f), respectively. For (d) and (i), we discard hidden neurons, whose T-rm of its input weight smaller than 0.1.
50, dout
var 0.005?.
25

---

Under review as a conference paper at ICLR 2022
Nou Indec
-00
Nea Inde
17
-00
New Indes
-19
Nuu IOcKS
27
(a) (a))(2)
(b) tanh(z)
(c) tanh(x)
(d) tanh(x)
Nou
-EA
Intec
(e) x tanh (a)
1 I tanh(x)
(g) I tanh(w)
(h) a tanh (z)
Index
Indox
tan)(x))
I tanl(
(k) tanh(r)
x² tanh (2)
Disuut
Dus
Diu.dl
Duast
Nos
New
(m) (agmoid(xr)
(n) sigmoid(x)
sigmoid(x)
(p) sigmoid(a)
Not nbOC
Nou vodox
20
Nou
(q) softplus(z)
ftpluss)
(s) softplus(z)
so softpuus(x)
Figure 17:
Five-layer NN. The first fourth columns of each for the input weights of neurons from the first to the fourth hidden layers, respectively. The color indicates D(u, v) of two hidden neurons' input weights, whose indexes are indicated by the abscissa and the ordinate, respec- tively. The training data is 80 points sampled from 5-dimensional function Σ 3 + 1), where each is uniformly sampled from [-4,2] 2]. n
80, d = 5, m = 18, dout = 1, var 0.008 Ir 1.5 10-5 1.5 x 10 5 1.5 10-5 1.5 10 1.5 10-6 and epoch is 10000, 10000, 26000, 10000, 20000 for tanh(x), stanh(x), x tetonn(()), sigmoid(x), so noftplus(x). respectively.
26

---

Under review as a conference paper at ICLR 2022
Diudd
Noa NoOC
Nou hobo:
Nou hode:
+85
-43
445
(anh(2)
tanh(z)
(c) tanh(x)
tanh(2)
Pau
Tix
800
ot tanh(ar)
tanh(z)
tanh(a)
(h) tanh(2)
Neu Index
ii 2 tanh(z)
x tanh(x)
(k) x tanh(x)
x tanh(x)
3Y
Noatnter
Into
Now
Neu Indec
New Indec
Nau Inte
New Iosmc
(m) sigmaidd(x) (z)
(n) sigmoid(x)
(0) sigmoid(x)
(p) sigmoid(x)
Nou HOCK
modox
Neu 193Ce
New 19304
New 193e4
New Hide4
(q) sof (xf
(r) sof 8o (f)
(s) sof (oftplus())
(1 softplus(t)
Five-layer NN. The first to fourth columns of each row are for the input weights of neurons from the first to the fourth hidden layers, respectively. The color indicates D(u, of two hidden neurons' input weights, whose indexes are indicated by the abscissa and the ordinate, respec- tively. The training data is 80 points sampled from 5-dimensional function Σd_ 3 sin (10z, + 1), where each n is uniformly sampled from [-4, 2]. = 80, d 5, m = 18, dout 1, var = 0.008 Ir 1x 14- 1 1x10-4 5x10-3 5x10 10 and epoch is 400, 400, 400, 3000, 360, 400 for tanh(x), æt)nk(()), ttnnh(()) x²tanh(x)), sigmoid(1r) coftplus(z)) respectively.
Figure 18:
27


Reviews:
[{"review_id": "wl7BW1QbPT", "reviewer": "Reviewer_ejGJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- Provide an interesting explanation of the condensation mechanism through the lens of multiplicity of activation functions\n\nWeaknesses: \n- Theoretical analysis is limited to the case of one-dimensional inputs and activation functions of multiplicity one \n- Experimental demonstrations are somehow limited -- could be further improved by considering how similar phenomenon occurs for a range of other datasets and for other loss functions (cross-entropy, etc.)\n\nMinor comments/questions:\n- the focus seems to be on how condensation appears during the initial training stage, but there is no discussion of how initial this stage is (in particular, the selection of the number of epochs in the plots shown in the paper seems unjustified)\n- how does the learning rate play a role in the appearance of condensation? Would be nice to at least mention this", "summary_of_the_paper": "This paper studies the role of activation functions (via their multiplicity at the origin) in the condensation of neural networks at the initial training stage. Condensation can be viewed as a feature learning process, where the wide network can be described effectively as a narrower network and the input weights condense on isolated orientations during training. This mechanism may provide a plausible explanation for the performance of the wide network. In particular, the paper shows empirically that the maximal number of condensed orientations is twice the multiplicity of the activation function used. Moreover, using polynomial approximations, the paper provides theoretical support in two cases: when the activation function is of multiplicity one and when the input is one-dimensional. ", "summary_of_the_review": "This paper provides a study of the role of multiplicity of activation functions in the condensation mechanism of neural network training, claiming that the maximal number of condensed orientations in the initial training stage is twice the multiplicity. Some experimental demonstrations are provided to support the claim but the theory is somehow limited. I believe that the paper could be further improved and will go with a weak reject for now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "as5XaNl0Kim", "reviewer": "Reviewer_KucV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper aims at giving an intriguing analysis of the properties of NN activations in early stages of learning, which would motivate the intrinsic generalization abilities of deep neural architectures (e.g., when over parametrized).\nThe paper includes both an empirical and a theoretical analysis.\nMy major concern with this paper is that I found the experimental session not fully convincing. First, the analysis is limited to a set of specific condition, for which it is shown that a sort of empirical relation is in place. These assumptions should be made very clear in order to avoid overstating and confusion in readers. Second, the paper focuses on claiming to analyze the behavior of the neurons activations at initialization and early stages of learning, with small weights. In this respect, I could find that the experiments use a number of 100 (or 1000) epochs, which seems something different from initial stages of training. An evolution analysis (e.g., showing the behavior over all the epochs) would have been much more informative in this regard. Moreover, while the theoretical analysis assumes small weight values, in some plots (e.g., Fig 4) the results are shown for weight values larger than a threshold (0.4), and this is a bit confusing.\n\nIn order to better understand the novelty of this contribution I also suggest considering clarifying the differences between this current work and the work in Luo et al., 2021.\n\nThe paper presents several typos and unclear sentences in the current form.\n\n[edit after the revision]\nI would like to sincerely thank the authors for the effort in improving the manuscript. I am going to increase slightly my evaluation of the paper. Thanks.\n", "summary_of_the_paper": "The paper studies the relation between condensation of neurons activations and the multiplicity of the used non-linearity. Essentially, it is found an empirical link between the multiplicity of the activation function and the number of condensation directions.", "summary_of_the_review": "The quality of the paper is hampered by\n- not fully convincing and unclear experimental analysis\n- low readability due to several typos in the manuscript\n- unclear improvement wrt the work in Luo et al, 2021", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aq3znCLu-Oo", "reviewer": "Reviewer_jDJ5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well written, organized, and easy to follow. The experimental part is comprehensive and a detailed visualization of condensation orientations is provided.\n- The conclusion that the number of condensation orientations is twice the multiplicity of the activation function gives an possible explanation why small initialization works as implicit regularization at initial training and is insightful for diving into the nonlinear dynamics of NN.\n\nWeaknesses:\n- For experiments on real datasets such as MNIST and CIFAR10, the output dimension $d_\\text{out}$ is still 1. In that case, what is the target value? Is it just the class index, e.g., 0-9? \n- It is hard to analyse the case of the general  multiplicity with high-dimensional input data. However, as Figure 2 displays condensation orientations for activation functions with the multiplicity>1 such as $x\\tanh(x)$ and $x^2\\tanh(x)$, I think the analysis of $p=2,3$ can make the conclusion more stronger.\n- An extension to more practical cases where the output dimension is greater than one is interesting and needs some discussions. Also, when combined with the softmax function for a classification task, does the conclusion still hold?\n", "summary_of_the_paper": "This paper investigates the condensation of weights of neural networks during the initial training stage. It showed theoretically and empirically that the maximal number of condensed orientations in the initial training stage is twice the multiplicity of the activation function under the small initialization of weights. This condensation restricts the capacity of NNs at the beginning, working as implicit regularization.", "summary_of_the_review": "This paper presents a possible perspective to explain why small initialization can serve as implicit regularization during the initial training stage and provides insights to investigate the nonlinear dynamics of neural networks. While it can be improved by conducting more experiments as well as analysis of more complicated cases, it is above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-_dDfTOvrfQ", "reviewer": "Reviewer_zuZq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The biggest thing that I would like is a better characterization of the experiments they run.  Currently they show multiplicity of 1 with little training and multiplicity >1 at much more training. I want the 2x2: what is the correlation matrix for the p>1 at short time scales and for p==1 at longer time scales? E.g., in Fig 1, does relu net at epoch 61 exhibit the bifurcation seen in the xtanh(x)? And is the xtanh(x) condensing on one before?\n  - More broadly, it would be great to see these matrices for a few example networks at a dozen timepoints over the course of training (spanning the periods before they condense to after they condense) -- this could be an informative appendix figure\n  - Also, please state what proportion of units are discarded because the l2 norm was below threshold (both in Fig 1 and in all cases)\n\n- Is there any rigorous or quantitative sense of what the authors mean by \"initial\"? \n\n- A key component of the intuitive argument and analytic results is that the weights have a very low magnitude upon initialization. It is important that the authors did indeed show that at least for CIFAR10, a network with small initialization achieves comparable performance to a more standardly initialized net and does so in a similar amount of training time. But I was left wondering: What are the implications of these results for the more standard case of larger initializations? Presumably this condensation on one or a few axes is not seen in these cases, but I'd be curious about the authors speculations about this. Do the authors have any suggestion as to whether there is some sense in which the net is effectively lower capacity in those cases?\n\n- Why is it natural to vary the nonlinearity across layers of a single network (e.g., second paragraph on page 5)? This feels relatively unmotivated to me, as that's not a standard approach in the field.\n\n- A minor point that is not necessary here, but would helpful in future cases: It is often clearer to use diverging colormaps for positive and negative values rather than a sequential colormap (e.g., https://matplotlib.org/stable/tutorials/colors/colormaps.html). The fact that beige was an anti-correlation, not orthogonality, took me longer to realize than it should have.\n\n- A few small typos:\n- pg 2: \"quantitatively theory explanation\" --> ?? \"quantitative theoretical explanation\"?\n- pg 4: \"full batch expect for...\" --> \"full batch *except* for...\"", "summary_of_the_paper": "This well-written paper takes a step forward in understanding the implicit regularization in neural net optimization. The authors offer empirical evidence that the complexity of the function initially learned by nets is related to the multiplicity of the activation function at zero (i.e., the number of derivatives with nonzero values when evaluated at zero). Then, analytically, they show that all input weights converge towards the same or opposite direction for a multiplicity of 1 (which is the case for common activation functions like tanh, sigmoid, softplus), and that this holds for any multiplicity in the special case of a 1-dimensional hidden layer. Broadly, this work is intriguing, but could stand to benefit from a few improvements, suggested below.", "summary_of_the_review": "This intriguing paper demonstrates, through experiment and analysis, a connection between the implicit regularization of certain networks and the multiplicity of their activation function.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: *Summary:* Study isolated orientations of weights for networks with small initialization depending on multiplicity of activation functions. 

*Strengths:* 
- Interesting analysis of properties in early stages depending on activations. 

*Weaknesses:* 
- Reviewers found the settings limited. 
- Reviewers found experiments limited.  

*Discussion:*

In response to ejGJ authors reiterate scope of covered cases and submit to consideration that their experiments should be adequate for basic research. Reviewer acknowledges the response, but maintains their assessment (limited scope of theory, limited experiments). KucV found the experimental part limited in scope, the settings unclear (notion of early stage, compatibility with theory), and review of previous works lacking. KucV’s sincerely acknowledged authors for their efforts to address their comments and improving the manuscript, and raised their score, but maintained the experimental analysis is not fully convincing and unclear, and the comparison with prior work insufficient. zuZq also expressed concerns with the experiments and the notions and settings under consideration. They also raised questions about the comparison with standard initialization. Authors made efforts to address zuZq concerns. zuZq acknowledged this but maintained initial position that the article is just marginally above threshold. jDJ5 found the paper well written and the conclusion insightful. However, also raised concerns about the experiments the settings under consideration. Authors made efforts to address jDJ5’s concerns, who appreciated this but was not convinced to raise their score. 

*Conclusion:*  
Two reviewers consider this article marginally above and two more marginally below the acceptance threshold. I find the article draws an interesting connection pertaining an interesting topic. However, the reviews and discussion conclude that the article lacks in several regards that in my view still could and should be improved. Therefore I am recommending reject at this time. I encourage the authors to revise and resubmit.
Decision: Reject