Year: 2021

Abstract:
Implicit regularization is important for understanding the learning of neural networks (NNs). Empirical works show that input weights of hidden neurons (the input weight of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) condense on isolated orientations with a small initialization. The condensation dynamics implies that the training implicitly regularizes a NN towards one with much smaller effective size. In this work, we utilize multilayer networks to show that the maximal number of condensed orientations in the initial training stage is twice the multiplicity of the activation function, where “multiplicity” is multiple roots of activation function at origin. Our theoretical analysis confirms experiments for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions, and the other is for the layer with one-dimensional input and arbitrary multiplicity. This work makes a step towards understanding how small initialization implicitly leads NNs to condensation at initial training stage, which lays a foundation for the future study of the nonlinear dynamics of NNs and its implicit regularization effect at a later stage of training.

heading: 1 INTRODUCTION
text: Over-parameterized neural networks often show good generalization performance on real-world problems by minimizing loss functions without explicit regularization (Breiman, 1995; Zhang et al., 2017). For over-parameterized NNs, there are infinite possible sets of training parameters that can reach a satisfying training loss. However, their generalization performances can be very different. It is important to study what implicit regularization is imposed aside to the loss function during the training that leads the NN to a specific type of solutions.
Empirical works suggest that NNs may learn the data from simple to complex patterns (Arpit et al., 2017; Xu et al., 2019; Rahaman et al., 2019; Xu et al., 2020; Jin et al., 2020; Kalimeris et al., 2019). For example, an implicit bias of frequency principle is widely observed that NNs often learn the target function from low to high frequency (Xu et al., 2019; Rahaman et al., 2019; Xu et al., 2020), which has been utilized to understand various phenomena (Ma et al., 2020; Xu & Zhou, 2021) and inspiring algorithm design Liu et al. (2020). The NN output, either simple or complex, is a collective result of all neurons. The study of how neuron weights evolve during the training is central to understanding the collective behavior, including the complexity, of the NN output.
Luo et al. (2021) establish a phase diagram to study the effect of initialization on weight evolution for two-layer ReLU NNs at the infinite-width limit and find three distinct regimes in the phase diagram, i.e., linear regime, critical regime and condensed regime. The non-linear regime, a largely unexplored non-linear regime, is named as condensed regime because the input weights of hidden neurons (the input weight or the feature of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term) condense on isolated orientations during the training (Luo et al., 2021). The three regimes are identified based on the relative change of input weights as the width approaches infinity, which tends to 0, O(1) and +∞, respectively. The condensation is a feature learning process, which is important to the learning of DNNs. Note that in the following, condensation is accompanied by a default assumption of small initialization or large relative change of input weights during training. For practical networks, such as resnet18-like (He et al., 2016) in learning CIFAR10, as shown in Fig. 1(a) and Table 1, we find that
the performance of networks with initialization in the condensed regime is very similar to the common initialization methods. However, the condensation phenomenon provides an intuitive explanation of the good performance as follows, which may lead to a quantitative theoretical explanation in future work. The condensation transforms a large network to a network of only a few effective neurons, leading to an output function with low complexity. Since the complexity bounds the generalization error (Bartlett & Mendelson, 2002), the study of condensation could provide insight to how NNs are implicitly regularized to achieve good generalization performance in practice.
For two-layer ReLU NN, Maennel et al. (2018) prove that, as the initialization of parameters goes to zero, the features of hidden neurons condense at finite number of orientations depending on the input data; when performing a linearly separable classification task with infinite data, Pellegrini & Biroli (2020) show that at mean-field limit, a two-layer infinite-width ReLU NN is effectively equal to a NN of one hidden neuron, i.e., condensation on a single orientation. Both works (Maennel et al., 2018; Pellegrini & Biroli, 2020) study the condensation behavior for ReLU-NNs at an initial training stage in which the magnitudes of NN parameters are far smaller from well-fitting an O(1) target function. However, it still remains unclear that for NNs of more general activation functions, how the condensation emerges at the initial training stage.
In this work, we show that the condensation at the initial stage is closely related to the multiplicity p at x = 0, which means the derivative of activation at x = 0 is zero up to the (p − 1)th-order and is non-zero for the p-th order. To verify their relation, we use the common activation function sigmoid(x), softplus(x), tanh(x), which are multiplicity one, and variants of tanh(x),i.e. x tanh(x) and x2 tanh(x) with multiplicity two and three, for our experiments. For comparison, we also show the initial condensation of ReLU(x), which is studied previously (Maennel et al., 2018) and has totally different properties at origin compared with tanh(x). Our experiments suggest that
the maximal number of condensed orientations is twice the multiplicity of the activation function used in general NNs. For finite-width two-layer NNs with small initialization at the initial training stage, each hidden neuron’s output in a finite domain around 0 can be approximated by a p-th order polynomial and so is the NN output function. Based on the p-th order approximation, we show a preliminary theoretical support for condensation by a theoretical analysis for two cases, one is for the activation function of multiplicity one with arbitrary dimension input, which contains many common activation functions, and the other is for the layer with one-dimensional input and arbitrary multiplicity. Therefore, small initialization imposes an implicit regularization that restricts the NN to be effectively much narrower neural network at the initial training stage.As commonly used activation functions, such as tanh(x), sigmoid(x), softplus(x), etc., are all multiplicity one, our study of initial training behavior lays an important basis for the further study of implicit regularization throughout the training.

heading: 2 RELATED WORKS
text: A research line studies how initialization affects the weight evolution of NNs with a sufficiently large or infinite width. For example, with an initialization in the neural tangent kernel (NTK) regime or lazy training regime (weights change slightly during the training), the gradient flow of infinite-width NN, can be approximated by a linear dynamics of random feature model (Jacot et al., 2018; Arora et al., 2019; Zhang et al., 2020; E et al., 2020; Chizat & Bach, 2019), whereas for the initialization in the mean-field regime (weights change significantly during the training), the gradient flow of infinite-width NN exhibits highly nonlinear dynamics (Mei et al., 2019; Rotskoff & Vanden-Eijnden, 2018; Chizat & Bach, 2018; Sirignano & Spiliopoulos, 2020). Pellegrini & Biroli (2020) analyze how the dynamics of each parameter transforms from a lazy regime (NTK initialization) to a rich regime (mean-field initialization) for an two-layer infinite-width ReLU NN to perform a linearly separable classification task with infinite data. Luo et al. (2021) systematically study the effect of initialization for two-layer ReLU NN with infinite width by establishing a phase diagram, which shows three distinct regimes, i.e., linear regime (similar to the lazy regime), critical regime and condensed regime (similar to the rich regime), based on the relative change of input weights as the width approaches infinity, which tends to 0, O(1) and +∞, respectively. Luo et al. (2021) also empirically find that, in the condensed regime, the features of hidden neurons (orientation of the input weight) condense in several isolated orientations, which is a strong feature learning behavior, an important characteristic of deep learning, however, in Luo et al. (2021), it is not clear how general of the condensation when other activation functions are used and why there is condensation.

heading: 3 PRELIMINARY: NEURAL NETWORKS AND INITIAL STAGE
text: A two-layer NN is
fθ(x) = m∑ j=1 ajσ(wj · x), (1)
where σ(·) is the activation function, wj = (w̄j , bj) ∈ Rd+1 is the neuron feature including the input weight and bias terms, and x = (x̄, 1) ∈ Rd+1 is combination of the input sample and scalar 1, θ is the set of all parameters, i.e., {aj ,wj}mj=1. For simplicity, we call wj as input weight or weight and x as input sample.
A L-layer NN can be recursively defined by feeding the output of the previous layer as the input to the current hidden layer i.e.,
x[0] = (x, 1), x[1] = (σ(W [1]x[0]), 1), x[l] = (σ(W [l]x[l−1]), 1), for l ∈ {2, 3, ..., L}
f(θ,x) = 1
α aᵀx[L] , fθ(x),
(2)
where W [l] = (W̄ [l], b[l]) ∈ Rml×(ml−1+1), and ml represents the dimension of the l-th hidden layer. For simplicity, we also call each row of W [l] as input weight or weight and x[l−1] as input
neurons. The target function is denoted as f∗(x). The training loss function is mean squared error
RS(θ) = 1
2n n∑ i=1 (fθ(xi)− f∗(xi))2. (3)
Without loss of generality, we assume that the output is one-dimensional for theoretical analysis, because for high-dimensional cases, we only need to sum the components directly. For summation, it does not affect the results of our theories. We consider the gradient flow training
θ̇ = −∇θRS(θ). (4)
For convenience, we characterize the activation function by the following definition.
Definition 1 (multiplicity p). Suppose that σ(x) satisfies the following condition, there exists a p ∈ N and p ≥ 1, such that the k-th order derivative σ(k)(0) = 0 for k = 1, 2, · · · , p − 1, and σ(p)(0) 6= 0, then we say σ has multiplicity p.
In the experiments, we study the condensation at the initial stage of training. For a fixed loss, the step we need to achieve it is highly related to the size of learning rate. Therefore, we propose a definition of the initial stage of training by the size of loss in this article, that is the stage before the value of loss function decays to 70% of its initial value. Such a definition is reasonable, for generally a loss could decay to 1% of its initial value or even lower. The loss of the all experiments in the article can be found in Appendix A.3, and they do meet the definition of the initial stage here.

heading: 4 INITIAL CONDENSATION OF INPUT WEIGHTS
text: It is intuitively believed that NNs are powerful at learning data features, which should be an important reason behind the success of deep learning. A simple way to define a learned feature of a neuron is by the orientation of its input weights. Previous work in Luo et al. (2021) show that there is a condensed regime, where the neuron features condense on isolated orientations during the training for two-layer ReLU NNs. The condensation implies that although there are many more neurons than samples, the number of effective neurons, i.e., the number of different used features in fitting, is often much smaller than the number of samples. Therefore, the condensation provides a potential mechanism that helps over-parameterized NNs avoid overfitting. However, it is still unclear how the condensation, for general NNs with small initialization, emerges during the training. In this section, we would empirically show how the condensation differs among NNs with activation functions of different multiplicities, followed by theoretical analysis in the next section.

heading: 4.1 EXPERIMENTAL SETUP
text: For Synthetic dataset and MNIST: Throughout this work, we use fully-connected neural network with size, d-m-· · · -m-dout. The input dimension d is determined by the training data. The output dimension is dout = 1 for synthetic data and dout = 10 for MNIST. The number of hidden neurons m is specified in each experiment. All parameters are initialized by a Gaussian distribution N(0, var). The total data size is n. The training method is Adam with full batch, learning rate lr and MSE loss. For synthetic data, we sample the training data uniformly from a sub-domain of Rd.
For CIFAR10 and CIFAR100 dataset: We use Resnet18-like neural network, which has been describe in Fig. 1 thoroughly. The input dimension d is determined by the training data. The output dimension is dout = 10 for CIFAR10 and dout = 100 for CIFAR100. All parameters are initialized by a Gaussian distribution N(0, var). The total data size is n. The training method is Adam with batch size 128, learning rate lr and Cross-entropy loss.

heading: 4.2 MULTIDIMENSIONAL DATA
text: We first show the condensation at initial training stage in fitting multidimensional dataset. Since the input is a multidimensional vector, the direction is also multidimensional. To characterize the condensation, we use D(u,v) to denote the inner product of the normalized vectors of two input weights, i.e., D(u,v) = uᵀv.
We use a two-layer fully-connected NN with size 5-50-1 to fit n = 80 training data sampled from a 5-dimensional function ∑5 k=1 3.5 sin(5xk + 1), where x = (x1, x2, · · · , x5)ᵀ ∈ R5 and each xk is uniformly sampled from [−4, 2]. As shown in Fig. 2(a), for activation function tanh(x), the color indicates D(u, v) of two hidden neurons’ weights at epoch 100, whose indexes are indicated by the abscissa and the ordinate. If the neurons are in the same beige block, D(u, v) ∼ 1 (navy-blue block, D(u, v) ∼ −1), their input weights have the same (opposite) direction. Obviously, input weights of hidden neurons condense at two opposite directions, i.e., one line. As the multiplicity increasing, NNs with x tanh(x) (Fig. 2(b)) and x2 tanhx (Fig. 2(c)) condense at two and three different lines, respectively. For activation function sigmoid(x) in Fig. 2(d) and softplus(x) in Fig. 2(e), which are frequently used and have multiplicity one, NNs also condense at two opposite directions. For ReLU in Fig. 2(f), for which the multiplicity definition cannot apply, the NN condenses at three directions, in which two are opposite. Through these experiments, we conjecture that the maximal number of condensed orientations is twice the multiplicity of the activation function used at initial training.
For multilayer NNs with different activation functions, we show that the condensation for all hidden layers is similar to the two-layer NNs. In deep networks, residual connection is often introduced to overcome the vanishing of gradient. To show the generality of condensation, we perform an experiment of six-layer NNs with residual connections. To show the difference of various activation functions, we set the activation functions for hidden layer 1 to hidden layer 5 as x2 tanh(x), x tanh(x), sigmoid(x), tanh(x) and softplus(x), respectively. The structure of the residual is hl+1(x) = σ(Wlhl(x) + bl) + hl(x), where hl(x) is the output of the l-th layer. As shown in Fig. 3, input weights condense at three, two, one, one and one lines for hidden layer 1 to hidden layer 5, respectively. Note that residual connections are not necessary. We show an experiment of the same structure as in Fig. 3 but without residual connections in Appendix A.6. To show the universality of condensation, we train resnet18-like neural networks to learn CIFAR10. We study
the condensation of the first fully connected layer of the network, using ReLU and x tanh (x) as the activation functions and initialization distributionN(0, ( 1m1.5 )
2). As shown in Fig. 1 (b) and (c), the condensations for activation ReLU(x) and x tanh (x) are consistent with Fig. 2 and our conjucture. More experiments on dataset CIFAR10 and CIFAR100 can be found in Appendix A.5.
We also find that when the training data is less oscillated, the NN may condense at fewer directions. For example, as shown in Fig. 4(a), compared with the high frequency function in Fig. 2, we only change the target function to be a lower-frequency function, i.e., ∑5 k=1 3.5 sin(2xk + 1). In this case, the NN with x2 tanh(x) only condenses at three directions, in which two are opposite. For MNIST data in Fig. 4(b), we find that, the NN with x2 tanh(x) condenses at one line, which may suggest that the function for fitting MNIST dataset is a low-frequency function. For CIFAR100 data in Fig. 4(c), we find that input weights of the first FC layer with x tanh(x) condense at only one line, which implies that features extracted by the convolution part of the NN may own low complexity.
To understand the mechanism of the initial condensation, we turn to experiments of 1-d input and two-layer NN, which can be clearly visualized in the next subsection.

heading: 4.3 1-D INPUT AND TWO-LAYER NN
text: For 1-d data, we visualize the evolution of the two-layer NN output and each weight, which confirms the connection between the condensation and the multiplicity of the activation function.
We display the outputs at initial training, epoch 1000, in Fig. 5. Due to the small magnitude of parameters, an activation function with multiplicity p can be well approximated by a p-th order polynominal, thus, the NN output can also be approximated by a p-th order polynominal. As shown in Fig. 5(a-c), the NN outputs with activation functions tanh(x), x tanh(x) and x2 tanh(x) overlap well with the auxiliary of a linear, a quadratic and a cubic polynominal curve, respectively. In Fig. 5(d), the NN output with ReLU activation function deviates from a linear function (red auxiliary line). Particularly, the NN output has several sharp turning points. This experiment, although simple, but convincingly shows that NN does not always learn a linear function at the initial training stage and the complexity of such learning depends on the activation function.
We visualize the direction field for input weight wj := (wj , bj), following the gradient flow,
ẇj = − aj n n∑ i=1 eiσ ′(wj · xi)xi,
where ei := fθ(xi)−f∗(xi). Since we only care about the direction ofwj and aj is a scalar at each epoch, we can visualize ẇj by ẇj/aj . For simplicity, we do not distinguish ẇj/aj and ẇj if there is no ambiguity. When we compute ẇj for different j’s, eixi for (i = 1, · · · , n) is independent with j. Then, at each epoch, for a set of {ei,xi}ni=1, we can consider the following direction field
ω̇ = − 1 n n∑ i=1 eixiσ ′(ω · xi).
When ω is set as wj , we can obtain ẇj . As shown in Fig. 6, around the original point, the field has one, two, three stables lines, on which a neuron would keep its direction, for tanh(x), x tanh(x), and x2 tanh(x), respectively. We also display the input weight of each neuron on the field by the green dots and their corresponding velocity directions by the orange arrows. Similarly to the high-dimensional cases, NNs with multiplicity p activation functions condense at p different lines for p = 1, 2, 3. Therefore, It is reasonable to conjecture that the maximal number of condensed orientations is twice the multiplicity of the activation function used. As shown in Fig. 6(d), the field and the condensation for the NN with ReLU(x) is much more complex.
Taken together, we have empirically shown that the multiplicity of the activation function is a key factor that determines the complexity of the initial output and condensation. In figures of this section, we only show the results of the final step of condensation. To facilitate the understanding of evolution of condensation in the initial stage, we show several steps during the initial stage of each example in Appendix A.4. From the evolution, we could directly observe how condensation occurs.

heading: 5 ANALYSIS OF THE INITIAL CONDENSATION OF INPUT WEIGHTS
text: In this section, we would present a preliminary analysis to understand how the multiplicity of the activation function affects the initial condensation. At each training step, we consider the velocity field of weights in each hidden layer of a neural networks.
Considering a network with L hidden layers, we use row vector W [k]j to represent the weight from the (k-1)-th layer to the j-th neuron in the k-th layer. For each k and j,W [k]j satisfies the following
dynamics, (see Appendix A.2)
ṙ = u · ẇ, u̇ = ẇ − (ẇ · u)u r . (5)
where w can beW [k]j ᵀ for all k’s and j’s, r = ‖w‖2 is the amplitude, and u = w/r.
Suppose the activation function has multiplicity p, i.e., σ(k)(0) = 0 for k = 1, 2, · · · , p − 1, and σ(p)(0) 6= 0. For convenience, we define an operator P satisfying Pw := ẇ − u(ẇ · u). Condensation refers to that the weight evolves towards a direction that will not change in the direction field and is defined as follows,
u̇ = 0 ⇔ Pw := ẇ − u(ẇ · u) = 0. Since ẇ ·u is a scalar, ẇ is parallel with u. u is a unit vector, therefore, we have u = ẇ/‖ẇ‖2. In this work, we consider NNs with sufficiently small parameters. For example, suppose r = ‖w‖2 ∼ O( ), is a small quantity. Dynamics (5) shows that O(ṙ) ∼ O(ẇ) and O(u̇) ∼ O(ṙ)/O( ). Therefore, the orientationuwould moves much more quickly than the amplitude r. In the following, we study the case of (i) p = 1 and (ii) ml = 1.
5.1 CASE 1: p = 1
Since we have (see Appendix A.2),
wᵀ = Ẇ [k] j = −
1
n n∑ i=1 (f(θ,xi)− yi) [diag{σ′(W [k]x[k−1]i )}(E [k+1:L]a)]jx [k−1] i ᵀ ,
where we use El = W [l] ᵀ diag{σ′(W [l]x[l−1])}, for l ∈ {2, 3, ..., L}, E[q:p] = EqEq+1...Ep.
For a fixed step, we only consider the gradient w.r.t. Ẇ [k]j . Suppose σ ′(0) 6= 0 and parameters are small. Denote ei := (f(θ,xi)− yi). By Taylor expansion,
Pw leading order ≈ Qw := − 1
n (diag{σ′(0)}(E[k+1:L]a))j n∑ i=1 eix [k−1] i
+
( 1
n (diag{σ′(0)}(E[k+1:L]a))j n∑ i=1 eix [k−1] i · u
) u = 0,
where operator Q is the leading-order approximation of operator P , and here E[k+1:L] is independent with i because diag{σ′(W [l]x[l−1])} ≈ diag{σ′(0)}. Since diag{σ′(0)} = I , and, WLOG, we assume a 6= 0, then
Qw = 0 ⇔ n∑
i=1
eix [k−1] i =
( n∑
i=1
eix [k−1] i · u
) u.
We have
u =
∑n i=1 eix [k−1] i
‖ ∑n
i=1 eix [k−1] i ‖2
or u = − ∑n i=1 eix [k−1] i
‖ ∑n
i=1 eix [k−1] i ‖2
.
This calculation shows that for layer k, the input weights for any hidden neuron j have the same two stable directions. Therefore, when parameters are sufficiently small, which implies that the orientation u would moves much more quickly than the amplitude r, all input weights towards converging to the same direction or the opposite direction, i.e., condensation on a line.
5.2 CASE 2: ml = 1
By the definition of the multiplicity p, we have
σ′(w · xi) = σ(p)(0)
(p− 1)! (w · xi)p−1 + o((w · xi)p−1).
where (·)p−1 and σp(·) operate on component here. Then up to the leading order in terms of the magnitude of θ, we have (see Appendix A.2)
Pw leading order ≈ Qw := −( 1
n n∑ i=1 eix [k−1] i (w ᵀx [k−1] i ) p−1)[diag{ σ (p)(0) (p− 1)! }(E[k+1:L]a)]j
+(( 1
n n∑ i=1 eix [k−1] i (w ᵀx [k−1] i ) p−1)[diag{ σ (p)(0) (p− 1)! }(E[k+1:L]a)]j · u)u.
WLOG, we also assume a 6= 0. And by definition, w = ru, we have
Qw = 0 ⇔ u = 1 n
∑n i=1 eix [k−1] i (u ᵀx [k−1] i ) p−1
‖ 1n ∑n i=1 eix [k−1] i (u ᵀx [k−1] i ) p−1‖2
or u = − 1 n
∑n i=1 eix [k−1] i (u ᵀx [k−1] i ) p−1
‖ 1n ∑n i=1 eix [k−1] i (u ᵀx [k−1] i ) p−1‖2 .
Since d+ 1 = 2, we denote u = (u1, u2)ᵀ ∈ R2 and x[k−1]i = ((x [k−1] i )1, (x [k−1] i )2) ᵀ ∈ R2, then,∑n i=1(u1(x [k−1] i )1 + u2(x [k−1] i )2)
p−1ei(x [k−1] i )1∑n
i=1(u1(x [k−1] i )1 + u2(x [k−1] i )2) p−1ei(x [k−1] i )2
= u1 u2 , û.
We obtain the equation for û, n∑
i=1
(û(x [k−1] i )1 + (x [k−1] i )2) p−1ei(x [k−1] i )1 = û n∑ i=1 (û(x [k−1] i )1 + (x [k−1] i )2) p−1ei(x [k−1] i )2.
Since it is an univariate p-th order equation, û = u1u2 has at most p complex roots. Because u is a unit vector, u at most has p pairs of values, in which each pair are opposite.
Taken together, our theoretical analysis is consistent with our experiments, that is, the maximal number of condensed orientations is twice the multiplicity of the activation function used when parameters are small. Besides, because commonly used activation functions, such as tanh(x), sigmoid(x), softplus(x), etc., are all multiplicity one, the theoretical analysis sheds light on practical training.

heading: 6 DISCUSSION
text: In this work, we have shown that the characteristic of the activation function, i.e., multiplicity, is a key factor to understanding the complexity of NN output and the weight condensation at initial training. The condensation restricts the NN to be effectively low-capacity at the initial training stage, even for finite-width NNs. During the training, the NN increases its capacity to better fit the data, leading to a potential explanation for their good generalization in practical problems. This work also serves as a starting point for further studying the condensation for multiple-layer neural networks throughout the training process.
As the scale of parameter initialization becomes larger, the condensation becomes weaker. The understanding from a complete condensation would benefit our understanding of the training process in common initialization, that is there is an effect, although not so strong, that can limit the complexity of the neural network in the initial training.
How small the initialization should be in order to see a clear condensation is studied in Luo et al. (2021) for two-layer ReLU NNs with infinite width. For general activation functions, the regime of the initialization for condensation depends on the NN width. A further study of the phase diagram for finite width NNs would be important.
For general multiplicity with high-dimensional input data, the theoretical analysis for the initial condensation is a very difficult problem, which is equivalent to count the number of the roots of a high-order high-dimensional polynomial with a special structure originated from NNs.

heading: A APPENDIX

heading: A.1 BASIC DEFINITIONS
text: In this study, we first consider the neural network with 2 hidden layers,
A two-layer NN is
fθ(x) = m∑ j=1 ajσ(wj · x), (6)
where σ(·) is the activation function, wj = (w̄j , bj) ∈ Rd+1 is the neuron feature including the input weight and bias terms, and x = (x̄, 1) ∈ Rd+1 is combination of the input sample and scalar 1, θ is the set of all parameters, i.e., {aj ,wj}mj=1. For simplicity, we call wj as input weight or weight and x as input sample.
Then, we consider the neural network with l hidden layers,
x[0] = (x, 1), x[1] = (σ(W [1]x[0]), 1), x[l] = (σ(W [l]x[l−1]), 1), for l ∈ {2, 3, ..., L}
f(θ,x) = 1
α aᵀx[L] , fθ(x),
(7)
whereW [l] = (W̄ [l], b[l]) ∈ R(ml×ml−1), and ml represents the dimension of the l-th hidden layer. The initialization of W [l]k,k′ , l ∈ {1, 2, 3, ..., L} and ak obey normal distribution W [l] k,k′ ∼ N (0, β2l ) for l ∈ {1, 2, 3, ..., L} and ak ∼ N (0, β2L+1). The loss function is mean squared error given below,
Rs(θ) = 1
2n n∑ i=1 (fθ(xi)− yi)2. (8)
For simplification, we denote fθ(x) as f in following.

heading: A.2 DERIVATIONS FOR CONCERNED QUANTITIES

heading: A.2.1 NEURAL NETWORKS WITH THREE HIDDEN LAYERS
text: In order to better understand the gradient of the parameter matrix of the multi-layer neural network, we first consider the case of the three-layer neural network,
fθ(x) := a ᵀσ(W [2]σ(W [1]x)), (9)
with the mean squared error as the loss function,
Rs(θ) = 1
2n n∑ i=1 (fθ(xi)− yi)2. (10)
We calculate df dW [2] and df dW [1] respectively, using differential form,
df = tr(( ∂f
∂x )ᵀdf). (11)
We consider df dW [2] first,
df = tr{d(aᵀσ(W [2]x[1]))} = tr{aᵀd(σ(W [2]x[1]))}
= tr{aᵀσ ′ (W [2]x[1]) dW [2]x[1]}
= tr{(a σ ′ (W [2]x[1])ᵀdW [2]x[1]}
= tr{x[1](a σ ′ (W [2]x[1])ᵀdW [2]}
= tr{((a σ ′ (W [2]x[1]))x[1] ᵀ )ᵀdW [2]},
(12)
where is Hadamard Product, and it is the multiplication of matrix elements of the same position. Hence,
df
dW [2] = (a σ
′ (W [2]x[1]))x[1] ᵀ
= diag{σ ′ (W [2]x[1])}ax[1] ᵀ .
(13)
Then, we consider df dW [1] ,
df = tr{(a σ ′ (W [2]x[1]))ᵀW [2]dσ(W [1]x)}
= tr{(W [2] ᵀ (a σ ′ (W [2]x[1])))ᵀσ ′ (W [1]x) d(W [1]x)}
= tr{((W [2] ᵀ (a σ ′ (W [2]x[1])) σ ′ (W [1]x))ᵀd(W [1]x)}
= tr{[((W [2] ᵀ (a σ ′ (W [2]x[1])) σ ′ (W [1]x))xᵀ]ᵀd(W [1])}.
(14)
Hence, we have,
df
dW [1] = ((W [2]
ᵀ (a σ ′ (W [2]x[1]))) σ ′ (W [1]x))xᵀ
= diag{σ ′ (W [1]x)}W [2] ᵀ diag{σ ′ (W [2]x[1])}axᵀ.
(15)
Through the chain rule, we can get the evolution equation ofW [1] andW [2],
dW [1]
dt = −dRs(θ) dW [1]
= − 1 n n∑ i=1 (f(θ,xi)− yi) df dW [1]
= − 1 n n∑ i=1 (f(θ,xi)− yi)diag{σ ′ (W [1]xi)}W [2] ᵀ diag{σ ′ (W [2]x [1] i )}ax ᵀ i ,
(16)
and dW [2]
dt = −dRs(θ) dW [2]
= − 1 n n∑ i=1 (f(θ,xi)− yi) df dW [1]
= − 1 n n∑ i=1 (f(θ,xi)− yi)diag{σ ′ (W [2]x [1] i )}ax [1] i ᵀ .
(17)

heading: A.2.2 L HIDDEN LAYERS CONDITION
text: And, we consider the neural network with L hidden layers,
df = tr{daᵀdσ(W [L]x[L−1])}
= tr{(a σ ′ (W [L]x[L−1]))ᵀdW [L]σ(W [L−1]x[L−2])}
= tr{(W [L] ᵀ ΛL) ᵀσ ′ (W [L−1]x[L−2]) dW [L−1]σ(W [L−2]x[L−3])} = tr{((W [L] ᵀ ΛL) σ ′ (W [L−1]x[L−2]))ᵀW [L−1]dσ(W [L−2]x[L−3])} = (W [L−1] ᵀ ΛL−1) ᵀdσ(W [L−2]x[L−3])
= . . .
= tr{ΛᵀkdW [k]x[k−1]} = tr{(Λkx[k−1] ᵀ )ᵀdW [k]},
(18)
where Λl , (W [l+1] ᵀ Λl+1) σ ′ (W [l]x[l−1]) for l = k, k + 1 . . . L − 1 and ΛL , a σ ′ (W [L]x[L−1]).
Hence, we get,
df
dW [k] = Λkx
[k−1]ᵀ. (19)
Through the chain rule, we can get the evolution equation ofW [k],
dW [k]
dt = −dRs(θ) dW [k]
= − 1 n n∑ i=1 (f(θ,xi)− yi) df dW [k]
= − 1 n n∑ i=1 (f(θ,xi)− yi)Λkx[k−1]i ᵀ .
(20)
Through a σ′(Wx) = diag{σ′(Wx)}a, Finally, the dynamic system can be obtained:
ȧ = da dt = − 1 n n∑ i=1 x [L] i (f(θ,xi)− yi) ,
Ẇ [L] = dW [L] dt = − 1 n n∑ i=1 diag{σ′(W [L]x[L−1]i )}ax [L−1] i ᵀ (f(θ,xi)− yi) ,
Ẇ [k] = dW [k] dt = − 1 n n∑ i=1 diag{σ′(W [k]x[k−1]i )}E [k+1:L]ax [k−1] i ᵀ (f(θ,xi)− yi) ∀i ∈ [1 : L− 1],
(21)
where we use El(x) = W [l] ᵀ diag{σ′(W [l]x[l−1])}. And E[q:p] = EqEq+1...Ep.
Let rk,j = ‖W [k]j ‖2. We have
d dt |rk,j |2 = d dt ‖W [k]j ‖ 2. (22)
Then we obtain
ṙk,jrk,j = Ẇ [k] j ·W [k] j . (23)
Finally, we get
ṙk,j = drk,j
dt = Ẇ
[k] j ·W [k] j /rk,j
= Ẇ [k] j · uk,j ,
(24)
where uk,j = W
[k] j
rk,j is a unit vector. Then we have,
u̇k,j = duk,j
dt =
d
dt
( W
[k] j
rk,j
)
= Ẇ
[k] j rk,j −W [k] j ṙk,j
r2k,j
= Ẇ
[k] j rk,j −W [k] j (Ẇ [k] j · uk,j)
r2k,j
= Ẇ
[k] j − uk,j(Ẇ [k] j · uk,j)
rk,j .
(25)
To conclude, the quantities we concern are summarized as follows,
 ȧ = − 1 n n∑ i=1 x [L] i (f(θ,xi)− yi) (26) Ẇ [L] = − 1 n n∑ i=1 diag{σ′(W [L]x[L−1]i )}ax [L−1] i ᵀ (f(θ,xi)− yi) , (27) Ẇ [k] = − 1 n n∑ i=1 diag{σ′(W [k]x[k−1]i )}E [k+1:L]ax [k−1] i ᵀ (f(θ,xi)− yi) ∀i ∈ [1 : L− 1] (28) ṙk,j = Ẇ [k] j · uk,j (29) u̇k,j = Ẇ [k] j − uk,j(Ẇ [k] j · uk,j)
rk,j , (30)
where we use El(x) = W [l] ᵀ diag{σ′(W [l]x[l−1])}. And E[q:p] = EqEq+1...Ep.
A.2.3 PROVE FOR Pw IN 5.2
We calculate Pw leading order ≈ Qw as following,
Pw≈Qw : = − 1 n n∑ i=1 eix [k−1] i [diag{σ ′(W [k]x [k−1] i )}(E [k+1:L]a)]j
+ ( 1
n n∑ i=1 eix [k−1] i [diag{σ ′(W [k]x [k−1] i )}(E [k+1:L]a)]j · u)u
= − 1 n n∑ i=1 eix [k−1] i [diag{ σ(p)(0) (p− 1)! (W [k]x[k−1]i ) p−1}(E[k+1:L]a)]j
+ ( 1
n n∑ i=1 eix [k−1] i [diag{ σ(p)(0) (p− 1)! (W [k]x[k−1]i ) p−1}(E[k+1:L]a)]j · u)u
= − 1 n n∑ i=1 eix [k−1] i [diag{(W [k]x [k−1] i ) p−1} diag{ σ (p)(0) (p− 1)! }(E[k+1:L]a)]j
+ ( 1
n n∑ i=1 eix [k−1] i [diag{(W [k]x [k−1] i ) p−1} diag{ σ (p)(0) (p− 1)! }(E[k+1:L]a)]j · u)u
= − 1 n n∑ i=1 eix [k−1] i [diag{(W [k]x [k−1] i ) p−1}]j diag{ σ(p)(0) (p− 1)! }(E[k+1:L]a)
+ ( 1
n n∑ i=1 eix [k−1] i [diag{(W [k]x [k−1] i ) p−1}]j diag{ σ(p)(0) (p− 1)! }(E[k+1:L]a) · u)u
= −( 1 n n∑ i=1 eix [k−1] i (W [k] j x [k−1] i ) p−1)[diag{ σ (p)(0) (p− 1)! }(E[k+1:L]a)]j
+ (( 1
n n∑ i=1 eix [k−1] i (W [k] j x [k−1] i ) p−1)[diag{ σ (p)(0) (p− 1)! }(E[k+1:L]a)]j · u)u,
(31) where (·)p−1 and σp(·) operate on component here.

heading: A.3 THE VERIFICATION OF THE INITIAL STAGE
text: We put the loss of the experiments in the main text here to show that they are indeed in the initial stage of training by the definition.
As is shown in Fig.7 and Fig.8, at the number of steps I drew the graphs in the article, loss satisfies the definition of the initial stage, so we consider that they are in the initial stage of training.
Only the epoch number could hardly reflect the initial stage, which also depends on the learning rate. Therefore, we use the loss to indicate the initial stage. Learning rate is not a sensitive to the appearance of condensation. However, a small learning rate enables us to clearly observe the condensation process in the initial stage under a gradient flow training. For example, when the learning rate is relatively small, the initial stage of training may be relatively long, while when the learning rate is relatively large, the initial stage of training may be relatively small.
We empirically find that to ensure the training process follows a gradient follow, where the loss decays monotonically, we have to select a smaller learning rate for large multiplicity p. Therefore, it looks like we have a longer training in our experiments with large p. Note that for a small learning rate in the experiments of small p, we can observe similar phenomena.
In all subsequent experiments in the Appendix, we will no longer show the loss graph of each experiment one by one, but we make sure that they are indeed in the initial stage of training.

heading: A.4 SEVERAL STEPS DURING THE EVOLUTION OF CONDENSATION AT THE INITIAL STAGE
text: In the article, we only give the results of the last step of each condense, while the details of the evolution of condensation are lacking, which may provide us a better understanding. Therefore, we show these details in Fig. 9, Fig. 10, Fig. 11 and Fig. 12, which also further illustrate the rationality of the experimental results and facilitate the understanding of the evolution of condensation in the initial stage.

heading: A.5 THE EXPERIMENTS ON CIFAR 10 AND CIFAR 100 WITH RESNET18-LIKE NEURAL NETWORK
text: The condensation of the Resnet18-like neural network on CIFAR10 and CIFAR100 is shown in Fig. 13 and Fig. 14, whose activation functions for fully-connected (FC) layers are tanh(x), ReLU(x), sigmoid(x), softplus(x) and x tanh(x), indicated by the corresponding sub-captions, respectively.

heading: A.6 MULTILAYER EXPERIMENTAL
text: The condensation of the six layer without residual connections is shown in 15, whose activation functions for hidden layer 1 to hidden layer 5 are x2 tanh(x), x tanh(x), sigmoid(x), tanh(x) and softplus(x), respectively.
The condensation of the three layer without residual connections is shown in 16, whose activation functions are same for each layer indicated by the corresponding sub-captions.
The condensation of the five layer without residual connections is shown in 17, whose activation functions are same for each layer indicated by the corresponding sub-captions.
The condensation of the five layer with residual connections is shown in 18, whose activation functions are same for each layer indicated by the corresponding sub-captions.

Review summaries:
This paper studies the role of activation functions (via their multiplicity at the origin) in the condensation of neural networks at the initial training stage. Condensation can be viewed as a feature learning process, where the wide network can be described effectively as a narrower network and the input weights condense on isolated orientations during training. This mechanism may provide a plausible explanation for the performance of the wide network. In particular, the paper shows empirically that the maximal number of condensed orientations is twice the multiplicity of the activation function used. Moreover, using polynomial approximations, the paper provides theoretical support in two cases: when the activation function is of multiplicity one and when the input is one-dimensional. 

This well-written paper takes a step forward in understanding the implicit regularization in neural net optimization. The authors offer empirical evidence that the complexity of the function initially learned by nets is related to the multiplicity of the activation function at zero (i.e., the number of derivatives with nonzero values when evaluated at zero). Then, analytically, they show that all input weights converge towards the same or opposite direction for a multiplicity of 1 (which is the case for common activation functions like tanh, sigmoid, softplus), and that this holds for any multiplicity in the special case of a 1-dimensional hidden layer. Broadly, this work is intriguing, but could stand to benefit from a few improvements, suggested below.

This paper investigates the condensation of weights of neural networks during the initial training stage. It showed theoretically and empirically that the maximal number of condensed orientations in the initial training stage is twice the multiplicity of the activation function under the small initialization of weights. This condensation restricts the capacity of NNs at the beginning, working as implicit regularization.