Towards Understanding the Condensation of Neural Networks at Initial Training

1 INTRODUCTION . Over-parameterized neural networks often show good generalization performance on real-world problems by minimizing loss functions without explicit regularization ( Breiman , 1995 ; Zhang et al. , 2017 ) . For over-parameterized NNs , there are infinite possible sets of training parameters that can reach a satisfying training loss . However , their generalization performances can be very different . It is important to study what implicit regularization is imposed aside to the loss function during the training that leads the NN to a specific type of solutions . Empirical works suggest that NNs may learn the data from simple to complex patterns ( Arpit et al. , 2017 ; Xu et al. , 2019 ; Rahaman et al. , 2019 ; Xu et al. , 2020 ; Jin et al. , 2020 ; Kalimeris et al. , 2019 ) . For example , an implicit bias of frequency principle is widely observed that NNs often learn the target function from low to high frequency ( Xu et al. , 2019 ; Rahaman et al. , 2019 ; Xu et al. , 2020 ) , which has been utilized to understand various phenomena ( Ma et al. , 2020 ; Xu & Zhou , 2021 ) and inspiring algorithm design Liu et al . ( 2020 ) . The NN output , either simple or complex , is a collective result of all neurons . The study of how neuron weights evolve during the training is central to understanding the collective behavior , including the complexity , of the NN output . Luo et al . ( 2021 ) establish a phase diagram to study the effect of initialization on weight evolution for two-layer ReLU NNs at the infinite-width limit and find three distinct regimes in the phase diagram , i.e. , linear regime , critical regime and condensed regime . The non-linear regime , a largely unexplored non-linear regime , is named as condensed regime because the input weights of hidden neurons ( the input weight or the feature of a hidden neuron consists of the weight from its input layer to the hidden neuron and its bias term ) condense on isolated orientations during the training ( Luo et al. , 2021 ) . The three regimes are identified based on the relative change of input weights as the width approaches infinity , which tends to 0 , O ( 1 ) and +∞ , respectively . The condensation is a feature learning process , which is important to the learning of DNNs . Note that in the following , condensation is accompanied by a default assumption of small initialization or large relative change of input weights during training . For practical networks , such as resnet18-like ( He et al. , 2016 ) in learning CIFAR10 , as shown in Fig . 1 ( a ) and Table 1 , we find that the performance of networks with initialization in the condensed regime is very similar to the common initialization methods . However , the condensation phenomenon provides an intuitive explanation of the good performance as follows , which may lead to a quantitative theoretical explanation in future work . The condensation transforms a large network to a network of only a few effective neurons , leading to an output function with low complexity . Since the complexity bounds the generalization error ( Bartlett & Mendelson , 2002 ) , the study of condensation could provide insight to how NNs are implicitly regularized to achieve good generalization performance in practice . For two-layer ReLU NN , Maennel et al . ( 2018 ) prove that , as the initialization of parameters goes to zero , the features of hidden neurons condense at finite number of orientations depending on the input data ; when performing a linearly separable classification task with infinite data , Pellegrini & Biroli ( 2020 ) show that at mean-field limit , a two-layer infinite-width ReLU NN is effectively equal to a NN of one hidden neuron , i.e. , condensation on a single orientation . Both works ( Maennel et al. , 2018 ; Pellegrini & Biroli , 2020 ) study the condensation behavior for ReLU-NNs at an initial training stage in which the magnitudes of NN parameters are far smaller from well-fitting an O ( 1 ) target function . However , it still remains unclear that for NNs of more general activation functions , how the condensation emerges at the initial training stage . In this work , we show that the condensation at the initial stage is closely related to the multiplicity p at x = 0 , which means the derivative of activation at x = 0 is zero up to the ( p − 1 ) th-order and is non-zero for the p-th order . To verify their relation , we use the common activation function sigmoid ( x ) , softplus ( x ) , tanh ( x ) , which are multiplicity one , and variants of tanh ( x ) , i.e . x tanh ( x ) and x2 tanh ( x ) with multiplicity two and three , for our experiments . For comparison , we also show the initial condensation of ReLU ( x ) , which is studied previously ( Maennel et al. , 2018 ) and has totally different properties at origin compared with tanh ( x ) . Our experiments suggest that the maximal number of condensed orientations is twice the multiplicity of the activation function used in general NNs . For finite-width two-layer NNs with small initialization at the initial training stage , each hidden neuron ’ s output in a finite domain around 0 can be approximated by a p-th order polynomial and so is the NN output function . Based on the p-th order approximation , we show a preliminary theoretical support for condensation by a theoretical analysis for two cases , one is for the activation function of multiplicity one with arbitrary dimension input , which contains many common activation functions , and the other is for the layer with one-dimensional input and arbitrary multiplicity . Therefore , small initialization imposes an implicit regularization that restricts the NN to be effectively much narrower neural network at the initial training stage.As commonly used activation functions , such as tanh ( x ) , sigmoid ( x ) , softplus ( x ) , etc. , are all multiplicity one , our study of initial training behavior lays an important basis for the further study of implicit regularization throughout the training . 2 RELATED WORKS . A research line studies how initialization affects the weight evolution of NNs with a sufficiently large or infinite width . For example , with an initialization in the neural tangent kernel ( NTK ) regime or lazy training regime ( weights change slightly during the training ) , the gradient flow of infinite-width NN , can be approximated by a linear dynamics of random feature model ( Jacot et al. , 2018 ; Arora et al. , 2019 ; Zhang et al. , 2020 ; E et al. , 2020 ; Chizat & Bach , 2019 ) , whereas for the initialization in the mean-field regime ( weights change significantly during the training ) , the gradient flow of infinite-width NN exhibits highly nonlinear dynamics ( Mei et al. , 2019 ; Rotskoff & Vanden-Eijnden , 2018 ; Chizat & Bach , 2018 ; Sirignano & Spiliopoulos , 2020 ) . Pellegrini & Biroli ( 2020 ) analyze how the dynamics of each parameter transforms from a lazy regime ( NTK initialization ) to a rich regime ( mean-field initialization ) for an two-layer infinite-width ReLU NN to perform a linearly separable classification task with infinite data . Luo et al . ( 2021 ) systematically study the effect of initialization for two-layer ReLU NN with infinite width by establishing a phase diagram , which shows three distinct regimes , i.e. , linear regime ( similar to the lazy regime ) , critical regime and condensed regime ( similar to the rich regime ) , based on the relative change of input weights as the width approaches infinity , which tends to 0 , O ( 1 ) and +∞ , respectively . Luo et al . ( 2021 ) also empirically find that , in the condensed regime , the features of hidden neurons ( orientation of the input weight ) condense in several isolated orientations , which is a strong feature learning behavior , an important characteristic of deep learning , however , in Luo et al . ( 2021 ) , it is not clear how general of the condensation when other activation functions are used and why there is condensation . 3 PRELIMINARY : NEURAL NETWORKS AND INITIAL STAGE . A two-layer NN is fθ ( x ) = m∑ j=1 ajσ ( wj · x ) , ( 1 ) where σ ( · ) is the activation function , wj = ( w̄j , bj ) ∈ Rd+1 is the neuron feature including the input weight and bias terms , and x = ( x̄ , 1 ) ∈ Rd+1 is combination of the input sample and scalar 1 , θ is the set of all parameters , i.e. , { aj , wj } mj=1 . For simplicity , we call wj as input weight or weight and x as input sample . A L-layer NN can be recursively defined by feeding the output of the previous layer as the input to the current hidden layer i.e. , x [ 0 ] = ( x , 1 ) , x [ 1 ] = ( σ ( W [ 1 ] x [ 0 ] ) , 1 ) , x [ l ] = ( σ ( W [ l ] x [ l−1 ] ) , 1 ) , for l ∈ { 2 , 3 , ... , L } f ( θ , x ) = 1 α aᵀx [ L ] , fθ ( x ) , ( 2 ) where W [ l ] = ( W̄ [ l ] , b [ l ] ) ∈ Rml× ( ml−1+1 ) , and ml represents the dimension of the l-th hidden layer . For simplicity , we also call each row of W [ l ] as input weight or weight and x [ l−1 ] as input neurons . The target function is denoted as f∗ ( x ) . The training loss function is mean squared error RS ( θ ) = 1 2n n∑ i=1 ( fθ ( xi ) − f∗ ( xi ) ) 2 . ( 3 ) Without loss of generality , we assume that the output is one-dimensional for theoretical analysis , because for high-dimensional cases , we only need to sum the components directly . For summation , it does not affect the results of our theories . We consider the gradient flow training θ̇ = −∇θRS ( θ ) . ( 4 ) For convenience , we characterize the activation function by the following definition . Definition 1 ( multiplicity p ) . Suppose that σ ( x ) satisfies the following condition , there exists a p ∈ N and p ≥ 1 , such that the k-th order derivative σ ( k ) ( 0 ) = 0 for k = 1 , 2 , · · · , p − 1 , and σ ( p ) ( 0 ) 6= 0 , then we say σ has multiplicity p. In the experiments , we study the condensation at the initial stage of training . For a fixed loss , the step we need to achieve it is highly related to the size of learning rate . Therefore , we propose a definition of the initial stage of training by the size of loss in this article , that is the stage before the value of loss function decays to 70 % of its initial value . Such a definition is reasonable , for generally a loss could decay to 1 % of its initial value or even lower . The loss of the all experiments in the article can be found in Appendix A.3 , and they do meet the definition of the initial stage here .

Summary: This well-written paper takes a step forward in understanding the implicit regularization in neural net optimization. The authors offer empirical evidence that the complexity of the function initially learned by nets is related to the multiplicity of the activation function at zero (i.e., the number of derivatives with nonzero values when evaluated at zero). Then, analytically, they show that all input weights converge towards the same or opposite direction for a multiplicity of 1 (which is the case for common activation functions like tanh, sigmoid, softplus), and that this holds for any multiplicity in the special case of a 1-dimensional hidden layer. Broadly, this work is intriguing, but could stand to benefit from a few improvements, suggested below.