Mildly Overparametrized Neural Nets can Memorize Training Data Efficiently

1 INTRODUCTION . In deep learning , highly non-convex objectives are optimized by simple algorithms such as stochastic gradient descent . There has been many theoretical analysis for the optimization landscape of neural networks ( e.g. , Brutzkus et al . ( 2017 ) ; Brutzkus & Globerson ( 2017 ) ; Ge et al . ( 2017b ) ; Wang et al . ( 2018 ) ) , but even very simple two-layer networks have spurious local optima ( Safran & Shamir , 2018 ) . In practice , it was observed that neural networks are able to fit the training data perfectly , even when the data/labels are randomly corrupted ( Zhang et al. , 2017 ) . Recently , a series of work ( Du et al . ( 2019 ) ; Allen-Zhu et al . ( 2019c ) ; Chizat & Bach ( 2018 ) ; Jacot et al . ( 2018 ) , see more references in Section 1.2 ) developed a theory of neural tangent kernels ( NTK ) that explains the success of training neural networks through overparametrization . Several results showed that if the number of neurons at each layer is much larger than the number of training samples , networks of different architectures ( multilayer/recurrent ) can all fit the training data perfectly . However , if one considers the number of parameters required for the current theoretical analysis , these networks are highly overparametrized . Consider fully connected networks for example . If a two-layer network has a hidden layer with r neurons , the number of parameters is at least rd where d is the dimension of the input . For deeper networks , if it has two consecutive hidden layers of size r , then the number of parameters is at least r2 . All of the existing works require the number of neurons r per-layer to be at least the number of training samples n ( in fact , most of them require r to be a polynomial of n ) . In these cases , the number of parameters can be at least nd or even n2 for deeper networks –much larger than the number of training samples n. Therefore , a natural question is whether neural networks can fit the training data in the mildly overparametrized regime - where the number of parameters is only a constant factor larger than the number of training data . To achieve this , one would want to use a small number of neurons in each layer - n/d for a two-layer network and √ n for a three-layer network . Yun et al . ( 2018 ) showed such networks have enough capacity to memorize any training data . In this paper we show with polynomial activation functions , simple optimization algorithms are guaranteed to find a solution that memorizes training data . 1.1 OUR RESULTS . In this paper , we give network architectures ( with polynomial activations ) such that every layer has width much smaller than the number of training samples n , the total number of parameters is linear in n , and simple optimization algorithms on such neural networks can fit any training data . We first give a warm-up result that works when the number of training samples is roughly d2 ( where d is the input dimension ) . Theorem 1 ( Informal ) . Suppose there are n ≤ ( d+1 2 ) training samples in general position , there exists a two-layer neural network with quadratic activations , such that the number of neurons in the hidden layer is 2d+2 , the total number of parameters isO ( d2 ) , and perturbed gradient descent can fit the network to any output . Here “ in general position ” will be formalized later as a deterministic condition that is true with probability 1 for random inputs , see Theorem 4 for details . In this case , the number of hidden neurons is only roughly the square root of the number of training samples , so the weights for these neurons need to be trained carefully in order to fit the data . Our analysis relies on an analysis of optimization landscape - we show that every local minimum for such neural network must also be globally optimal ( and has 0 training error ) . As a result , the algorithm can converge from an arbitrary initialization . Of course , the result above is limited as the number of training samples can not be larger thanO ( d2 ) . We can extend the result to handle a larger number of training samples : Theorem 2 ( Informal ) . Suppose number of training samples n ≤ dp for some constant p , if the training samples are in general position there exists a three-layer neural network with polynomial activations , such that the number of neurons r in each layer is Op ( √ n ) , and perturbed gradient descent on the weights of the middle layer can fit the network to any output . Here Op considers p as a constants and hides constant factors that only depend on p. We consider “ in general position ” in the smoothed analysis framework ( Spielman & Teng , 2004 ) - given arbitrary inputs x1 , x2 , ... , xn ∈ Rd , fix a perturbation radius √ v , the actual inputs is x̄j = xj+x̃j where x̃j ∼ N ( 0 , vI ) . The guarantee of training algorithm will depend inverse polynomially on the perturbation v ( note that the architecture - in particular the number of neurons - is independent of v ) . The formal result is given in Theorem 5 in Section 4 . Later we also give a deterministic condition for the inputs , and prove a slightly weaker result ( see Theorem 6 ) . 1.2 RELATED WORKS . Optimization Landscape for Networks without Overparametrization Many works ( Brutzkus & Globerson , 2017 ; Tian , 2017 ; Li & Yuan , 2017 ; Soltanolkotabi , 2017 ; Zhong et al. , 2017 ; Ge et al. , 2017b ) analyzed the optimization landscape of 2-layer neural networks . However , these works either work on a single neuron or have very strong assumptions on the input x ( such as x is Gaussian ) . It is also known that even with strong assumptions on input x gradient descent on the standard objective can get stuck in spurious local minima when the network has more than a constant number of neurons ( Safran & Shamir , 2018 ) . Neural Tangent Kernel Many results in the framework of neural tangent kernel show that networks with different architecture can all memorize the training data , including two-layer ( Du et al. , 2019 ) , multi-layer ( Du et al. , 2018 ; Allen-Zhu et al. , 2019c ; Zou & Gu , 2019 ) , recurrent neural network ( Allen-Zhu et al. , 2019b ) . However , all of these works require the number of neurons in each layer to be at least quadratic in the number of training samples . Oymak & Soltanolkotabi ( 2019 ) improved the number of neurons required for two-layer networks , but their bound is still larger than the number of training samples . There are also more works for NTK on generalization guarantees ( e.g. , Allen-Zhu et al . ( 2019a ) ) , fine-grained analysis for specific inputs ( Arora et al. , 2019b ) and empirical performances ( Arora et al. , 2019c ) , but they are not directly related to our results . Representation Power of Neural Networks For standard neural networks with ReLU activations , Yun et al . ( 2018 ) showed that networks of similar size as Theorem 2 can memorize any training data . Their construction is delicate and it is not clear whether gradient descent can find such a solution . Matrix Factorizations and Quadratic Activations Since the activation function for our twolayer net is quadratic , training of the network is very similar to matrix factorization problem . Many existing works analyzed the optimization landscape and implicit bias for problems related to matrix factorization in various settings ( Bhojanapalli et al. , 2016 ; Ge et al. , 2016 ; 2017a ; Park et al. , 2016 ; Gunasekar et al. , 2017 ; Li et al. , 2018 ; Arora et al. , 2019a ) . In this line of work , Du & Lee ( 2018 ) and Soltanolkotabi et al . ( 2018 ) are the most similar to our two-layer result . Du & Lee ( 2018 ) showed how gradient descent can learn a two-layer neural network that represents any positive semidefinite matrix . However positive definite matrices can not be used to memorize arbitrary data , and our two-layer network can represent an arbitrary matrix . Soltanolkotabi et al . ( 2018 ) is very similar to our two-layer result except they require the input to be Gaussian . Extending this condition to our deterministic condition is crucial to our main ( 3-layer ) result . Existing works on mildly overparametrization There are several works on overparametrization that does not fall exactly into the NTK regime . Brutzkus et al . ( 2017 ) ; Wang et al . ( 2018 ) works for linear separable setting . However in this setting gradient descent also works even if the network has only a single neuron ( no overparametrization ) , so the results only show that overparametrization does not hurt . Li & Liang ( 2018 ) requires an interesting but strong assumption on the input data . Our work is different as we make mild assumptions on the data distribution and the network size is necessary ( up to constant factors ) to achieve 0 training error without further assumptions . Interpolating Methods Of course , simply memorizing the data may not be useful in machine learning . However , recently several works ( Belkin et al. , 2018 ; 2019 ; Liang et al. , 2019 ; Mei & Montanari , 2019 ) showed that learning regimes that interpolate/memorize data can also have generalization guarantees . Proving generalization for our architectures is an interesting open problem . 2 PRELIMINARIES . In this section , we introduce notations , the two neural network architectures used for Theorem 1 and 2 , and the perturbed gradient descent algorithm . 2.1 NOTATIONS . We use [ n ] to denote the set { 1 , 2 , ... , n } . For a vector x , we use ‖x‖2 to denote its ` 2 norm , and sometimes ‖x‖ as a shorthand . For a matrix M , we use ‖M‖F to denote its Frobenius norm , ‖M‖ to denote its spectral norm . We will also use λi ( M ) and σi ( M ) to denote the i-th largest eigenvalue and singular value of matrix M , and λmin ( M ) , σmin ( M ) to denote the smallest eigenvalue/singular value . For the results of three-layer networks , our activation is going to be xp , where p is considered as a small constant . We use Op ( ) , Ωp ( ) to hide factors that only depend on p. For vectors x , y ∈ Rd , the tensor product is denoted by ( x ⊗ x ) ∈ Rd2 . We use x⊗p ∈ Rdp as a shorthand for p-th power of x in terms of tensor product . For two matrices M , N ∈ Rd1×d2 , we use M ⊗N ∈ Rd21×d22 denote the Kronecker product of 2 matrices . 2.2 NETWORK ARCHITECTURES . In this section , we introduce the neural net architectures we use . As we discussed , Theorem 1 uses a two-layer network ( see Figure 1 ( a ) ) and Theorem 2 uses a three-layer network ( see Figure 1 ( b ) ) . Two-layer Neural Network For the two-layer neural network , suppose the input samples x are in Rd , the hidden layer has r hidden neurons ( for simplicity , we assume r is even , in Theorem 4 we will show that r = 2d+ 2 is enough ) . The activation function of the hidden layer is σ ( x ) = x2 . We use wi ∈ Rd to denote the input weight of hidden neuron i . These weight vectors are collected as a weight matrix W = [ w1 , w2 , . . . , wr ] ∈ Rd×r . The output layer has only 1 neuron , and we use ai ∈ R to denote the its input weight from hidden neuron i . There is no nonlinearity for the output layer . For simplicity , we fix the parameters ai , i ∈ [ r ] in the way that ai = 1 for all 1 ≤ i ≤ r2 and ai = −1 for all r2 + 1 ≤ i ≤ r. Given x as the input , the output of the neural network is y = ∑r i=1 ai ( w T i x ) 2 . Hidden layer Input layer Output layer Hidden layer 1 Hidden layer 2 Input layer Output layer If the training samples are { ( xj , yj ) } j≤n , we define the empirical risk of the neural network with parameters W to be f ( W ) = 1 4n n∑ j=1 ( r∑ i=1 ai ( w T i xj ) 2 − yj ) 2 . Three-layer neural network For Theorem 2 , we use a more complicated , three-layer neural network . In this network , the first layer has a polynomial activation τ ( x ) = xp , and the next two layers are the same as the two-layer network . We use R = [ r1 , . . . , rk ] T ∈ Rk×d to denote the weight parameter of the first layer . The first hidden layer has k neurons with activation τ ( x ) = xp where p is the parameter in Theorem 2 . Given input x , the output of the first hidden layer is denoted as z , and satisfy zi = ( rTi xj ) p.The second hidden layer has r neurons ( again we will later show r = 2k + 2 is enough ) . The weight matrix for second layer is denoted as W = [ w1 , . . . , wr ] ∈ Rk×r where each wi ∈ Rk is the weight for a neuron in the second hidden layer . The activation for the second hidden layer is σ ( x ) = x2 . The third layer has weight a and is initialized the same way as before , where a1 = a2 = · · · = ar/2 = 1 , and ar/2+1 = · · · = ar = −1 . The final output y can be computed as y = ∑r i=1 ai ( w T i z ) 2 . Given inputs ( x1 , y1 ) , ... , ( xn , yn ) , suppose zi is the output of the first hidden layer for xi , the empirical loss is defined as : f ( W ) = 1 4n n∑ j=1 ( r∑ i=1 ai ( w T i zj ) 2 − yj ) 2 . Note that only the second-layer weight W is trainable . The first layer with weights R acts like a random feature layer that maps xi ’ s into a new representation zi ’ s .

Summary: This paper studies the mildly over-parameterized neural networks. In particular, it shows that when the width is at least O(sqrt{n}) where n is the number of samples, PDG fits the training data. The analysis is done for 2-layer or 3-layer networks with (mainly) quadratic activations, with only one-layer weights (the weights before the quadratic activation) trainable. For 2-layer network, that means the first layer weights are trainable. Except for O(sqrt{n}) width, the other settings of this paper are quite restricted and it is not clear how it reflects the training of the neural networks.