Gradient Descent Provably Optimizes Over-parameterized Neural Networks

One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth . This paper demystifies this surprising phenomenon for two-layer fully connected ReLU activated neural networks . For anm hidden node shallow neural network with ReLU activation and n training data , we show as long asm is large enough and no two inputs are parallel , randomly initialized gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function . Our analysis relies on the following observation : over-parameterization and random initialization jointly restrict every weight vector to be close to its initialization for all iterations , which allows us to exploit a strong convexity-like property to show that gradient descent converges at a global linear rate to the global optimum . We believe these insights are also useful in analyzing deep models and other first order methods . 1 INTRODUCTION . Neural networks trained by first order methods have achieved a remarkable impact on many applications , but their theoretical properties are still mysteries . One of the empirical observation is even though the optimization objective function is non-convex and non-smooth , randomly initialized first order methods like stochastic gradient descent can still find a global minimum . Surprisingly , this property is not correlated with labels . In Zhang et al . ( 2016 ) , authors replaced the true labels with randomly generated labels , but still found randomly initialized first order methods can always achieve zero training loss . A widely believed explanation on why a neural network can fit all training labels is that the neural network is over-parameterized . For example , Wide ResNet ( Zagoruyko and Komodakis ) uses 100x parameters than the number of training data . Thus there must exist one such neural network of this architecture that can fit all training data . However , the existence does not imply why the network found by a randomly initialized first order method can fit all the data . The objective function is neither smooth nor convex , which makes traditional analysis technique from convex optimization not useful in this setting . To our knowledge , only the convergence to a stationary point is known ( Davis et al. , 2018 ) . ∗Equal contribution . In this paper we demystify this surprising phenomenon on two-layer neural networks with rectified linear unit ( ReLU ) activation . Formally , we consider a neural network of the following form . f ( W , a , x ) = 1√ m m∑ r=1 arσ ( w > r x ) ( 1 ) where x ∈ Rd is the input , wr ∈ Rd is the weight vector of the first layer , ar ∈ R is the output weight and σ ( · ) is the ReLU activation function : σ ( z ) = z if z ≥ 0 and σ ( z ) = 0 if z < 0 . We focus on the empirical risk minimization problem with a quadratic loss . Given a training data set { ( xi , yi ) } ni=1 , we want to minimize L ( W , a ) = n∑ i=1 1 2 ( f ( W , a , xi ) − yi ) 2 . ( 2 ) Our main focus of this paper is to analyze the following procedure . We fix the second layer and apply gradient descent ( GD ) to optimize the first layer1 W ( k + 1 ) = W ( k ) − η ∂L ( W ( k ) , a ) ∂W ( k ) . ( 3 ) where η > 0 is the step size . Here the gradient formula for each weight vector is 2 ∂L ( W , a ) ∂wr = 1√ m n∑ i=1 ( f ( W , a , xi ) − yi ) arxiI { w > r xi ≥ 0 } . ( 4 ) Though this is only a shallow fully connected neural network , the objective function is still nonsmooth and non-convex due to the use of ReLU activation function . 3 Even for this simple function , why randomly initialized first order method can achieve zero training error is not known . Many previous works have tried to answer this question or similar ones . Attempts include landscape analysis ( Soudry and Carmon , 2016 ) , partial differential equations ( Mei et al . ) , analysis of the dynamics of the algorithm ( Li and Yuan , 2017 ) , optimal transport theory ( Chizat and Bach , 2018 ) , to name a few . These results often make strong assumptions on the labels and input distributions or do not imply why randomly initialized first order method can achieve zero training loss . See Section 2 for detailed comparisons between our result and previous ones . In this paper , we rigorously prove that as long as no two inputs are parallel and m is large enough , with randomly initialized a and W ( 0 ) , gradient descent achieves zero training loss at a linear convergence rate , i.e. , it finds a solution W ( K ) with L ( W ( K ) ) ≤ in K = O ( log ( 1/ ) ) iterations.4 Thus , our theoretical result not only shows the global convergence but also gives a quantitative convergence rate in terms of the desired accuracy . Analysis Technique Overview Our proof relies on the following insights . First we directly analyze the dynamics of each individual prediction f ( W , a , xi ) for i = 1 , . . . , n. This is different from many previous work ( Du et al. , 2017b ; Li and Yuan , 2017 ) which tried to analyze the dynamics of the parameter ( W ) we are optimizing . Note because the objective function is non-smooth and non-convex , analysis of the parameter space dynamics is very difficult . In contrast , we find the dynamics of prediction space is governed by the spectral property of a Gram matrix ( which can vary in each iteration , c.f . Equation ( 6 ) ) and as long as this Gram matrix ’ s least eigenvalue is lower bounded , gradient descent enjoys a linear rate . It is easy to show as long as no two inputs are parallel , in the initialization phase , this Gram matrix has a lower bounded least eigenvalue . ( c.f . Theorem 3.1 ) . Thus the problem reduces to showing the Gram matrix at later iterations is close to that in 1In Section 3.2 , we also extend our technique to analyze the setting where we train both layers jointly . 2 Note ReLU is not continuously differentiable . One can view ∂L ( W ) ∂wr as a convenient notation for the right hand side of ( 4 ) and this is the update rule used in practice . 3We remark that if one fixes the first layer and only optimizes the output layer , then the problem becomes a convex and smooth one . If m is large enough , one can show the global minimum has zero training loss ( Nguyen and Hein , 2018 ) . Though for both cases ( fixing the first layer and fixing the output layer ) , gradient descent achieves zero training loss , the learned prediction functions are different . 4Here we omit the polynomial dependency on n and other data-dependent quantities . the initialization phase . Our second observation is this Gram matrix is only related to the activation patterns ( I { w > r xi ≥ 0 } ) and we can use matrix perturbation analysis to show if most of the patterns do not change , then this Gram matrix is close to its initialization . Our third observation is we find over-parameterization , random initialization , and the linear convergence jointly restrict every weight vector wr to be close to its initialization . Then we can use this property to show most of the patterns do not change . Combining these insights we prove the first global quantitative convergence result of gradient descent on ReLU activated neural networks for the empirical risk minimization problem . Notably , our proof only uses linear algebra and standard probability bounds so we believe it can be easily generalized to analyze deep neural networks . Notations We let [ n ] = { 1 , 2 , . . . , n } . Given a set S , we use unif { S } to denote the uniform distribution over S. Given an event E , we use I { A } to be the indicator on whether this event happens . We use N ( 0 , I ) to denote the standard Gaussian distribution . For a matrix A , we use Aij to denote its ( i , j ) -th entry . We use ‖·‖2 to denote the Euclidean norm of a vector , and use ‖·‖F to denote the Frobenius norm of a matrix . If a matrix A is positive semi-definite , we use λmin ( A ) to denote its smallest eigenvalue . We use 〈· , ·〉 to denote the standard Euclidean inner product between two vectors . 2 COMPARISON WITH PREVIOUS RESULTS . In this section , we survey an incomplete list of previous attempts in analyzing why first order methods can find a global minimum . Landscape Analysis A popular way to analyze non-convex optimization problems is to identify whether the optimization landscape has some good geometric properties . Recently , researchers found if the objective function is smooth and satisfies ( 1 ) all local minima are global and ( 2 ) for every saddle point , there exists a negative curvature , then the noise-injected ( stochastic ) gradient descent ( Jin et al. , 2017 ; Ge et al. , 2015 ; Du et al. , 2017a ) can find a global minimum in polynomial time . This algorithmic finding encouraged researchers to study whether the deep neural networks also admit these properties . For the objective function defined in Equation ( 2 ) , some partial results were obtained . Soudry and Carmon ( 2016 ) showed if md ≥ n , then at every differentiable local minimum , the training error is zero . However , since the objective is non-smooth , it is hard to show gradient descent convergences to a differentiable local minimum . Xie et al . ( 2017 ) studied the same problem and related the loss to the gradient norm through the least singular value of the “ extended feature matrix ” D at the stationary points . However , they did not prove the convergence rate of the gradient norm . Interestingly , our analysis relies on the Gram matrix which is DD > . Landscape analyses of ReLU activated neural networks for other settings have also been studied in many previous works ( Ge et al. , 2017 ; Safran and Shamir , 2016 ; Zhou and Liang , 2017 ; Freeman and Bruna , 2016 ; Hardt and Ma , 2016 ; Nguyen and Hein , 2018 ) . These works establish favorable landscape properties but none of them implies that gradient descent converges to a global minimizer of the empirical risk . More recently , some negative results have also been discovered ( Safran and Shamir , 2018 ; Yun et al. , 2018a ) and new procedures have been proposed to test local optimality and escape strict saddle points at non-differentiable points ( Yun et al. , 2018b ) . However , the new procedures can not find global minima as well . For other activation functions , some previous works showed the landscape does have the desired geometric properties ( Du and Lee , 2018 ; Soltanolkotabi et al. , 2018 ; Nguyen and Hein , 2017 ; Kawaguchi , 2016 ; Haeffele and Vidal , 2015 ; Andoni et al. , 2014 ; Venturi et al. , 2018 ; Yun et al. , 2018a ) . However , it is unclear how to extend their analyses to our setting . Analysis of Algorithm Dynamics Another way to prove convergence result is to analyze the dynamics of first order methods directly . Our paper also belongs to this category . Many previous works assumed ( 1 ) the input distribution is Gaussian and ( 2 ) the label is generated according to a planted neural network . Based on these two ( unrealistic ) conditions , it can be shown that randomly initialized ( stochastic ) gradient descent can learn a ReLU ( Tian , 2017 ; Soltanolkotabi , 2017 ) , a single convolutional filter ( Brutzkus and Globerson , 2017 ) , a convolutional neural network with one filter and one output layer ( Du et al. , 2018b ) and residual network with small spectral norm weight matrix ( Li and Yuan , 2017 ) .5 Beyond Gaussian input distribution , Du et al . ( 2017b ) showed for learning a convolutional filter , the Gaussian input distribution assumption can be relaxed but they still required the label is generated from an underlying true filter . Comparing with these work , our paper does not try to recover the underlying true neural network . Instead , we focus on providing theoretical justification on why randomly initialized gradient descent can achieve zero training loss , which is what we can observe and verify in practice . Jacot et al . ( 2018 ) established an asymptotic result showing for the multilayer fully-connected neural network with a smooth activation function , if every layer ’ s weight matrix is infinitely wide , then for finite training time , the convergence of gradient descent can be characterized by a kernel . Our proof technique relies on a Gram matrix which is the kernel matrix in their paper . Our paper focuses on the two-layer neural network with ReLU activation function ( non-smooth ) and we are able to prove the Gram matrix is stable for infinite training time . The most related paper is by Li and Liang ( 2018 ) who observed that when training a two-layer full connected neural network , most of the patterns ( I { w > r xi ≥ 0 } ) do not change over iterations , which we also use to show the stability of the Gram matrix . They used this observation to obtain the convergence rate of GD on a two-layer over-parameterized neural network for the cross-entropy loss . They need the number of hidden nodes m scales with poly ( 1/ ) where is the desired accuracy . Thus unless the number of hidden nodes m → ∞ , their result does not imply GD can achieve zero training loss . We improve by allowing the amount of over-parameterization to be independent of the desired accuracy and show GD can achieve zero training loss . Furthermore , our proof is much simpler and more transparent so we believe it can be easily generalized to analyze other neural network architectures . Other Analysis Approaches Chizat and Bach ( 2018 ) used optimal transport theory to analyze continuous time gradient descent on over-parameterized models . They required the second layer to be infinitely wide and their results on ReLU activated neural network is only at the formal level . Mei et al . analyzed SGD for optimizing the population loss and showed the dynamics can be captured by a partial differential equation in the suitable scaling limit . They listed some specific examples on input distributions including mixture of Gaussians . However , it is still unclear whether this framework can explain why first order methods can minimize the empirical risk . Daniely ( 2017 ) built connection between neural networks with kernel methods and showed stochastic gradient descent can learn a function that is competitive with the best function in the conjugate kernel space of the network . Again this work does not imply why first order methods can achieve zero training loss .

Summary: This work considers optimizing a two-layer over-parameterized ReLU network with the squared loss and given a data set with arbitrary labels. It is shown that for a sufficiently large number of hidden neurons (polynomially in number of samples) gradient descent converges to a global minimum with a linear convergence rate. The proof idea is to show that a certain Gram matrix of the data, which depends also on the weights, has a lower bounded minimum eigenvalue throughout the optimization process. Then, it is shown that this property implies convergence of gradient descent.