The loss landscape of overparameterized neural networks

1 INTRODUCTION . In recent years , it has become clear that neural nets are incredibly effective at a wide variety of tasks . Why they work so well is less understood . During training neural nets implement a form of curve fitting , by minimizing a loss function L : Rn → R. Evidence suggests that they work better when there are more parameters than data points . Here we explore the geometry of the loss landscape of overparameterized neural networks , as a first step toward understanding their unreasonable effectiveness . A priori one might imagine that the loss function L of a neural network looks like a typical function from Rn to R - in particular , nonconvex , with discrete global minima , many “ bad ” local minima that are not global minima , and complicated geometry . However , it turns out that in at least one important way , the loss function of an overparameterized neural network does not look like a typical function . In the overparameterized setting where the neural net has n parameters and is trained on d data points , with n > d , it is generally the case that L is nonconvex . However , the locus M ⊂ Rn of global minima of L is often not discrete . Indeed we find M is often quite the opposite – rather than being a discrete set , M is often an n− d dimensional submanifold of Rn . We expect this to be true in a very general setting . The neural network can be any depth , and while our strongest results are for feedforward neural networks , many of the results in this paper apply for a broader set of architectures . In practice , neural nets commonly have orders of magnitude more weights than data points , so in practical applications M is typically a very high-dimensional submanifold of Rn . Our contributions include : • Giving a heuristic that explains that in the overparameterized setting , generically the locus of global minima of the loss function should be an n− d dimensional submanifold of Rn , and that at a global minimum , the Hessian ofL has d positive eigenvalues , n−d eigenvalues equal to 0 , and no negative eigenvalues . • Proving that for a broad class of activation functions , a large enough feedforward neural network with one hidden layer can fit a fixed training dataset exactly . • Proving that for an overparameterized feedforward neural network with a rectified smooth activation function , the locus of global minima for the loss function is a nonempty smooth n− d dimensional submanifold of Rn . These results provide a theoretical setting for several lines of recent work . It has been widely suggested that gradient descent is successful in producing good solutions in deep learning because many local minima are close to global minima and give similar results LeCun et al . ( 2015 ) , Choromanska et al . ( 2015 ) . As a result , the loss landscape has been studied in papers such as Freeman & Bruna ( 2017 ) . Several authors have proved that under various strong assumptions , all local minima are global Liang et al . ( 2018 ) , Ge et al . ( 2017 ) , Laurent & Brecht ( 2017 ) , Kawaguchi ( 2016 ) . This is not true in “ real world ” settings for deep learning . But recent work has suggested that in some settings there are paths between local minima such that the value of the loss function stays low along the paths Garipov et al . ( 2018 ) , Draxler et al . ( 2018 ) . We show a stronger result in a more general setting . We prove that in a very general setting , there are not only one-dimensional paths between minima , but very high dimensional manifolds . In Section 2 , we study the geometry of the loss landscape in the general overparameterized setting . In Section 2.1 we study the global geometry of the landscape . We study the locus M = L−1 ( 0 ) because L is a nonnegative function , so as long as M is nonempty , M is the locus of global minima of L. We show that if nonempty , M is an n − d dimensional submanifold of Rn . In Section 2.2 , we consider the local geometry of the loss landscape near M , and show that if M is nonempty then at a global minimum m ∈ M , the Hessian of L has n − d zero eigenvalues , d positive ones , and no negative ones . This is consistent with previous observations in papers such as Chaudhari et al . ( 2016 ) and Wu et al . ( 2017 ) . In Section 3 , we write down a proof of a commonly believed fact , that given a fixed dataset , a large enough one-hidden-layer feedforward neural net can memorize it exactly , i.e . learn it with zero training error . This is commonly assumed , and in the case that the activation function used is ReLU , proved in Zhang et al . ( 2016 ) . We require a somewhat more general statement , but our argument is based on the one given there . In Section 4 , we combine the results of the previous two sections to show that in the setting of overparameterized feedforward neural networks , if they are wide enough then the locus M = L−1 ( 0 ) is in fact a nonempty smooth n− d dimensional submanifold of Rn . 1.1 ASSUMPTIONS . In this paper , we will always consider the overparameterized setting . So in all of our analyses , we assume that the number of parameters n of the neural net is greater than the number of data points d that it is training on . 2 GEOMETRY OF THE LOSS LANDSCAPE L - GENERAL CASE . 2.1 POSITIVE DIMENSIONALITY OF M. Suppose we have a neural net of any architecture ( e.g . feedforward , LSTM , etc . ) , with weights ( w1 , ... ) and biases ( b1 , ... ) , n parameters in total . Suppose this net uses a smooth activation function σ and is training on a data set { ( xi → yi ) } with d data points . Each entry in the data set is given as a pair of vectors , xi ∈ Rp ( e.g . a vector of pixel values ) and yi ∈ R. We assume that the xi are distinct and that n > d. Let fw , b be the function given by the neural net with the parameters w , b . For each data points ( xi → yi ) , let fi ( w , b ) = fw , b ( xi ) − yi . Assume that each fi ( w , b ) is smooth in w and b . For most choices of architecture , this is implied by the assumption that the activation function σ is smooth . For example , for any feedforward neural network , if σ is smooth then fi ( w , b ) is smooth for each i . Let the loss function used in training be the commonly used L ( w , b ) = ∑ ( fw , b ( xi ) − yi ) 2 . Define fi ( w , b ) : Rn → R as fi ( w , b ) = fw , b ( xi ) − yi , so L ( w , b ) can be written as L ( w , b ) = ∑ fi ( w , b ) 2 . It is clear from the definition that L ( w , b ) ≥ 0 , so if nonempty , M = L−1 ( 0 ) is the locus of global minima of L. Therefore , in this section we will focus on understanding the geometry of M in the case that it is nonempty . Note that M = ⋂ Mi , where Mi = fi ( w , b ) −1 ( 0 ) . The rest of this section is devoted to the following theorem . Theorem 2.1 . In the setting described above , the set M = L−1 ( 0 ) is generically ( that is , possibly after an arbitrarily small change to the data set ) a smooth n− d dimensional submanifold ( possibly empty ) of Rn . Remark 2.2 . The loss function L̂ ( w , b ) = ∑ |fi ( w , b ) | is also regularly used . Note that L̂−1 ( 0 ) = L−1 ( 0 ) , so Theorem 2.1 also shows that the locus M̂ = L̂−1 ( 0 ) is a smooth n− d dimensional submanifold of Rn . We start with a heuristic argument for Theorem 2.1 . Heuristic : For each fi , the regular values of fi : Rn → R are dense . So generically we expect that each Mi = f−1i ( 0 ) is a smooth codimension 1 submanifold of Rn . Generically the intersection of d smooth codimension 1 submanifolds of Rn is a smooth codimension d submanifold , so one would expect that M = ⋂ Mi is smooth of codimension d. With this heuristic in mind , we proceed to a proof of Theorem 2.1 . Proof . We construct a function H related to L. Let H : Rn → Rd be defined as the function H ( w , b ) = ( f1 ( w , b ) , ... , fd ( w , b ) ) . By construction , L = |H|2 , and importantly , M = L−1 ( 0 ) = H−1 ( ( 0 , ... , 0 ) ) . If ( 0 , ... , 0 ) is a regular value of H , proceed to next paragraph . If not , pick any > 0 . By Sard ’ s theorem , regular values of H are generic . We may therefore choose a regular value r = ( r1 , ... , rd ) with |r| < . We use r to perturb the data , replacing xi → yi by xi → ỹi where ỹi = yi + ri . Let f̃i ( w , b ) = fi ( w , b ) − ri = fw , b ( xi ) − ỹi , L̃ ( w , b ) = ∑ f̃2i , and H̃ ( w , b ) = ( f̃1 ( w , b ) , ... , f̃d ( w , b ) ) = H ( w , b ) − ( r1 , ... , rd ) . Note that | ( x , y ) − ( x , ỹ ) | < , meaning that after taking an arbitrarily small perturbation of the data , ( 0 , ... , 0 ) is a regular value of H̃ . After possibly replacing H by H̃ , ( 0 , ... , 0 ) is a regular value of H , meaning M = H−1 ( ( 0 , ... , 0 ) ) is either empty or smooth of codimension d in Rn , as desired . 2.2 THE LOCAL GEOMETRY OF THE LOSS FUNCTION NEAR M. Proposition 2.3 . Consider the submanifold M = L−1 ( 0 ) = ⋂ Mi , where Mi = f−1i ( 0 ) . If each Mi is a smooth codimension 1 submanifold of Rn , M is nonempty , and the Mi intersect transversally at every point of M , then at any point m ∈M , the Hessian of L evaluated at m has d positive eigenvalues and n− d eigenvalues equal to 0 . Proof . Let H ( L ) denote the Hessian of L. At every point p ∈ Rn , the Hessian of L evaluated at p , H ( L ) |p , is a real symmetric matrix , hence has a basis of eigenvectors . Since m is a minimum ( in fact a global minimum ) of L , the eigenvalues of H ( L ) |m are nonnegative . Since M is n − d dimensional , the kernel ofH ( L ) |m is at least n− d dimensional . It remains to show that it is also at most n− d dimensional . Well , L = ∑d i=1 f 2 i , so H ( L ) = d∑ i=1 H ( f2i ) . Let us consider each summand . The linear transformation H ( f2i ) has n − 1 eigenvalues equal to 0 and a unique nonzero eigenvalue λi . This follows from the fact that Mi is a smooth codimension 1 submanifold . Furthermore , λi is positive as 0 is a global minimum of fi . Let vi denote the eigenvector corresponding to this positive eigenvalue . The vectors v1 , ... , vd are linearly independent because the Mi intersect transversally at m. An elementary linear algebra calculation shows that if an n × n matrix A is the sum of d matrices A = ∑d i=1Ai , n > d , where each Ai has a unique nonzero eigenvalue λi and the corresponding vectors vi are all linearly independent , then A has d nonnegative eigenvalues and n− d eigenvalues equal to 0 . We conclude that in our setting , H ( L ) has d positive eigenvalues and n− d eigenvalues equal to 0 , as desired .

Summary: The paper shows that the set of global minimums of an overparametrized network with smooth activation function is almost surely a high dimensional manifold. In particular, the dimension of this manifold is exactly n-d, where n is the number of parameters and d is the number of datas. To the best of my knowledge, this is the first proper proof of such non-surprising result.