No Spurious Local Minima: on the Optimization Landscapes of Wide and Deep Neural Networks

Empirical studies suggest that wide neural networks are comparably easy to optimize , but mathematical support for this observation is scarce . In this paper , we analyze the optimization landscapes of deep learning with wide networks . We prove especially that constraint and unconstraint empirical-risk minimization over such networks has no spurious local minima . Hence , our theories substantiate the common belief that increasing network widths not only improves the expressiveness of deep-learning pipelines but also facilitates their optimizations . 1 INTRODUCTION . Deep learning depends on optimization problems that seem impossible to solve , and yet , deeplearning pipelines outperform their competitors in many applications . A common suspicion is that the optimizations are often easier than they appear to be . In particular , while most objective functions are nonconvex and , therefore , might have spurious local minima , recent findings suggest that optimizations are not hampered by spurious local minima as long as the neural networks are sufficiently wide . For example , Dauphin et al . ( 2014 ) suggest that saddle points , rather than local minima , are the main challenges for optimizations over wide networks ; Goodfellow et al . ( 2014 ) give empirical evidence for stochastic-gradient descent to converge to a global minimum of the objective function of wide networks ; Livni et al . ( 2014 ) show that the optimizations over some classes of wide networks can be reduced to a convex problem ; Soudry & Carmon ( 2016 ) suggest that differentiable local minima of objective functions over wide networks are typically global minima ; Nguyen & Hein ( 2018 ) indicate that critical points in wide networks are often global minima ; and Allen-Zhu et al . ( 2019 ) and Du et al . ( 2019 ) suggest that stochastic-gradient descent typically converges to a global minimum for large networks . These findings raise the question of whether common optimization landscapes over wide ( but finite ) neural networks have no spurious local minima altogether . Progress in this direction has recently been made in Venturi et al . ( 2019 ) and then Lacotte & Pilanci ( 2020 ) . Broadly speaking , we call a local minimum spurious if there is no nonincreasing path to a global minimum ( see Section 2.2 for a formal definition ) . While the absence of spurious local minima does not preclude saddle points or suboptimal local minima in general , it means that one can move from every local minimum to a global minimum without increasing the objective function at any point—see Figure 1 for an illustration . Venturi et al . ( 2019 ) prove that there are no spurious local minima if the networks are sufficiently wide . Their theory has two main features that had not been established before : First , it holds for the entire landscapes—rather than for subsets of them . This feature is crucial : even randomized algorithms typically converge to sets of Lebesgue measure zero with probability one , that is , statements about “ almost all ” local minima are not necessarily meaningful . Second , their theory allows for arbitrary convex loss functions . This feature is important , for example , in view of the trends toward robust alternatives of the least-squares loss ( Belagiannis et al. , 2015 ; Jiang et al. , 2018 ; Wang et al. , 2016 ) . On the other hand , their theory has three major limitations : it is restricted to polynomial activation , which is convenient mathematically but much less popular than ReLU activation , it disregards regularizers and constraints , which have become standard in deep learning and in machine learning at large ( Hastie et al. , 2015 ) , and it restricts to shallow networks , that is , networks with only one hidden layer , which contrasts the deep architectures that are used in practice ( LeCun et al. , 2015 ) . Lacotte & Pilanci ( 2020 ) made progress on two of these limitations : first , their theory caters to ReLU activation rather than polynomial activation ; second , their theory allows for weight decay , which is a standard way to regularize estimators . However , their work is still restricted to one-hidden-layer networks . The interesting question is , therefore , whether such results can also be established for deep networks . And more generally , it would be highly desirable to have a theory for the absence of spurious local minima in a broad deep-learning framework . In this paper , we establish such a theory . We prove that the optimization landscapes of empirical-risk minimizers over wide feedforward networks have no spurious local minima . Our theory combines the features of the two mentioned works , as it applies to the entire optimization landscapes , allows for a wide spectrum of loss functions and activation functions , and constraint and unconstraint estimation . Moreover , it generalizes these works , as it allows for multiple outputs and arbitrary depths . Additionally , our proof techniques are considerably different from the ones used before and , therefore , might be of independent interest . Guide to the paper Sections 2 and 5 are the basic parts of the paper : they contain our main result and a short discussion of its implications . Readers who are interested in the underpinning principles should also study Section 3 , and readers who want additional insights on the proof techniques are referred to Section 4 . The actual proofs are stated in the Appendix . 2 DEEP-LEARNING FRAMEWORK AND MAIN RESULT . In this section , we specify the deep-learning framework and state our main result . The framework includes a wide range of feedforward neural networks ; in particular , it allows for arbitrarily many outputs and layers , a range of activation and loss functions , and constraint as well as unconstraint estimation . Our main result guarantees that if the networks are sufficiently wide , the objective function of the empirical-risk minimizer does not have any spurious local minima . 2.1 FEEDFORWARD NEURAL NETWORKS . We consider input data from a domain Dx ⊂ Rd and output data from a domain Dy ⊂ Rm . Typical examples are regression data with Dy = Rm and classification data with Dy = { ±1 } m. We model the data with layered , feedforward neural networks , that is , we study sets of functions G ··= { gΘ : Dx → Rm : Θ ∈M } ⊂ G ··= { gΘ : Dx → Rm : Θ ∈M } with gΘ [ x ] ··= Θlfl [ Θl−1 · · · f1 [ Θ0x ] ] for x ∈ Dx ( 1 ) and M⊂M ··= { Θ = ( Θl , . . . , Θ0 ) : Θj ∈ Rp j+1×pj } . The quantities p0 = d and pl+1 = m are the input and output dimensions , respectively , l the depth of the networks , and w ··= min { p1 , . . . , pl } the minimal width of the networks . The functions fj : Rpj → Rpj are called the activation functions . We assume that the activation functions are elementwise functions in the sense that fj [ b ] = ( fj [ b1 ] , . . . , fj [ bpj ] ) > for all b ∈ Rp j , where fj : R→ R is an arbitrary function . This allows for an unlimited variety in the type of activation , including ReLU fj : b 7→ max { 0 , b } , leaky ReLU fj : b 7→ max { 0 , b } + min { 0 , cb } for a fixed c ∈ ( 0 , 1 ) , polynomial fj : b 7→ cbk for fixed c ∈ ( 0 , ∞ ) and k ∈ [ 1 , ∞ ) , and sigmoid activation fj : b 7→ 1/ ( 1 + e−b ) as popular examples , and it allows for different activation functions in each layer . We study the most common approaches to parameter estimation in this setting : constraint and unconstraint empirical-risk minimization . The loss function l : Rm × Rm → R is assumed convex in its first argument ; this includes all standard loss functions , such as the least-squares loss l : ( a , b ) 7→ ||a − b||22 , the absolut-deviation loss l : ( a , b ) 7→ ||a − b||1 ( both typically used for regression ) , the logistic loss l : ( a , b ) 7→ − ( 1 + b ) log [ 1 + a ] − ( 1 − b ) log [ 1 − a ] , the hinge loss l : ( a , b ) 7→ max { 0 , 1− ab } ( both typically used for binary classification Dy = { ±1 } ) , and so forth . The optimization domain is the set M ··= { Θ ∈M : r [ Θ ] ≤ 1 } for a constraint r : M → R. Given data ( x1 , y1 ) , . . . , ( xn , yn ) ∈ Dx × Dy , the empirical-risk minimizers are then the networks gΘ̂erm with Θ̂erm ∈ arg min Θ∈M { n∑ i=1 l [ gΘ [ xi ] , yi ] } . ( 2 ) It has been shown that constraints can facilitate the optimization as well as improve generalization— see Krizhevsky et al . ( 2012 ) and Livni et al . ( 2014 ) , among others . For ease of presentation , we limit ourselves to the following class of constraints : r [ Θ ] ··= max { ar max j∈ { 1 , ... , l } |||Θj |||1 , br|||Θ0|||q } for all Θ ∈M ( 3 ) for fixed tuning parameters ar , br ∈ [ 0 , ∞ ) , a parameter q ∈ ( 0 , ∞ ] , and ||| · |||q the usual row-wise ` q “ -norm , ” that is , |||Θj |||q ··= maxk ( ∑ i| ( Θj ) ki|q ) 1/q for q ∈ ( 0 , ∞ ) and |||Θj |||∞ ··= maxki| ( Θj ) ki| . This class of constraints includes the following four important cases : • Unconstraint estimation : ar = br = 0 . In other words , M =M . Unconstraint estimation had been the predominant approach in the earlier days of deep learning and is still used today ( Anthony & Bartlett , 1999 ) . • Connection sparsity : q = 1 . This constraint yields connection-sparse networks , which have received considerable attention recently ( Barron & Klusowski , 2018 ; 2019 ; Kim et al. , 2016 ; Taheri et al. , 2020 ) . • Strong sparsity : q < 1 . Nonconvex constraints have been popular in statistics for many years ( Fan & Li , 2001 ; Zhang , 2010 ) , but our paper is probably the first one that includes such constraints in a theoretical analysis in deep learning . • Input constraints : ar = 0 . Some researchers have argued for applying certain constraints , such as node-sparsity , only to the input level ( Feng & Simon , 2017 ) . In general , while our proof techniques also apply to many other types of constraints , there are two main reasons for using the mentioned sparsity-inducing constraints to illustrate our results : First , sparsity has become very popular in deep learning , because it can lower the burden on memory and optimization as well as increase interpretability ( Hebiri & Lederer , 2020 ) . And second , the above examples allow us to demonstrate that the discussed features of wide networks do not depend on smooth and convex constraints such as weight decay . Our theory can also be adjusted to the regularized versions of the empirical-risk minimizers , that is , for the networks indexed by any parameter in the set arg min Θ∈M { n∑ i=1 l [ gΘ [ xi ] , yi ] + r [ Θ ] } . The proofs are virtually the same as for the constraint versions ; we omit the details for the sake of brevity . One line of research develops statistical theories for constraint and unconstraint empirical-risk minimizers—see Bartlett & Mendelson ( 2002 ) and Lederer ( 2020 ) , among others . As detailed above , empirical-risk minimizers are the networks whose parameters are global minima of the objective function Θ 7→ l [ gΘ ] ··= n∑ i=1 l [ gΘ [ xi ] , yi ] ( 4 ) over M for fixed data ( x1 , y1 ) , . . . , ( xn , yn ) . While the function gΘ 7→ l [ gΘ ] is convex by assumption , the objective function Θ 7→ l [ gΘ ] is usually nonconvex . It is thus unclear , per se , whether deep-learning pipelines can be expected to yield global minima of the objective function and , therefore , whether the statistical theories are valid in practice . Our goal is , broadly speaking , to establish conditions under which global minimization of ( 4 ) can indeed be expected .

Summary: This paper studies the optimization landscape of the training loss of deep neural networks. For  a general setup, the paper shows that if the network width is greater than $2m(n+1)^l$, then any parameter value has a path to a global minimum on which the training loss does not increase. Here, $m$ is the output dimension, $n$ is the number of training examples, and $l$ is the number of hidden layers.