No Spurious Local Minima: on the Optimization Landscapes of Wide and Deep Neural Networks

Empirical studies suggest that wide neural networks are comparably easy to optimize , but mathematical support for this observation is scarce . In this paper , we analyze the optimization landscapes of deep learning with wide networks . We prove especially that constraint and unconstraint empirical-risk minimization over such networks has no spurious local minima . Hence , our theories substantiate the common belief that increasing network widths not only improves the expressiveness of deep-learning pipelines but also facilitates their optimizations . 1 INTRODUCTION . Deep learning depends on optimization problems that seem impossible to solve , and yet , deeplearning pipelines outperform their competitors in many applications . A common suspicion is that the optimizations are often easier than they appear to be . In particular , while most objective functions are nonconvex and , therefore , might have spurious local minima , recent findings suggest that optimizations are not hampered by spurious local minima as long as the neural networks are sufficiently wide . For example , Dauphin et al . ( 2014 ) suggest that saddle points , rather than local minima , are the main challenges for optimizations over wide networks ; Goodfellow et al . ( 2014 ) give empirical evidence for stochastic-gradient descent to converge to a global minimum of the objective function of wide networks ; Livni et al . ( 2014 ) show that the optimizations over some classes of wide networks can be reduced to a convex problem ; Soudry & Carmon ( 2016 ) suggest that differentiable local minima of objective functions over wide networks are typically global minima ; Nguyen & Hein ( 2018 ) indicate that critical points in wide networks are often global minima ; and Allen-Zhu et al . ( 2019 ) and Du et al . ( 2019 ) suggest that stochastic-gradient descent typically converges to a global minimum for large networks . These findings raise the question of whether common optimization landscapes over wide ( but finite ) neural networks have no spurious local minima altogether . Progress in this direction has recently been made in Venturi et al . ( 2019 ) and then Lacotte & Pilanci ( 2020 ) . Broadly speaking , we call a local minimum spurious if there is no nonincreasing path to a global minimum ( see Section 2.2 for a formal definition ) . While the absence of spurious local minima does not preclude saddle points or suboptimal local minima in general , it means that one can move from every local minimum to a global minimum without increasing the objective function at any point—see Figure 1 for an illustration . Venturi et al . ( 2019 ) prove that there are no spurious local minima if the networks are sufficiently wide . Their theory has two main features that had not been established before : First , it holds for the entire landscapes—rather than for subsets of them . This feature is crucial : even randomized algorithms typically converge to sets of Lebesgue measure zero with probability one , that is , statements about “ almost all ” local minima are not necessarily meaningful . Second , their theory allows for arbitrary convex loss functions . This feature is important , for example , in view of the trends toward robust alternatives of the least-squares loss ( Belagiannis et al. , 2015 ; Jiang et al. , 2018 ; Wang et al. , 2016 ) . On the other hand , their theory has three major limitations : it is restricted to polynomial activation , which is convenient mathematically but much less popular than ReLU activation , it disregards regularizers and constraints , which have become standard in deep learning and in machine learning at large ( Hastie et al. , 2015 ) , and it restricts to shallow networks , that is , networks with only one hidden layer , which contrasts the deep architectures that are used in practice ( LeCun et al. , 2015 ) . Lacotte & Pilanci ( 2020 ) made progress on two of these limitations : first , their theory caters to ReLU activation rather than polynomial activation ; second , their theory allows for weight decay , which is a standard way to regularize estimators . However , their work is still restricted to one-hidden-layer networks . The interesting question is , therefore , whether such results can also be established for deep networks . And more generally , it would be highly desirable to have a theory for the absence of spurious local minima in a broad deep-learning framework . In this paper , we establish such a theory . We prove that the optimization landscapes of empirical-risk minimizers over wide feedforward networks have no spurious local minima . Our theory combines the features of the two mentioned works , as it applies to the entire optimization landscapes , allows for a wide spectrum of loss functions and activation functions , and constraint and unconstraint estimation . Moreover , it generalizes these works , as it allows for multiple outputs and arbitrary depths . Additionally , our proof techniques are considerably different from the ones used before and , therefore , might be of independent interest . Guide to the paper Sections 2 and 5 are the basic parts of the paper : they contain our main result and a short discussion of its implications . Readers who are interested in the underpinning principles should also study Section 3 , and readers who want additional insights on the proof techniques are referred to Section 4 . The actual proofs are stated in the Appendix . 2 DEEP-LEARNING FRAMEWORK AND MAIN RESULT . In this section , we specify the deep-learning framework and state our main result . The framework includes a wide range of feedforward neural networks ; in particular , it allows for arbitrarily many outputs and layers , a range of activation and loss functions , and constraint as well as unconstraint estimation . Our main result guarantees that if the networks are sufficiently wide , the objective function of the empirical-risk minimizer does not have any spurious local minima . 2.1 FEEDFORWARD NEURAL NETWORKS . We consider input data from a domain Dx ⊂ Rd and output data from a domain Dy ⊂ Rm . Typical examples are regression data with Dy = Rm and classification data with Dy = { ±1 } m. We model the data with layered , feedforward neural networks , that is , we study sets of functions G ··= { gΘ : Dx → Rm : Θ ∈M } ⊂ G ··= { gΘ : Dx → Rm : Θ ∈M } with gΘ [ x ] ··= Θlfl [ Θl−1 · · · f1 [ Θ0x ] ] for x ∈ Dx ( 1 ) and M⊂M ··= { Θ = ( Θl , . . . , Θ0 ) : Θj ∈ Rp j+1×pj } . The quantities p0 = d and pl+1 = m are the input and output dimensions , respectively , l the depth of the networks , and w ··= min { p1 , . . . , pl } the minimal width of the networks . The functions fj : Rpj → Rpj are called the activation functions . We assume that the activation functions are elementwise functions in the sense that fj [ b ] = ( fj [ b1 ] , . . . , fj [ bpj ] ) > for all b ∈ Rp j , where fj : R→ R is an arbitrary function . This allows for an unlimited variety in the type of activation , including ReLU fj : b 7→ max { 0 , b } , leaky ReLU fj : b 7→ max { 0 , b } + min { 0 , cb } for a fixed c ∈ ( 0 , 1 ) , polynomial fj : b 7→ cbk for fixed c ∈ ( 0 , ∞ ) and k ∈ [ 1 , ∞ ) , and sigmoid activation fj : b 7→ 1/ ( 1 + e−b ) as popular examples , and it allows for different activation functions in each layer . We study the most common approaches to parameter estimation in this setting : constraint and unconstraint empirical-risk minimization . The loss function l : Rm × Rm → R is assumed convex in its first argument ; this includes all standard loss functions , such as the least-squares loss l : ( a , b ) 7→ ||a − b||22 , the absolut-deviation loss l : ( a , b ) 7→ ||a − b||1 ( both typically used for regression ) , the logistic loss l : ( a , b ) 7→ − ( 1 + b ) log [ 1 + a ] − ( 1 − b ) log [ 1 − a ] , the hinge loss l : ( a , b ) 7→ max { 0 , 1− ab } ( both typically used for binary classification Dy = { ±1 } ) , and so forth . The optimization domain is the set M ··= { Θ ∈M : r [ Θ ] ≤ 1 } for a constraint r : M → R. Given data ( x1 , y1 ) , . . . , ( xn , yn ) ∈ Dx × Dy , the empirical-risk minimizers are then the networks gΘ̂erm with Θ̂erm ∈ arg min Θ∈M { n∑ i=1 l [ gΘ [ xi ] , yi ] } . ( 2 ) It has been shown that constraints can facilitate the optimization as well as improve generalization— see Krizhevsky et al . ( 2012 ) and Livni et al . ( 2014 ) , among others . For ease of presentation , we limit ourselves to the following class of constraints : r [ Θ ] ··= max { ar max j∈ { 1 , ... , l } |||Θj |||1 , br|||Θ0|||q } for all Θ ∈M ( 3 ) for fixed tuning parameters ar , br ∈ [ 0 , ∞ ) , a parameter q ∈ ( 0 , ∞ ] , and ||| · |||q the usual row-wise ` q “ -norm , ” that is , |||Θj |||q ··= maxk ( ∑ i| ( Θj ) ki|q ) 1/q for q ∈ ( 0 , ∞ ) and |||Θj |||∞ ··= maxki| ( Θj ) ki| . This class of constraints includes the following four important cases : • Unconstraint estimation : ar = br = 0 . In other words , M =M . Unconstraint estimation had been the predominant approach in the earlier days of deep learning and is still used today ( Anthony & Bartlett , 1999 ) . • Connection sparsity : q = 1 . This constraint yields connection-sparse networks , which have received considerable attention recently ( Barron & Klusowski , 2018 ; 2019 ; Kim et al. , 2016 ; Taheri et al. , 2020 ) . • Strong sparsity : q < 1 . Nonconvex constraints have been popular in statistics for many years ( Fan & Li , 2001 ; Zhang , 2010 ) , but our paper is probably the first one that includes such constraints in a theoretical analysis in deep learning . • Input constraints : ar = 0 . Some researchers have argued for applying certain constraints , such as node-sparsity , only to the input level ( Feng & Simon , 2017 ) . In general , while our proof techniques also apply to many other types of constraints , there are two main reasons for using the mentioned sparsity-inducing constraints to illustrate our results : First , sparsity has become very popular in deep learning , because it can lower the burden on memory and optimization as well as increase interpretability ( Hebiri & Lederer , 2020 ) . And second , the above examples allow us to demonstrate that the discussed features of wide networks do not depend on smooth and convex constraints such as weight decay . Our theory can also be adjusted to the regularized versions of the empirical-risk minimizers , that is , for the networks indexed by any parameter in the set arg min Θ∈M { n∑ i=1 l [ gΘ [ xi ] , yi ] + r [ Θ ] } . The proofs are virtually the same as for the constraint versions ; we omit the details for the sake of brevity . One line of research develops statistical theories for constraint and unconstraint empirical-risk minimizers—see Bartlett & Mendelson ( 2002 ) and Lederer ( 2020 ) , among others . As detailed above , empirical-risk minimizers are the networks whose parameters are global minima of the objective function Θ 7→ l [ gΘ ] ··= n∑ i=1 l [ gΘ [ xi ] , yi ] ( 4 ) over M for fixed data ( x1 , y1 ) , . . . , ( xn , yn ) . While the function gΘ 7→ l [ gΘ ] is convex by assumption , the objective function Θ 7→ l [ gΘ ] is usually nonconvex . It is thus unclear , per se , whether deep-learning pipelines can be expected to yield global minima of the objective function and , therefore , whether the statistical theories are valid in practice . Our goal is , broadly speaking , to establish conditions under which global minimization of ( 4 ) can indeed be expected .

Summary: This work showed that for a wide deep neural network, the optimization landscapes of empirical-risk minimizers over wide feedforward networks have no spurious local minima. The theory combines the features of the two mentioned works, as it applies to the entire optimization landscapes, allows for a wide spectrum of loss functions and activation functions, and constraint and unconstraint estimation. The proof defines two important concepts: path relation and block parameters and utilizes these two concepts and propositions and lemmas developed upon them to prove the main result.