Depth creates no more spurious local minima in linear networks

1 INTRODUCTION . One major mystery in deep learning is that deep neural networks do not seem to suffer from spurious local minima . Understanding this mystery has become one of the most important topics in the machine learning theory . This problem turns to be quite challenging . Even for the subcase of the deep linear network , much still remains unknown despite a long line of studies on this subject . In this paper , we prove a useful structural property about the deep linear networks . We show that for any convex differentiable loss function , any spurious local minima in a deep linear network should already be present in a two layer linear network . Hence , depth does not create more spurious local minima in linear networks . This reduction greatly simplifies the study about the existence of spurious local minima in deep linear networks . When applied to the quadratic loss , it leads to the first unconditional proof that there is no local minima in deep linear networks . Baldi & Hornik ( 1989 ) started the investigation on the existence of spurious local minima in linear networks . They showed that , under mild assumptions , for quadratic loss , two layer linear networks do not have spurious local minima . They also conjectured it is true for deep linear networks . This conjecture is only proved recently by Kawaguchi ( 2016 ) . For the special case of linear residual networks , Hardt & Ma ( 2017 ) showed that there are no spurious local minima through a simpler argument . While most existing work have been on quadratic loss functions , Laurent & von Brecht ( 2018 ) showed a surprisingly general result for any convex differentiable loss . In ( Laurent & von Brecht , 2018 ) , the authors consider the special linear networks which have no bottlenecks , i.e . when the narrowest layer is on the either end . They showed that for any convex differential loss function , there is no spurious local minima in such networks . In addition to its generality , the proof in ( Laurent & von Brecht , 2018 ) is quite intuitive through a novel perturbation argument . However , their special cases excludes networks with bottleneck layers , commonly used in the practice and studied in the literature ( Baldi & Hornik , 1989 ; Kawaguchi , 2016 ) . We build on the work in ( Laurent & von Brecht , 2018 ) and further develop the technique to show that for general deep linear networks , whether there are spurious local minima is reduced to the two layer case . Theorem 1 . Given any convex differentiable function f : Rm×n → R. For any k ≥ 2 , let Lk ( M1 , . . . , Mk ) = f ( Mk · · ·M1 ) where Mi ∈ Rdi×di−1 for 1 ≤ i ≤ k with dk = m and d0 = n. Let d = min0≤i≤k di . Define L2 ( A , B ) = f ( AB ) for A ∈ Rm×d , B ∈ Rd×n . Then Lk has no spurious local minima iff L2 has no spurious local minima . We emphasize that in the above theorem , f depends on both the data and the loss function . Hence the reduction is instance specific and does not depend on the global property of a family of loss functions . To prove Theorem 1 , we show a non-degeneracy property of local minima , namely , any spurious local minimum must have rank same as the width of the bottleneck . This property can be useful for the two layer case too – it implies that each layer in a spurious local minimum can not be rankdeficient , hence covering subcases which may otherwise require onerous analysis ( Baldi & Hornik , 1989 ; Zhou & Liang , 2018 ) . Theorem 2 . With the same notation as in Theorem 1 , if A , B is a spurious local minimum of L2 , then both A , B must have full rank . As an application of Theorem 1 , for quadratic loss , Theorem 1 , together with ( Baldi & Hornik , 1989 ) , immediately implies the main result in ( Kawaguchi , 2016 ) . We can further remove all the assumptions needed in ( Baldi & Hornik , 1989 ; Kawaguchi , 2016 ) , using the recent result by Zhou & Liang ( 2018 ) ( Theorem 2 ( 1 ) ) , hence providing the first unconditional proof of the non-existence of spurious local minima in deep linear networks for quadratic loss functions . Below ‖ · ‖ denotes the Frobenius norm . Corollary 1 . For anyX ∈ Rd0×n , Y ∈ Rdk×n , let L ( M1 , . . . , Mk ) = ‖Mk · · ·M1X−Y ‖2 . Then L has no spurious local minima . Theorem 1 can be further generalized to “ multi-tower ” linear networks . Define a multi-tower linear network as the sum of multiple deep linear networks ( towers ) , i.e . M1k1 · · ·M11 + . . . + Msks · · ·Ms1 ) , where Mij ∈ Rdi , j×di , j−1 with di , ki = m and di,0 = n. For 1 ≤ i ≤ s , let bi = minj di , j denote the bottleneck size of each tower i . Write b = ∑ i bi . Corollary 2 . For any differentiable convex loss f , a multi-tower linear network has no spurious local minima iff the linear network AB , where A ∈ Rm×b , B ∈ Rb×n , has no spurious local minima . Moreover , if b ≥ m , n , there is no spurious local minima in this network . There are two main technical ideas in our proof . First , we show a reduction of any local minimum of the multi-layer network to a critical point of a two-layer network . Secondly , we generalize the local perturbation argument in ( Laurent & von Brecht , 2018 ) to weaker critical conditions . Besides the conceptual importance of our results , these technical contributions might be useful for understanding deep networks too . 1.1 RELATED WORK . In Baldi & Hornik ( 1989 ) , it is shown that , under mild assumptions on data , two layer linear network with quadratic loss has no spurious local minima . This is probably the first positive result on this long line of investigation . Kawaguchi ( 2016 ) showed that it holds for deep linear network too . The tour de force proof in ( Kawaguchi , 2016 ) works by examining the Hessian using powerful tools from the matrix theory . There have been much subsequent work to simplify and generalize the result . For example , Lu & Kawaguchi ( 2017 ) came up with a different argument . Yun et al . ( 2018 ; 2019 ) showed simpler arguments for special cases and considered more general non-linear networks . Hardt & Ma ( 2017 ) showed that under certain assumptions , there might not even be stationary point in the deep linear residual network . Venturi et al . ( 2018 ) defined a notion of spurious valleys and showed that for quadratic losses , there is no spurious valley in deep linear networks . Venturi et al . ( 2018 ) was able to remove all the assumption in ( Baldi & Hornik , 1989 ) under this weaker notion . The mild assumption in Baldi & Hornik ( 1989 ) , which was also needed in Kawaguchi ’ s proof , was removed by Zhou & Liang ( 2018 ) , which leads to our Corollary 1 . Laurent & von Brecht ( 2018 ) considers the special case of linear networks with the narrowest layer on the either end . It uses a novel perturbation argument to show that for any convex differentiable loss , there is no spurious local minimum in such network . However , the special case considered in ( Laurent & von Brecht , 2018 ) excludes networks through low rank approximation such as autoencoders . But it is really the intuitive yet powerful result in ( Laurent & von Brecht , 2018 ) that motivated this work . There have been recent studies on the gradient descent convergence on the deep linear networks ( Arora et al. , 2018 ; Bartlett et al. , 2018 ; Arora et al. , 2019 ) . It has been shown by Arora et al . ( 2018 ) that , under certain conditions , increasing the depth of linear networks can speed up the convergence , which is another positive property of deep linear networks . There have been much work ( Soltanolkotabi et al. , 2018 ; Li & Liang , 2018 ; Allen-Zhu et al. , 2018a ; b ; Du et al. , 2018a ; Zou et al. , 2018 ; Du et al. , 2018b ) recently on studying the optimization landscape and convergence of non-linear networks . They focus mostly on shallow networks with over-parameterized wide layers . 2 PRELIMINARIES . We define notations used through the paper . We state some simple facts and the main theorem from Laurent & von Brecht ( 2018 ) which we need in our proof . Deep linear networks . Denote by Rm×n all the matrices with m rows and n columns . For 1 ≤ i ≤ k , letMi ∈ Rdi×di−1 . For i ≥ j , denote byMi · · ·Mj the matrix product ofMi ·Mi−1 . . . ·Mj . A ( deep ) linear network with parameters M1 , · · · , Mk is defined as Φ ( x ) = Mk · · ·M1x . We call k the depth of the network and d0 , dk the input and the output dimensions , respectively . Define d = min0≤i≤k di be the narrowest width . We say a network has a bottleneck if both d0 > d and dk > d. A multi-tower linear network is defined as the sum of multiple linear networks ( towers ) with the same input and output dimensions . Empirical loss . Given training data D which consist of examples of pairs of x , y where the input feature vector x ∈ Rd0 , and the label y in some arbitrary set , we wish to minimize the total loss : L ( M1 , · · · , Mk ; D ) = ∑ ( x , y ) ∈D fy ( Φ ( x ) ) = ∑ ( x , y ) ∈D fy ( Mk · · ·M1x ) . Define f ( A ) = ∑ ( x , y ) ∈D fy ( Ax ) . Then L ( M1 , · · · , Mk ; D ) = f ( Mk · · ·M1 ) . If fy ’ s are all convex differentiable functions1 , then clearly f is convex differentiable too . Below we omit D and consider the loss function L : Rd1×d0 × . . . × Rdk×dk−1 → R where L ( M1 , · · · , Mk ) = f ( Mk · · ·M1 ) for some f : Rdk×d0 → R. Derivative . Denote by ∂L∂M the matrix form of the partial derivative of L with respect to M . If L has only one variable M , we write L′ = ∂L∂M . For simplicity , we sometimes abuse the notation by using the same symbol for the variable and the value and omit the value . If L ( X , Y ) = f ( XY ) , by the chain rule , ∂L∂X = f ′ ( XY ) Y T and ∂L∂Y = X T f ′ ( XY ) . Local minimum . For any loss function L , M1 , . . . , Mk is a local minimum of L if there exists an open ball B , in Frobenius norm , centered at M1 , . . . , Mk such that L ( M1 , . . . , Mk ) ≤ L ( M ′1 , · · · , M ′k ) for any ( M ′1 , · · · , M ′k ) ∈ B . A local minimum is called spurious if it is not a global minimum . If L is differentiable , then any local minimum is a critical point of L. In particular , if L ( M1 , · · · , Mk ) = f ( Mk · · ·M1 ) , then M1 , · · · , Mk satisfy that ∂L∂Mi = ( Mk · · ·Mi+1 ) T f ′ ( Mk · · ·M1 ) ( Mi−1 · · ·M1 ) T = 0 We need the following theorem from Laurent & von Brecht ( 2018 ) : Theorem 3 . Let Lk ( M1 , . . . , Mk ) = f ( Mk · · ·M1 ) where f : Rdk×d0 → R is a convex differentiable function . If there is no bottleneck , i.e . d0 or dk = min0≤i≤k di , then any local minimum of Lk is a global minimum of f .

Summary: The motivation of this paper is training deep neural network seems to not suffer from local minima, and it tries to explain this phenomenon by showing that all local minima of deep neural network is global minima. The paper shows that for any convex differentiable loss function, a deep linear neural network has no so called spurious local minima, which to be specific, are local minima that are not global minima, as long as it is true for two-layer Neural Network. The motivation is that combining with existing result that no spurious local minima exists for quadratic loss in two-layer Neural Network, this relation connecting between two-layer and deeper linear neural network immediately implies an existing result that all local minima are global minima, removing all assumptions. The result also holds for general “multi-tower” linear networks.