A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks

1 INTRODUCTION . Deep learning builds upon the mysterious ability of gradient-based optimization methods to solve related non-convex problems . Immense efforts are underway to mathematically analyze this phenomenon . The prominent landscape approach focuses on special properties of critical points ( i.e . points where the gradient of the objective function vanishes ) that will imply convergence to global optimum . Several papers ( e.g . Ge et al . ( 2015 ) ; Lee et al . ( 2016 ) ) have shown that ( given certain smoothness properties ) it suffices for critical points to meet the following two conditions : ( i ) no poor local minima — every local minimum is close in its objective value to a global minimum ; and ( ii ) strict saddle property — every critical point that is not a local minimum has at least one negative eigenvalue to its Hessian . While condition ( i ) does not always hold ( cf . Safran and Shamir ( 2018 ) ) , it has been established for various simple settings ( e.g . Soudry and Carmon ( 2016 ) ; Kawaguchi ( 2016 ) ) . Condition ( ii ) on the other hand seems less plausible , and is in fact provably false for models with three or more layers ( cf . Kawaguchi ( 2016 ) ) , i.e . for deep networks . It has only been established for problems involving shallow ( two layer ) models , e.g . matrix factorization ( Ge et al . ( 2016 ) ; Du et al . ( 2018a ) ) . The landscape approach as currently construed thus suffers from inherent limitations in proving convergence to global minimum for deep networks . A potential path to circumvent this obstacle lies in realizing that landscape properties matter only in the vicinity of trajectories that can be taken by the optimizer , which may be a negligible portion of the overall parameter space . Several papers ( e.g . Saxe et al . ( 2014 ) ; Arora et al . ( 2018 ) ) have taken this trajectory-based approach , primarily in the context of linear neural networks — fully-connected neural networks with linear activation . Linear networks are trivial from a representational perspective , but not so in terms of optimization — they lead to non-convex training problems with multiple minima and saddle points . Through a mix of theory and experiments , Arora et al . ( 2018 ) argued that such non-convexities may in fact be beneficial for gradient descent , in the sense that sometimes , adding ( redundant ) linear layers to a classic linear prediction model can accelerate the optimization . This phenomenon challenges the holistic landscape view , by which convex problems are always preferable to non-convex ones . Even in the linear network setting , a rigorous proof of efficient convergence to global minimum has proved elusive . One recent progress is the analysis of Bartlett et al . ( 2018 ) for linear residual networks — a particular subclass of linear neural networks in which the input , output and all hidden dimensions are equal , and all layers are initialized to be the identity matrix ( cf . Hardt and Ma ( 2016 ) ) . Through a trajectory-based analysis of gradient descent minimizing ` 2 loss over a whitened dataset ( see Section 2 ) , Bartlett et al . ( 2018 ) show that convergence to global minimum at a linear rate — loss is less than > 0 after O ( log 1 ) iterations — takes place if one of the following holds : ( i ) the objective value at initialization is sufficiently close to a global minimum ; or ( ii ) a global minimum is attained when the product of all layers is positive definite . The current paper carries out a trajectory-based analysis of gradient descent for general deep linear neural networks , covering the residual setting of Bartlett et al . ( 2018 ) , as well as many more settings that better match practical deep learning . Our analysis draws upon the trajectory characterization of Arora et al . ( 2018 ) for gradient flow ( infinitesimally small learning rate ) , together with significant new ideas necessitated due to discrete updates . Ultimately , we show that when minimizing ` 2 loss of a deep linear network over a whitened dataset , gradient descent converges to the global minimum , at a linear rate , provided that the following conditions hold : ( i ) the dimensions of hidden layers are greater than or equal to the minimum between those of the input and output ; ( ii ) layers are initialized to be approximately balanced ( see Definition 1 ) — this is met under commonplace near-zero , as well as residual ( identity ) initializations ; and ( iii ) the initial loss is smaller than any loss obtainable with rank deficiencies — this condition will hold with probability close to 0.5 if the output dimension is 1 ( scalar regression ) and standard ( random ) near-zero initialization is employed . Our result applies to networks with arbitrary depth and input/output dimensions , as well as any configuration of hidden layer widths that does not force rank deficiency ( i.e . that meets condition ( i ) ) . The assumptions on initialization ( conditions ( ii ) and ( iii ) ) are necessary , in the sense that violating any one of them may lead to convergence failure . Moreover , in the case of scalar regression , they are met with constant probability under a random initialization scheme . We are not aware of any similarly general analysis for efficient convergence of gradient descent to global minimum in deep learning . The remainder of the paper is organized as follows . In Section 2 we present the problem of gradient descent training a deep linear neural network by minimizing the ` 2 loss over a whitened dataset . Section 3 formally states our assumptions , and presents our convergence analysis . Key ideas brought forth by our analysis are demonstrated empirically in Section 4 . Section 5 gives a review of relevant literature , including a detailed comparison of our results against those of Bartlett et al . ( 2018 ) . Finally , Section 6 concludes . 2 GRADIENT DESCENT FOR DEEP LINEAR NEURAL NETWORKS . We denote by ‖v‖ the Euclidean norm of a vector v , and by ‖A‖F the Frobenius norm of a matrixA . We are given a training set { ( x ( i ) , y ( i ) ) } mi=1 ⊂ Rdx × Rdy , and would like to learn a hypothesis ( predictor ) from a parametric familyH : = { hθ : Rdx → Rdy | θ ∈ Θ } by minimizing the ` 2 loss:1 min θ∈Θ L ( θ ) : = 1 2m ∑m i=1 ‖hθ ( x ( i ) ) − y ( i ) ‖2 . When the parametric family in question is the class of linear predictors , i.e . H = { x 7→ Wx |W ∈ Rdy×dx } , the training loss may be written as L ( W ) = 12m‖WX − Y ‖ 2 F , where X ∈ Rdx×m and Y ∈ Rdy×m are matrices whose columns hold instances and labels respectively . Suppose now that the dataset is whitened , i.e . has been transformed such that the empirical ( uncentered ) covariance matrix for instances — Λxx : = 1mXX > ∈ Rdx×dx — is equal to identity . Standard calculations ( see Appendix A ) show that in this case : L ( W ) = 1 2 ‖W − Λyx‖2F + c , ( 1 ) 1Much of the analysis in this paper can be extended to loss types other than ` 2 . In particular , the notion of deficiency margin ( Definition 2 ) can be generalized to account for any convex loss , and , so long as the loss is differentiable , a convergence result analogous to Theorem 1 will hold in the idealized setting of perfect initial balancedness and infinitesimally small learning rate ( see proof of Lemma 1 ) . We leave to future work treatment of approximate balancedness and discrete updates in this general setting . where Λyx : = 1mY X > ∈ Rdy×dx is the empirical ( uncentered ) cross-covariance matrix between instances and labels , and c is a constant ( that does not depend onW ) . Denoting Φ : = Λyx for brevity , we have that for linear models , minimizing ` 2 loss over whitened data is equivalent to minimizing the squared Frobenius distance from a target matrix Φ : minW∈Rdy×dx L 1 ( W ) : = 1 2 ‖W − Φ‖2F . ( 2 ) Our interest in this work lies on linear neural networks — fully-connected neural networks with linear activation . A depth-N ( N ∈ N ) linear neural network with hidden widths d1 , . . . , dN−1 ∈ N corresponds to the parametric family of hypotheses H : = { x 7→ WNWN−1 · · ·W1x |Wj ∈ Rdj×dj−1 , j = 1 , . . . , N } , where d0 : = dx , dN : = dy . Similarly to the case of a ( directly parameterized ) linear predictor ( Equation ( 2 ) ) , with a linear neural network , minimizing ` 2 loss over whitened data can be cast as squared Frobenius approximation of a target matrix Φ : minWj∈Rdj×dj−1 , j=1 , ... , N L N ( W1 , . . . , WN ) : = 1 2 ‖WNWN−1 · · ·W1 − Φ‖2F . ( 3 ) Note that the notation LN ( · ) is consistent with that of Equation ( 2 ) , as a network with depth N = 1 precisely reduces to a ( directly parameterized ) linear model . We focus on studying the process of training a deep linear neural network by gradient descent , i.e . of tackling the optimization problem in Equation ( 3 ) by iteratively applying the following updates : Wj ( t+ 1 ) ←Wj ( t ) − η ∂LN ∂Wj ( W1 ( t ) , . . . , WN ( t ) ) , j = 1 , . . . , N , t = 0 , 1 , 2 , . . . , ( 4 ) where η > 0 is a configurable learning rate . In the case of depth N = 1 , the training problem in Equation ( 3 ) is smooth and strongly convex , thus it is known ( cf . Boyd and Vandenberghe ( 2004 ) ) that with proper choice of η , gradient descent converges to global minimum at a linear rate . In contrast , for any depth greater than 1 , Equation ( 3 ) comprises a fundamentally non-convex program , and the convergence properties of gradient descent are highly non-trivial . Apart from the caseN = 2 ( shallow network ) , one can not hope to prove convergence via landscape arguments , as the strict saddle property is provably violated ( see Section 1 ) . We will see in Section 3 that a direct analysis of the trajectories taken by gradient descent can succeed in this arena , providing a guarantee for linear rate convergence to global minimum . We close this section by introducing additional notation that will be used in our analysis . For an arbitrary matrix A , we denote by σmax ( A ) and σmin ( A ) its largest and smallest ( respectively ) singular values.2 For d ∈ N , we use Id to signify the identity matrix in Rd×d . Given weights W1 , . . . , WN of a linear neural network , we let W1 : N be the direct parameterization of the end-to-end linear mapping realized by the network , i.e . W1 : N : = WNWN−1 · · ·W1 . Note that LN ( W1 , . . . , WN ) = L 1 ( W1 : N ) , meaning the loss associated with a depth-N network is equal to the loss of the corresponding end-to-end linear model . In the context of gradient descent , we will oftentimes use ` ( t ) as shorthand for the loss at iteration t : ` ( t ) : = LN ( W1 ( t ) , . . . , WN ( t ) ) = L 1 ( W1 : N ( t ) ) . ( 5 )

Summary: This paper continues the recent line of study on the convergence behaviour of gradient descent for deep linear neural networks. For more than 2 layers, the optimization problem is nonconvex and it is known strict saddle points exist. The main contribution is a relaxation of the balancedness condition in previous work by Arora et al and a new deficiency margin condition, which together allowed the authors to prove that gradient descent will converge to an epsilon solution in at most O(log 1/epsilon) iterations (under reasonable assumptions on step size and other parameters). Examples on how to satisfy the two conditions are discussed. Overall, the obtained results appear to be a solid contribution beyond our current understanding of deep linear neural networks, and potentially may be helpful towards our understanding of deep nonlinear neural networks.