Gradient descent aligns the layers of deep linear networks

1 INTRODUCTION . Efforts to explain the effectiveness of gradient descent in deep learning have uncovered an exciting possibility : it not only finds solutions with low error , but also biases the search for low complexity solutions which generalize well ( Zhang et al. , 2017 ; Bartlett et al. , 2017 ; Soudry et al. , 2017 ; Gunasekar et al. , 2018 ) . This paper analyzes the implicit regularization of gradient descent and gradient flow on deep linear networks and linearly separable data . For strictly decreasing losses , the optimum is at infinity , and we establish various alignment phenomena : • For each weight matrix Wi , the corresponding normalized weight matrix Wi/‖Wi‖F asymptotically equals its rank-1 approximation uiv > i , where the Frobenius norm ‖Wi‖F satisfies ‖Wi‖F → ∞ . In other words , ‖Wi‖2/‖Wi‖F → 1 , and asymptotically only the rank-1 approximation of Wi contributes to the final predictor , a form of implicit regularization . • Adjacent rank-1 weight matrix approximations are aligned : |v > i+1ui| → 1 . • For the logistic loss , the first right singular vector v1 ofW1 is aligned with the data , meaning v1 converges to the unique maximum margin predictor ū defined by the data . Moreover , the linear predictor induced by the network , wprod : = WL · · ·W1 , is also aligned with the data , meaning wprod/‖wprod‖→ ū . Simultaneously , this work proves that the risk is globally optimized : it asymptotes to 0 . Alignment and risk convergence are proved simultaneously ; the phenomena are coupled within the proofs . Since the layers align , they can be viewed as a minimum norm solution : they do not “ waste norm ” on components which are killed off when the layers are multiplied together . Said another way , given data ( ( xi , yi ) ) ni=1 , the normalized matrices ( W1/‖W1‖F , . . . , WL/‖WL‖F ) asymptotically solve a maximum margin problem which demands all weight matrices be small , not merely their product : max WL∈R1×dL−1 ‖WL‖F =1 · · · max W1∈Rd1×d0 ‖W1‖F =1 min i yi ( WL · · ·W1 ) xi . The paper is organized as follows . This introduction continues with related work , notation , and assumptions in Sections 1.1 and 1.2 . The analysis of gradient flow is in Section 2 , and gradient descent is analyzed in Section 3 . The paper closes with future directions in Section 4 ; a particular highlight is a preliminary experiment on CIFAR-10 which establishes empirically that a form of the alignment phenomenon occurs on the standard nonlinear network AlexNet . 1.1 RELATED WORK . On the implicit regularization of gradient descent , Soudry et al . ( 2017 ) show that for linear predictors and linearly separable data , the gradient descent iterates converge to the same direction as the maximum margin solution . Ji & Telgarsky ( 2018 ) further characterize such an implicit bias for general nonseparable data . Gunasekar et al . ( 2018 ) consider gradient descent on fully connected linear networks and linear convolutional networks . In particular , for the exponential loss , assuming the risk is minimized to 0 and the gradients converge in direction , they show that the whole network converges in direction to the maximum margin solution . These two assumptions are on the gradient descent process itself , and specifically the second one might be hard to interpret and justify . Compared with Gunasekar et al . ( 2018 ) , this paper proves that the risk converges to 0 and the weight matrices align ; moreover the proof here proves the properties simultaneously , rather than assuming one and deriving the other . Lastly , Arora et al . ( 2018 ) show for deep linear networks ( and later Du et al . ( 2018 ) for ReLU networks ) that gradient flow does not change the difference between squared Frobenius norms of any two layers . We use a few of these tools in our proofs ; please see Sections 2 and 3 for details . For a smooth ( nonconvex ) function , Lee et al . ( 2016 ) show that any strict saddle can be avoided almost surely with small step sizes . If there are only countably many saddle points and they are all strict , and if gradient descent iterates converge , then this implies ( almost surely ) they converge to a local minimum . In the present work , since there is no finite local minimum , gradient descent will go to infinity and never converge , and thus these results of Lee et al . ( 2016 ) do not show that the risk converges to 0 . There has been a rich literature on linear networks . Saxe et al . ( 2013 ) analyze the learning dynamics of deep linear networks , showing that they exhibit some learning patterns similar to nonlinear networks , such as a long plateau followed by a rapid risk drop . Arora et al . ( 2018 ) show that depth can help accelerate optimization . On the landscape properties of deep linear networks , Lu & Kawaguchi ( 2017 ) ; Laurent & von Brecht ( 2017 ) show that under various structural assumptions , all local optima are global . Zhou & Liang ( 2018 ) give a necessary and sufficient characterization of critical points for deep linear networks . 1.2 NOTATION , SETTING , AND ASSUMPTIONS . Consider a data set { ( xi , yi ) } ni=1 , where xi ∈ Rd , ‖xi‖ ≤ 1 , and yi ∈ { −1 , +1 } . The data set is assumed to be linearly separable , i.e. , there exists a unit vector uwhich correctly classifies every data point : for any 1 ≤ i ≤ n , yi〈u , xi〉 > 0 . Furthermore , let γ : = max‖u‖=1 min1≤i≤n yi〈u , xi〉 > 0 denote the maximum margin , and ū : = arg max‖u‖=1 min1≤i≤n yi〈u , xi〉 denote the maximum margin solution ( the solution to the hard-margin SVM ) . A linear network of depth L is parameterized by weight matrices WL , . . . , W1 , where Wk ∈ Rdk×dk−1 , d0 = d , and dL = 1 . Let W = ( WL , . . . , W1 ) denote all parameters of the network . The ( empirical ) risk induced by the network is given by R ( W ) = R ( WL , . . . , W1 ) = 1 n n∑ i=1 ` ( yiWL · · ·W1xi ) = 1 n n∑ i=1 ` ( 〈wprod , zi〉 ) , where wprod : = ( WL · · ·W1 ) > , and zi : = yixi . The loss ` is assumed to be continuously differentiable , unbounded , and strictly decreasing to 0 . Examples include the exponential loss ` exp ( x ) = e−x and the logistic loss ` log ( x ) = ln ( 1 + e−x ) . Assumption 1 . ` ′ < 0 is continuous , limx→−∞ ` ( x ) =∞ and limx→∞ ` ( x ) = 0 . This paper considers gradient flow and gradient descent , where gradient flow { W ( t ) ∣∣t ≥ 0 , t ∈ R } can be interpreted as gradient descent with infinitesimal step sizes . It starts from some W ( 0 ) at t = 0 , and proceeds as dW ( t ) dt = −∇R ( W ( t ) ) . By contrast , gradient descent { W ( t ) ∣∣t ≥ 0 , t ∈ Z } is a discrete-time process given by W ( t+ 1 ) = W ( t ) − ηt∇R ( W ( t ) ) , where ηt is the step size at time t. We assume that the initialization of the network is not a critical point and induces a risk no larger than the risk of the trivial linear predictor 0 . Assumption 2 . The initialization W ( 0 ) satisfies∇R ( W ( 0 ) ) 6= 0 andR ( W ( 0 ) ) ≤ R ( 0 ) = ` ( 0 ) . It is natural to require that the initialization is not a critical point , since otherwise gradient flow/descent will never make a progress . The requirement R ( W ( 0 ) ) ≤ R ( 0 ) can be easily satisfied , for example , by making W1 ( 0 ) = 0 and WL ( 0 ) · · ·W2 ( 0 ) 6= 0 . On the other hand , if R ( W ( 0 ) ) > R ( 0 ) , gradient flow/descent may never minimize the risk to 0 . Proofs of those claims are given in Appendix A . 2 RESULTS FOR GRADIENT FLOW . In this section , we consider gradient flow . Although impractical when compared with gradient descent , gradient flow can simplify the analysis and highlight proof ideas . For convenience , we usually useW , Wk , andwprod , but they all change with ( the continuous time ) t. Only proof sketches are given here ; detailed proofs are deferred to Appendix B . 2.1 RISK CONVERGENCE . One key property of gradient flow is that it never increases the risk : dR ( W ) dt = 〈 ∇R ( W ) , dW dt 〉 = −‖∇R ( W ) ‖2 = − L∑ k=1 ∥∥∥∥ ∂R∂Wk ∥∥∥∥2 F ≤ 0 . ( 1 ) We now state the main result : under Assumptions 1 and 2 , gradient flow minimizes the risk , Wk and wprod all go to infinity , and the alignment phenomenon occurs . Theorem 1 . Under Assumptions 1 and 2 , gradient flow iterates satisfy the following properties : • limt→∞R ( W ) = 0 . • For any 1 ≤ k ≤ L , limt→∞ ‖Wk‖F =∞ . • For any 1 ≤ k ≤ L , letting ( uk , vk ) denote the first left and right singular vectors of Wk , lim t→∞ ∥∥∥∥ Wk‖Wk‖F − ukv > k ∥∥∥∥ F = 0 . Moreover , for any 1 ≤ k < L , limt→∞ ∣∣〈vk+1 , uk〉∣∣ = 1 . As a result , lim t→∞ ∣∣∣∣∣∣ 〈 wprod∏L k=1 ‖Wk‖F , v1 〉∣∣∣∣∣∣ = 1 , and thus limt→∞ ‖wprod‖ =∞ . Theorem 1 is proved using two lemmas , which may be of independent interest . To show the ideas , let us first introduce a little more notation . Recall that R ( W ) denotes the empirical risk induced by the deep linear network W . Abusing the notation a little , for any linear predictor w ∈ Rd , we also useR ( w ) to denote the risk induced by w. With this notation , R ( W ) = R ( wprod ) , while ∇R ( wprod ) = 1 n n∑ i=1 ` ′ ( 〈wprod , zi〉 ) zi = 1 n n∑ i=1 ` ′ ( WL · · ·W1zi ) zi is in Rd and different from∇R ( W ) , which has ∑L k=1 dkdk−1 entries , as given below : ∂R ∂Wk = W > k+1 · · ·W > L ∇R ( wprod ) > W > 1 · · ·W > k−1 . Furthermore , for any R > 0 , let B ( R ) = { W ∣∣∣∣ max1≤k≤L ‖Wk‖F ≤ R } . The first lemma shows that for any R > 0 , the time spent by gradient flow in B ( R ) is finite . Lemma 1 . Under Assumption 1 and 2 , for any R > 0 , there exists a constant ( R ) > 0 , such that for any t ≥ 1 and any W ∈ B ( R ) , ‖∂R/∂W1‖F ≥ ( R ) . As a result , gradient flow spends a finite amount of time in B ( R ) for any R > 0 , and max1≤k≤L ‖Wk‖F is unbounded . Here is the proof sketch . If ‖Wk‖F are bounded , then ‖∇R ( wprod ) ‖ will be lower bounded by a positive constant , therefore if ‖∂R/∂W1‖F = ‖WL · · ·W2‖‖∇R ( wprod ) ‖ can be arbitrarily small , then ‖WL · · ·W2‖ and ‖wprod‖ can also be arbitrarily small , and thusR ( W ) can be arbitrarily close toR ( 0 ) . This can not happen after t = 1 , otherwise it will contradict Assumption 2 and eq . ( 1 ) . To proceed , we need the following properties of linear networks from prior work ( Arora et al. , 2018 ; Du et al. , 2018 ) . For any time t ≥ 0 and any 1 ≤ k < L , W > k+1 ( t ) Wk+1 ( t ) −W > k+1 ( 0 ) Wk+1 ( 0 ) = Wk ( t ) W > k ( t ) −Wk ( 0 ) W > k ( 0 ) . ( 2 ) To see this , just notice that W > k+1 ∂R ∂Wk+1 = W > k+1 · · ·W > L ∇R ( wprod ) > W > 1 · · ·W > k = ∂R ∂Wk W > k . Taking the trace on both sides of eq . ( 2 ) , we have∥∥Wk+1 ( t ) ∥∥2F −∥∥Wk+1 ( 0 ) ∥∥2F =∥∥Wk ( t ) ∥∥2F −∥∥Wk ( 0 ) ∥∥2F . ( 3 ) In other words , the difference between the squares of Frobenius norms of any two layers remains a constant . Together with Lemma 1 , it implies that all ‖Wk‖F are unbounded . However , even if ‖Wk‖F are large , it does not follow necessarily that ‖wprod‖ is also large . Lemma 2 shows that this is indeed true : for gradient flow , as ‖Wk‖F get larger , adjacent layers also get more aligned to each other , which ensures that their product also has a large norm . For 1 ≤ k ≤ L , let σk , uk , and vk denote the first singular value ( the 2-norm ) , the first left singular vector , and the first right singular vector of Wk , respectively . Furthermore , define D : = ( max 1≤k≤L ‖Wk ( 0 ) ‖2F ) − ‖WL ( 0 ) ‖2F + L−1∑ k=1 ∥∥∥Wk ( 0 ) W > k ( 0 ) −W > k+1 ( 0 ) Wk+1 ( 0 ) ∥∥∥ 2 , which depends only on the initialization . If for any 1 ≤ k < L , Wk ( 0 ) W > k ( 0 ) = W > k+1 ( 0 ) Wk+1 ( 0 ) , then D = 0 . Lemma 2 . The gradient flow iterates satisfy the following properties : • For any 1 ≤ k ≤ L , ‖Wk‖2F − ‖Wk‖22 ≤ D. • For any 1 ≤ k < L , 〈vk+1 , uk〉2 ≥ 1− ( D+‖Wk+1 ( 0 ) ‖22+‖Wk ( 0 ) ‖ 2 2 ) /‖Wk+1‖22 . • Suppose max1≤k≤L ‖Wk‖F →∞ , then ∣∣∣〈wprod/∏Lk=1 ‖Wk‖F , v1〉∣∣∣→ 1 . The proof is based on eq . ( 2 ) and eq . ( 3 ) . If Wk ( 0 ) W > k ( 0 ) = W > k+1 ( 0 ) Wk+1 ( 0 ) , then eq . ( 2 ) gives that Wk+1 and Wk have the same singular values , and Wk+1 ’ s right singular vectors and Wk ’ s left singular vectors are the same . If it is true for any two adjacent layers , since WL is a row vector , all layers have rank 1 . With general initialization , we have similar results when ‖Wk‖F is large enough so that the initialization is negligible . Careful calculations give the exact results in Lemma 2 . An interesting point is that the implicit regularization result in Lemma 2 helps establish risk convergence in Theorem 1 . Specifically , by Lemma 2 , if all layers have large norms , ‖WL · · ·W2‖ will be large . If the risk is not minimized to 0 , ‖∇R ( wprod ) ‖ will be lower bounded by a positive constant , and thus ‖∂R/∂W1‖F = ‖WL · · ·W2‖‖∇R ( wprod ) ‖ will be large . Invoking eq . ( 1 ) , Lemma 1 and eq . ( 3 ) gives a contradiction . Since the risk has no finite optimum , ‖Wk‖F →∞ .

Summary: In this work the authors prove several claims regarding the inductive bias of gradient descent and gradient flow trained on deep linear networks with linearly separable data. They show that asymptotically gradient descent minimizes the risk, each weight matrix converges to its rank one approximation and the top singular vectors of two adjacent weight matrices align. Furthermore, for the logistic and exponential loss the induced linear predictor converges to the max margin solution.