On the Global Convergence  of Training Deep Linear ResNets

1 INTRODUCTION . Despite the remarkable power of deep neural networks ( DNNs ) trained using stochastic gradient descent ( SGD ) in many machine learning applications , theoretical understanding of the properties of this algorithm , or even plain gradient descent ( GD ) , remains limited . Many key properties of the learning process for such systems are also present in the idealized case of deep linear networks . For example , ( a ) the objective function is not convex ; ( b ) errors back-propagate ; and ( c ) there is potential for exploding and vanishing gradients . In addition to enabling study of systems with these properties in a relatively simple setting , analysis of deep linear networks also facilitates the scientific understanding of deep learning because using linear networks can control for the effect of architecture choices on the expressiveness of networks ( Arora et al. , 2018 ; Du & Hu , 2019 ) . For these reasons , deep linear networks have received extensive attention in recent years . One important line of theoretical investigation of deep linear networks concerns optimization landscape analysis ( Kawaguchi , 2016 ; Hardt & Ma , 2016 ; Freeman & Bruna , 2016 ; Lu & Kawaguchi , 2017 ; Yun et al. , 2018 ; Zhou & Liang , 2018 ) , where major findings include that any critical point of a deep linear network with square loss function is either a global minimum or a saddle point , and identifying conditions on the weight matrices that exclude saddle points . Beyond landscape analysis , another research direction aims to establish convergence guarantees for optimization algorithms ( e.g . GD , SGD ) for training deep linear networks . Arora et al . ( 2018 ) studied the trajectory of gradient flow and showed that depth can help accelerate the optimization of deep linear networks . Ji & Telgarsky ( 2019 ) ; Gunasekar et al . ( 2018 ) investigated the implicit bias of GD for training deep linear networks and deep linear convolutional networks respectively . More recently , Bartlett et al . ( 2019 ) ; Arora et al . ( 2019a ) ; Shamir ( 2018 ) ; Du & Hu ( 2019 ) analyzed the optimization trajectory of GD for training deep linear networks and proved global convergence rates under certain assumptions on the training data , initialization , and neural network structure . Inspired by the great empirical success of residual networks ( ResNets ) , Hardt & Ma ( 2016 ) considered identity parameterizations in deep linear networks , i.e. , parameterizing each layer ’ s weight matrix as I ` W , which leads to the so-called deep linear ResNets . In particular , Hardt & Ma ( 2016 ) established the existence of small norm solutions for deep residual networks with sufficiently large depth L , and proved that there are no critical points other than the global minimum when the maximum spectral norm among all weight matrices is smaller than Op1 { Lq . Motivated by this intriguing finding , Bartlett et al . ( 2019 ) studied the convergence rate of GD for training deep linear networks with identity initialization , which is equivalent to zero initialization in deep linear ResNets . They assumed whitened data and showed that GD can converge to the global minimum if ( i ) the training loss at the initialization is very close to optimal or ( ii ) the regression matrix Φ is symmetric and positive definite . ( In fact , they proved that , when Φ is symmetric and has negative eigenvalues , GD for linear ResNets with zero-initialization does not converge . ) Arora et al . ( 2019a ) showed that GD converges under substantially weaker conditions , which can be satisfied by random initialization schemes . The convergence theory of stochastic gradient descent for training deep linear ResNets is largely missing ; it remains unclear under which conditions SGD can be guaranteed to find the global minimum . In this paper , we establish the global convergence of both GD and SGD for training deep linear ResNets without any condition on the training data . More specifically , we consider the training of L-hidden-layer deep linear ResNets with fixed linear transformations at input and output layers . We prove that under certain conditions on the input and output linear transformations , GD and SGD can converge to the global minimum of the training loss function . Moreover , when specializing to appropriate Gaussian random linear transformations , we show that , as long as the neural network is wide enough , both GD and SGD with zero initialization on all hidden weights can find the global minimum . There are two main ingredients of our proof : ( i ) establishing restricted gradient bounds and a smoothness property ; and ( ii ) proving that these properties hold along the optimization trajectory and further lead to global convergence . We point out the second aspect is challenging especially for SGD due to the uncertainty of its optimization trajectory caused by stochastic gradients . We summarize our main contributions as follows : • We prove the global convergence of GD and SGD for training deep linear ResNets . Specifically , we derive a generic condition on the input and output linear transformations , under which both GD and SGD with zero initialization on all hidden weights can find global minima . Based on this condition , one can design a variety of input and output transformations for training deep linear ResNets . • When applying appropriate Gaussian random linear transformations , we show that as long as the neural network width satisfies m “ Ωpkrκ2q , with high probability , GD can converge to the global minimum up to an -error within Opκ logp1 { qq iterations , where k , r are the output dimension and the rank of training data matrix X respectively , and κ “ } X } 22 { σ2rpXq denotes the condition number of the covariance matrix of the training data . Compared with previous convergence results for training deep linear networks from Du & Hu ( 2019 ) , our condition on the neural network width is independent of the neural network depth L , and is strictly better by a factor of OpLκq . • Using the same Gaussian random linear transformations , we also establish the convergence guarantee of SGD for training deep linear ResNets . We show that if the neural network width satisfies m “ rΩ ` krκ2 log2p1 { q ¨n2 { B2 ˘ , with constant probability , SGD can converge to the global minimum up to an -error within rO ` κ2 ´1 logp1 { q ¨ n { B ˘ iterations , where n is the training sample size and B is the minibatch size of stochastic gradient . This is the first global convergence rate of SGD for training deep linear networks . Moreover , when the global minimum of the training loss is 0 , we prove that SGD can further achieve linear rate of global convergence , and the condition on the neural network width does not depend on the target error . As alluded to above , we analyze networks with d inputs , k outputs , and m ě maxtd , ku nodes in each hidden layer . Linear transformations that are fixed throughout training map the inputs to the first hidden layer , and the last hidden layer to the outputs . We prove that our bounds hold with high probability when these input and output transformations are randomly generated by Gaussian distributions . If , instead , the input transformation simply copies the inputs onto the first d compo- nents of the first hidden layer , and the output transformation takes the first k components of the last hidden layer , then our analysis does not provide a guarantee . There is a good reason for this : a slight modification of a lower bound argument from Bartlett et al . ( 2019 ) demonstrates that GD may fail to converge in this case . However , we describe a similarly simple , deterministic , choice of input and output transformations such that wide enough networks always converge . The resulting condition on the network width is weaker than that for Gaussian random transformations , and thus improves on the corresponding convergence guarantee for linear networks , which , in addition to requiring wider networks , only hold with high probability for random transformations . 1.1 ADDITIONAL RELATED WORK . In addition to what we discussed above , a large bunch of work focusing on the optimization of neural networks with nonlinear activation functions has emerged . We will briefly review them in this subsection . It is widely believed that the training loss landscape of nonlinear neural networks is highly nonconvex and nonsmooth ( e.g. , neural networks with ReLU/LeakyReLU activation ) , thus it is fundamentally difficult to characterize the optimization trajectory and convergence performance of GD and SGD . Some early work ( Andoni et al. , 2014 ; Daniely , 2017 ) showed that wide enough ( polynomial in sample size n ) neural networks trained by GD/SGD can learn a class of continuous functions ( e.g. , polynomial functions ) in polynomial time . However , those works only consider training some of the neural network weights rather than all of them ( e.g. , the input and output layers ) 1 . In addition , a series of papers investigated the convergence of gradient descent for training shallow networks ( typically 2-layer networks ) under certain assumptions on the training data and initialization scheme ( Tian , 2017 ; Du et al. , 2018b ; Brutzkus et al. , 2018 ; Zhong et al. , 2017 ; Li & Yuan , 2017 ; Zhang et al. , 2018 ) . However , the assumptions made in these works are rather strong and not consistent with practice . For example , Tian ( 2017 ) ; Du et al . ( 2018b ) ; Zhong et al . ( 2017 ) ; Li & Yuan ( 2017 ) ; Zhang et al . ( 2018 ) assumed that the label of each training data is generated by a teacher network , which has the same architecture as the learned network . Brutzkus et al . ( 2018 ) assumed that the training data is linearly separable . Li & Liang ( 2018 ) addressed this drawback ; they proved that for two-layer ReLU network with cross-entropy loss , as long as the neural network is sufficiently wide , under mild assumptions on the training data SGD with commonly-used Gaussian random initialization can achieve nearly zero expected error . Du et al . ( 2018c ) proved the similar results of GD for training two-layer ReLU networks with square loss . Beyond shallow neural networks , Allen-Zhu et al . ( 2019 ) ; Du et al . ( 2019 ) ; Zou et al . ( 2019 ) generalized the global convergence results to multi-layer over-parameterized ReLU networks . Chizat et al . ( 2019 ) showed that training over-parameterized neural networks actually belongs to a so-called “ lazy training ” regime , in which the model behaves like its linearization around the initialization . Furthermore , the parameter scaling is more essential than over-paramterization to make the model learning within the “ lazy training ” regime . Along this line of research , several follow up works have been conducted . Oymak & Soltanolkotabi ( 2019 ) ; Zou & Gu ( 2019 ) ; Su & Yang ( 2019 ) ; Kawaguchi & Huang ( 2019 ) improved the convergence rate and over-parameterization condition for both shallow and deep networks . Arora et al . ( 2019b ) showed that training a sufficiently wide deep neural network is almost equivalent to kernel regression using neural tangent kernel ( NTK ) , proposed in Jacot et al . ( 2018 ) . Allen-Zhu et al . ( 2019 ) ; Du et al . ( 2019 ) ; Zhang et al . ( 2019 ) proved the global convergence for training deep ReLU ResNets . Frei et al . ( 2019 ) proved the convergence of GD for training deep ReLU ResNets under an over-parameterization condition that is only logarithmic in the depth of the network , which partially explains why deep residual networks are preferable to fully connected ones . However , all the results in Allen-Zhu et al . ( 2019 ) ; Du et al . ( 2019 ) ; Zhang et al . ( 2019 ) ; Frei et al . ( 2019 ) require a very stringent condition on the network width , which typically has a high-degree polynomial dependence on the training sample size n. Besides , the results in Allen-Zhu et al . ( 2019 ) ; Zhang et al . ( 2019 ) also require that all data points are separated by a positive distance and have unit norm . As shown in Du & Hu ( 2019 ) and will be proved in this paper , for deep linear ( residual ) networks , there is no assumption on the training data , and the condition on the network width is significantly milder , which is independent of the sample size n. While achieving a stronger result for linear networks than for nonlinear ones is not surprising , we believe that our analysis , conducted in the idealized deep linear case , can provide useful insights to understand optimization in the nonlinear case . 1In Daniely ( 2017 ) , the weight changes in all hidden layers make negligible contribution to the final output , thus can be approximately treated as only training the output layer . Two concurrent works analyze gradient descent applied to deep linear ( residual ) networks ( Hu et al. , 2020 ; Wu et al. , 2019 ) . Hu et al . ( 2020 ) consider deep linear networks with orthogonal initialization , and Wu et al . ( 2019 ) consider zero initialization on the last layer and identity initialization for the rest of the layers , which are similar to our setting . However , there are several differences between their work and ours . One major difference is that Hu et al . ( 2020 ) and Wu et al . ( 2019 ) only prove global convergence for GD , but our results cover both GD and SGD . In addition , Hu et al . ( 2020 ) focuses on proving the global convergence of GD for sufficiently wide networks , while we provide a generic condition on the input and output linear transformations for ensuring global convergence . Wu et al . ( 2019 ) assumes whitened data and proves a OpL3 logp1 { qq bound on the number of iterations required for GD to converge , where we establish a Oplogp1 { qq2 bound .

Summary: In this paper, the authors study the convergence of (stochastic) gradient descent in training deep linear residual networks, where linear transformation at input and output layers are fixed and matrices in other layers are trained. They first establish a global convergence of GD/SGD under some conditions on the fixed linear transformations. They they showed that for Gaussian random input and output transformation, global convergence still holds under conditions on the width of networks strictly milder than the literature. Linear convergence rate of SG/SGD are also established.