Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data

In this paper , we study gradient methods for training deep linear neural networks with binary cross-entropy loss . In particular , we show global directional convergence guarantees from a polynomial rate to a linear rate for ( deep ) linear networks with spherically symmetric data distribution , which can be viewed as a specific zero-margin dataset . Our results do not require the assumptions in other works such as small initial loss , presumed convergence of weight direction , or overparameterization . We also characterize our findings in experiments . 1 Introduction . Local v.s . non-local analysis . Deep neural networks have been successfully trained with simple gradient-based methods , despite the inherent non-convexity of the objective function . Recently , a number of works proved the convergence of gradient methods to global minima for ultra-wide neural networks [ 1 , 4 , 12 , 13 , 20 , 31 ] . These works essentially performed “ local analysis ” because in their proofs the parameters stay close to initialization during training . In the large width setting , there exists a global minimum near a random initial point , thus staying in the local region can still result in convergence to global minima . Despite the technical convenience of handling a local region , in practical training , the parameters often travel far from initialization ( see , e.g. , [ 15 ] ) . To understand the practical optimization trajectory , it is important to develop a non-local convergence analysis . Deep linear nets with quadratic loss : “ local analysis ” . To understand the behavior of gradient descent ( GD ) for general neural nets , we need to understand GD for deep linear nets . A number of works analyzed linear networks with the square loss . Bartlett et al . [ 7 ] provided that gradient descent converges to the target matrix at a linear rate from identity initialization , while assuming the target matrix is either close to identity or positive definite . Arora et al . [ 2 ] proved linear convergence of deep linear networks if the initialization has a positive “ deficiency margin '' and is nearly balanced . Later , a few works followed a similar idea to neural tangent kernel ( NTK ) [ 20 ] to establish convergence analysis . Du and Hu [ 11 ] showed GD with Gaussian random initialization converges to a global minimum at a linear rate if the width of every hidden layer is large enough . Hu et al . [ 19 ] improved the width requirement to be independent of depth , by utilizing orthogonal weight initialization , but they assume each layer to have the same width . All the above works are not non-local analysis . Deep linear nets with quadratic loss : non-local analysis . Eftekhari [ 14 ] provided non-local convergence analysis for deep linear nets with quadratic loss . When one layer has only one neuron ( including scalar output case ) and assumming that the input data are whitened , they proved that gradient flow converges to global minimizers starting from balanced initialization . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we are interested in the classification task with binary cross-entropy loss . To our knowledge , there was no non-local convergence analysis of gradient flow for deep linear nets in such a scheme . Deep linear nets with logit loss : final-phase analysis . Several recent works [ 17 , 38 , 23 , 21 , 37 ] studied the exponential-type loss for deep linear networks . Specifically , Gunasekar et al . [ 17 ] , Nacson et al . [ 38 ] proved the convergence to max-margin solution , but under the assumption that the weight direction and the loss have converged to global optima , which is a “ final-point analysis ” . Lyu and Li [ 37 ] , Ji and Telgarsky [ 21 , 23 ] also proved the convergence to max-margin solution under the assumption that the initial point has already obtained zero classification error ( weaker assumption than [ 17 , 38 ] ) , i.e. , they analyzed the “ final phase ” of training . In this work , we would like to understand the entire training dynamics , not just the “ late training ” period . 1.1 Our Contributions . In this paper , we analyze gradient flow for deep linear networks with logit loss ( i.e . binary crossentropy loss ) . The main contributions of this paper are summarized as follows : • Convergence result . We prove the global convergence of gradient flow for minimizing a population logit loss with deep linear nets , under the assumption that the input data is spherically symmetric ( Theorem 3 ) . This assumption covers the standard Gaussian distribution and uniform distribution on the sphere . To our knowledge , this is the first global analysis of gradient flow on deep linear nets with logit loss , though under certain strong assumptions . We emphasize that our analysis is beyond the “ lazy training ” scheme . • Convergence rate . We also establish explicit convergence rate of gradient flow ( Theorem 3 ) . Denote θ ( t ) as the angle between we ( t ) and v , where we ( t ) is the collection of parameters at time t following gradient flow , and v is the directional global minimizer . In the first phase , we have polynomial convergence rate of cos ( θ ( t ) ) , that is , cos ( θ ( t ) ) ≥ 1−O ( ( N1t ) − 2N1π ) , where N1 + 2 is the depth of the deep linear network . In the second phase , we have linear convergence rate of cos ( θ ( t ) ) , that is , cos ( θ ( t ) ) ≥ 1− eO ( −t ) . And the second phase begins when the induced weight norm changes from descending to ascending mentioned below . • Weight norm change pattern . We prove that the induced weight norm goes through descending and ascending periods . If the initial induced weight norm starts with descending behavior , then after finite time , it will change to ascending and continues increasing to infinity . If the initial induced weight norm begins with ascending behavior , then it would increase to infinity directly . We also verify our results in numerical experiments including the descending and ascending behavior of weight norm and the convergence rates in our setting . 1.2 Additional Related work . Directional convergence for exponentially-tailed loss . Previously , we mentioned a few works on deep linear nets for classification . Next , we discuss convergence results on classification for more general settings ( not just deep linear nets ) . For binary classification problems , oftentimes it is the direction of the predictor that matters , while the norm is less important . For a finite number of linearly separable data , there can be infinitely many directions that lead to zero loss , and classical optimization results do not show which one GD converges to . In many recent works [ 22 , 24 , 25 , 38 , 39 , 46 , 16 ] , it is shown that for binary classification problems with exponential-type loss ( which can be viewed as the 1-layer 1-linear-neuron net case ) and linearly separable data , GD converges to the ` 2 maximum margin direction . These works assume there is a positive margin . Ji and Telgarsky [ 22 ] , Ji et al . [ 25 ] also consider a non-separable case , but still requires a positive margin in the “ separable part ” . Moreover , some works [ 21 , 23 , 17 , 38 ] study the convergence of GD for training deep linear nets . Lyu and Li [ 37 ] further studied deep homogeneous networks ( covering deep linear nets and deep ReLU nets ) . Yun et al . [ 51 ] provided a unified framework that connects several existing results under a general tensor formulation of linear networks . A few other works study wide non-linear neural nets , e.g. , Chizat and Bach [ 9 ] , Bach and Chizat [ 5 ] improved previous results in the infinitely wide two-layer case , identifying the learned classifier as the solution of a convex max-margin problem . There are two differences between those works and ours . First , none of these works have performed non-local convergence analysis for deep linear nets as the current work does . Second , they consider finite data with a positive margin , and we consider infinitely many data with a zero margin . We underline that previous convergence rates are inversely proportional to such a margin , which is vacuous in our zero-margin setting . More discussions on non-local analysis of deep linear nets with quadratic loss . We mentioned Eftekhari [ 14 ] earlier ; here we add more discussions . Besides the main difference in the loss functions , there are two other differences between Eftekhari [ 14 ] and our work . First , they assume whitened data , while we assume spherically symmetric data . Second , their convergence rate result requires an extra assumption that the inverse spectral gap of the target matrix is small ( [ 14 , Theorem 4.4 ] ) , while our convergence rate result does not require extra assumption . Besides the work [ 14 ] , another earlier work [ 6 ] performed “ non-local analysis ” , but as pointed out by [ 14 ] , their work only ensures convergence to the minimizer in a constrained subset , not necessarily the global minimizer ( see Appendix F in [ 14 ] for detail ) . Global landscape analysis . In our work , the term “ non-local analysis ” should actually be called “ non-local convergence analysis ” . There is another line of works on “ non-local landscape analysis ” ( see , e.g. , [ 26 , 36 , 27 , 42 , 52 , 41 , 30 , 10 , 34 , 33 , 35 , 49 , 44 ] and the surveys [ 48 , 47 ] ) which analyze the properties of stationary points , local minima , strict local minima , etc . These works do not directly analyze the convergence of GD , thus are different from our work . 2 Preliminaries . We consider a binary classification problem where the data are generated as follows . Let v ∈ Rd be a fixed vector satisfying ‖v‖ = 1 . The input data point x ∈ Rd is sampled from a distribution D and the label for each x is y ( x ) = sgn ( v > x ) ∈ { −1 , +1 } 1 . Suppose we want to learn the parameter w for a predictor φ ( · , w ) : Rd → R ( the specific form of φ will be discussed later ) . We use the population logit loss ` ( z ) : = ln ( 1 + e−z ) ( binary cross-entropy ) with z ( x , w ) = y ( x ) · φ ( x , w ) . The parameter is learned by solving the optimization problem min w L ( w ) : = Ex∼D ln ( 1 + e−y ( x ) φ ( x , w ) ) . ( 1 ) We focus on the following standard gradient methods . 1 ) Gradient flow : We initialize w ( 0 ) , and for every t > 0 let w ( t ) be the solution of the differential equation : ẇ ( t ) = −∇L ( w ( t ) ) . 2 ) Gradient descent : We initialize w ( 0 ) and set a sequence of positive learning rates { ηk } ∞k=1 . At each iteration n > 0 , we do a single step in the negative direction of the gradient , that is , w ( k + 1 ) = w ( k ) − ηk∇L ( w ( k ) ) . Notation . We denote vectors by lowercase bold letters ( e.g. , u , x , v ) . Moreover , ‖ · ‖ denotes the standard ` 2-norm for a vector , or the spectral norm for a given matrix . Let Σ = Ex∼Dxx > be the population covariance matrix , and λmax ( A ) be the maximum eigenvalue of a real symmetric matrix A . We by Sd−1 denote the surface of an d-dimensional unit sphere , and by Ik the k-dimensional identity matrix . We let w : = w/‖w‖ whenever ‖w‖ 6= 0 . Given vectors w = ( w1 , . . . , wd ) > ∈ Rd and v = ( v1 , . . . , vd ) > ∈ Rd , we let θ ( w , v ) : = arccos [ w > v/ ( ‖w‖ · ‖v‖ ) ] ∈ [ 0 , π ] denote the angle between w and v , θ ( t ) : = θ ( w ( t ) , v ) and θ ( n ) : = θ ( w ( n ) , v ) . We let Dw , v be the marginal distribution of x = ( x1 , . . . , xd ) > on the subspace spanned by two linearly independent vectors w , v ( as a distribution over R2 ) , and Dv similarly . Let D2 : = De1 , e2 where e1 , . . . , ed are the d-dimensional coordinate directions , and c0 : = Ex∼D2‖x‖ . Additionally , we use the standard O ( · ) , Ω ( · ) and Θ ( · ) to hide universal constant factors . We say the algorithm achieves superpolynomial convergence if it takes O ( lnα ( 1/ ) ) iterations ( for a constant α > 0 ) to achieve -error . 1Here , sgn ( z ) = 1 if z ≥ 0 , otherwise −1 .

Summary: The authors study both shallow and deep linear networks under the assumption of spherically symmetrical input data. They show that gradient descent provides a super-polynomial convergence rate in this setting, which does not rely on any assumptions on the margin of the data or over-parameterization of the network. They provide some experiments demonstrating their results when their assumptions exactly hold, as well as with deep non-linear activation networks where their assumptions no longer hold.