Faster Directional Convergence of Linear Neural Networks under Spherically Symmetric Data

In this paper , we study gradient methods for training deep linear neural networks with binary cross-entropy loss . In particular , we show global directional convergence guarantees from a polynomial rate to a linear rate for ( deep ) linear networks with spherically symmetric data distribution , which can be viewed as a specific zero-margin dataset . Our results do not require the assumptions in other works such as small initial loss , presumed convergence of weight direction , or overparameterization . We also characterize our findings in experiments . 1 Introduction . Local v.s . non-local analysis . Deep neural networks have been successfully trained with simple gradient-based methods , despite the inherent non-convexity of the objective function . Recently , a number of works proved the convergence of gradient methods to global minima for ultra-wide neural networks [ 1 , 4 , 12 , 13 , 20 , 31 ] . These works essentially performed “ local analysis ” because in their proofs the parameters stay close to initialization during training . In the large width setting , there exists a global minimum near a random initial point , thus staying in the local region can still result in convergence to global minima . Despite the technical convenience of handling a local region , in practical training , the parameters often travel far from initialization ( see , e.g. , [ 15 ] ) . To understand the practical optimization trajectory , it is important to develop a non-local convergence analysis . Deep linear nets with quadratic loss : “ local analysis ” . To understand the behavior of gradient descent ( GD ) for general neural nets , we need to understand GD for deep linear nets . A number of works analyzed linear networks with the square loss . Bartlett et al . [ 7 ] provided that gradient descent converges to the target matrix at a linear rate from identity initialization , while assuming the target matrix is either close to identity or positive definite . Arora et al . [ 2 ] proved linear convergence of deep linear networks if the initialization has a positive “ deficiency margin '' and is nearly balanced . Later , a few works followed a similar idea to neural tangent kernel ( NTK ) [ 20 ] to establish convergence analysis . Du and Hu [ 11 ] showed GD with Gaussian random initialization converges to a global minimum at a linear rate if the width of every hidden layer is large enough . Hu et al . [ 19 ] improved the width requirement to be independent of depth , by utilizing orthogonal weight initialization , but they assume each layer to have the same width . All the above works are not non-local analysis . Deep linear nets with quadratic loss : non-local analysis . Eftekhari [ 14 ] provided non-local convergence analysis for deep linear nets with quadratic loss . When one layer has only one neuron ( including scalar output case ) and assumming that the input data are whitened , they proved that gradient flow converges to global minimizers starting from balanced initialization . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we are interested in the classification task with binary cross-entropy loss . To our knowledge , there was no non-local convergence analysis of gradient flow for deep linear nets in such a scheme . Deep linear nets with logit loss : final-phase analysis . Several recent works [ 17 , 38 , 23 , 21 , 37 ] studied the exponential-type loss for deep linear networks . Specifically , Gunasekar et al . [ 17 ] , Nacson et al . [ 38 ] proved the convergence to max-margin solution , but under the assumption that the weight direction and the loss have converged to global optima , which is a “ final-point analysis ” . Lyu and Li [ 37 ] , Ji and Telgarsky [ 21 , 23 ] also proved the convergence to max-margin solution under the assumption that the initial point has already obtained zero classification error ( weaker assumption than [ 17 , 38 ] ) , i.e. , they analyzed the “ final phase ” of training . In this work , we would like to understand the entire training dynamics , not just the “ late training ” period . 1.1 Our Contributions . In this paper , we analyze gradient flow for deep linear networks with logit loss ( i.e . binary crossentropy loss ) . The main contributions of this paper are summarized as follows : • Convergence result . We prove the global convergence of gradient flow for minimizing a population logit loss with deep linear nets , under the assumption that the input data is spherically symmetric ( Theorem 3 ) . This assumption covers the standard Gaussian distribution and uniform distribution on the sphere . To our knowledge , this is the first global analysis of gradient flow on deep linear nets with logit loss , though under certain strong assumptions . We emphasize that our analysis is beyond the “ lazy training ” scheme . • Convergence rate . We also establish explicit convergence rate of gradient flow ( Theorem 3 ) . Denote θ ( t ) as the angle between we ( t ) and v , where we ( t ) is the collection of parameters at time t following gradient flow , and v is the directional global minimizer . In the first phase , we have polynomial convergence rate of cos ( θ ( t ) ) , that is , cos ( θ ( t ) ) ≥ 1−O ( ( N1t ) − 2N1π ) , where N1 + 2 is the depth of the deep linear network . In the second phase , we have linear convergence rate of cos ( θ ( t ) ) , that is , cos ( θ ( t ) ) ≥ 1− eO ( −t ) . And the second phase begins when the induced weight norm changes from descending to ascending mentioned below . • Weight norm change pattern . We prove that the induced weight norm goes through descending and ascending periods . If the initial induced weight norm starts with descending behavior , then after finite time , it will change to ascending and continues increasing to infinity . If the initial induced weight norm begins with ascending behavior , then it would increase to infinity directly . We also verify our results in numerical experiments including the descending and ascending behavior of weight norm and the convergence rates in our setting . 1.2 Additional Related work . Directional convergence for exponentially-tailed loss . Previously , we mentioned a few works on deep linear nets for classification . Next , we discuss convergence results on classification for more general settings ( not just deep linear nets ) . For binary classification problems , oftentimes it is the direction of the predictor that matters , while the norm is less important . For a finite number of linearly separable data , there can be infinitely many directions that lead to zero loss , and classical optimization results do not show which one GD converges to . In many recent works [ 22 , 24 , 25 , 38 , 39 , 46 , 16 ] , it is shown that for binary classification problems with exponential-type loss ( which can be viewed as the 1-layer 1-linear-neuron net case ) and linearly separable data , GD converges to the ` 2 maximum margin direction . These works assume there is a positive margin . Ji and Telgarsky [ 22 ] , Ji et al . [ 25 ] also consider a non-separable case , but still requires a positive margin in the “ separable part ” . Moreover , some works [ 21 , 23 , 17 , 38 ] study the convergence of GD for training deep linear nets . Lyu and Li [ 37 ] further studied deep homogeneous networks ( covering deep linear nets and deep ReLU nets ) . Yun et al . [ 51 ] provided a unified framework that connects several existing results under a general tensor formulation of linear networks . A few other works study wide non-linear neural nets , e.g. , Chizat and Bach [ 9 ] , Bach and Chizat [ 5 ] improved previous results in the infinitely wide two-layer case , identifying the learned classifier as the solution of a convex max-margin problem . There are two differences between those works and ours . First , none of these works have performed non-local convergence analysis for deep linear nets as the current work does . Second , they consider finite data with a positive margin , and we consider infinitely many data with a zero margin . We underline that previous convergence rates are inversely proportional to such a margin , which is vacuous in our zero-margin setting . More discussions on non-local analysis of deep linear nets with quadratic loss . We mentioned Eftekhari [ 14 ] earlier ; here we add more discussions . Besides the main difference in the loss functions , there are two other differences between Eftekhari [ 14 ] and our work . First , they assume whitened data , while we assume spherically symmetric data . Second , their convergence rate result requires an extra assumption that the inverse spectral gap of the target matrix is small ( [ 14 , Theorem 4.4 ] ) , while our convergence rate result does not require extra assumption . Besides the work [ 14 ] , another earlier work [ 6 ] performed “ non-local analysis ” , but as pointed out by [ 14 ] , their work only ensures convergence to the minimizer in a constrained subset , not necessarily the global minimizer ( see Appendix F in [ 14 ] for detail ) . Global landscape analysis . In our work , the term “ non-local analysis ” should actually be called “ non-local convergence analysis ” . There is another line of works on “ non-local landscape analysis ” ( see , e.g. , [ 26 , 36 , 27 , 42 , 52 , 41 , 30 , 10 , 34 , 33 , 35 , 49 , 44 ] and the surveys [ 48 , 47 ] ) which analyze the properties of stationary points , local minima , strict local minima , etc . These works do not directly analyze the convergence of GD , thus are different from our work . 2 Preliminaries . We consider a binary classification problem where the data are generated as follows . Let v ∈ Rd be a fixed vector satisfying ‖v‖ = 1 . The input data point x ∈ Rd is sampled from a distribution D and the label for each x is y ( x ) = sgn ( v > x ) ∈ { −1 , +1 } 1 . Suppose we want to learn the parameter w for a predictor φ ( · , w ) : Rd → R ( the specific form of φ will be discussed later ) . We use the population logit loss ` ( z ) : = ln ( 1 + e−z ) ( binary cross-entropy ) with z ( x , w ) = y ( x ) · φ ( x , w ) . The parameter is learned by solving the optimization problem min w L ( w ) : = Ex∼D ln ( 1 + e−y ( x ) φ ( x , w ) ) . ( 1 ) We focus on the following standard gradient methods . 1 ) Gradient flow : We initialize w ( 0 ) , and for every t > 0 let w ( t ) be the solution of the differential equation : ẇ ( t ) = −∇L ( w ( t ) ) . 2 ) Gradient descent : We initialize w ( 0 ) and set a sequence of positive learning rates { ηk } ∞k=1 . At each iteration n > 0 , we do a single step in the negative direction of the gradient , that is , w ( k + 1 ) = w ( k ) − ηk∇L ( w ( k ) ) . Notation . We denote vectors by lowercase bold letters ( e.g. , u , x , v ) . Moreover , ‖ · ‖ denotes the standard ` 2-norm for a vector , or the spectral norm for a given matrix . Let Σ = Ex∼Dxx > be the population covariance matrix , and λmax ( A ) be the maximum eigenvalue of a real symmetric matrix A . We by Sd−1 denote the surface of an d-dimensional unit sphere , and by Ik the k-dimensional identity matrix . We let w : = w/‖w‖ whenever ‖w‖ 6= 0 . Given vectors w = ( w1 , . . . , wd ) > ∈ Rd and v = ( v1 , . . . , vd ) > ∈ Rd , we let θ ( w , v ) : = arccos [ w > v/ ( ‖w‖ · ‖v‖ ) ] ∈ [ 0 , π ] denote the angle between w and v , θ ( t ) : = θ ( w ( t ) , v ) and θ ( n ) : = θ ( w ( n ) , v ) . We let Dw , v be the marginal distribution of x = ( x1 , . . . , xd ) > on the subspace spanned by two linearly independent vectors w , v ( as a distribution over R2 ) , and Dv similarly . Let D2 : = De1 , e2 where e1 , . . . , ed are the d-dimensional coordinate directions , and c0 : = Ex∼D2‖x‖ . Additionally , we use the standard O ( · ) , Ω ( · ) and Θ ( · ) to hide universal constant factors . We say the algorithm achieves superpolynomial convergence if it takes O ( lnα ( 1/ ) ) iterations ( for a constant α > 0 ) to achieve -error . 1Here , sgn ( z ) = 1 if z ≥ 0 , otherwise −1 .

Summary: In this paper the authors present a convergence proof of linear neural networks trained with a logit loss. The analysis is based on previous works of Telgarsky, and is only concerned with binary classification of linearly separable points with 0 margin. In addition the authors make the assumption that the data distribution of the points are isotropic namely, spherically symmetric. With this the authors prove a directional convergence result, that states while the parameters magnitude approaches infinity as seen in previous works with separable data and logit losses,  the direction of the parameters converges to the orthogonal vector to the decision hyperplane. The authors present the result for shallow (1 layer) networks as well as deep linear networks. In addition to convergence the authors observe a trend of decreasing and then increasing parameter magnitudes that they also analyze empirically.