The Geometry of Sign Gradient Descent

1 INTRODUCTION . We consider an unconstrained , continuous optimization problem , minx∈Rd f ( x ) , with a differentiable and lower-bounded objective f : Rd → R. The prototypical first-order optimization algorithm to solve this problem is gradient descent ( GD ) , which iteratively updates xt+1 = xt−αt∇ft , where ft : = f ( xt ) . Several recent works have considered the sign gradient descent ( signGD ) method , xt+1 = xt − αt sign ( ∇ft ) , ( 1 ) where the application of the sign is to be understood elementwise . This method is attractive in distributed optimization where it conveniently reduces the communication cost to a single bit per gradient coordinate ( e.g. , Seide et al. , 2014 ; Karimireddy et al. , 2019 ) , but its interest extends beyond that . Balles & Hennig ( 2018 ) and Bernstein et al . ( 2018 ) point out that sign gradient descent is closely related to the popular Adam method ( Kingma & Ba , 2015 ) and demonstrate that it often achieves similar practical performance on deep learning tasks . Studying sign gradient descent can thus be seen as a step towards a better understanding of the ubiquitous Adam method . 1.1 OVERVIEW AND CONTRIBUTIONS . In this work , we make the following contributions : 1 . We review the concept of steepest descent with respect to arbitrary norms , which includes both gradient descent and sign gradient descent . While these ideas feature in previous works , we provide a concise and general write-up which we hope to be useful in its own right . We emphasize the role of the smoothness constant of f w.r.t . a norm as a major driver of the performance of the associated steepest descent method . 2 . Since sign gradient descent is steepest descent w.r.t . the maximum norm , we study the corresponding smoothness constant L∞ , isolating properties of the objective function which favor sign gradient descent . We pinpoint two requirements on its Hessian : ( ( i ) some degree of “ diagonal dominance ” and ( ii ) the maximal eigenvalue being much larger than the average eigenvalue . 3 . We verify our theory experimentally on both quadratic functions and deep networks . 1.2 RESTRICTIONS . While modern machine learning often relies on stochastic optimization , this work focuses on the geometric aspects of sign gradient descent and we restrict ourselves to the deterministic setting to facilitate this analysis . Furthermore , our interest in signGD is from an optimization perspective and we will not consider the question of generalization . It should be noted that sign gradient descent has been observed to lead to slightly deteriorating generalization performance compared to gradient descent in neural network training ( Wilson et al. , 2017 ; Balles & Hennig , 2018 ) . 2 STEEPEST DESCENT AND SMOOTHNESS CONSTANTS . In this section , we review the steepest descent1 framework , which encompasses both gradient descent and sign gradient descent . Embedding both methods in a common framework will facilitate our comparative analysis . Steepest descent with respect to a norm ‖ · ‖ is defined via the update rule xt+1 ∈ arg min x∈Rd ( ft + 〈∇ft , x− xt〉+ 1 2αt ‖x− xt‖2 ) , ( 2 ) minimizing a local linear approximation of f plus a quadratic norm penalty . This minimizer need not be unique , in which case steepest descent is to be understood as choosing any solution . It is straight-forward to see that gradient descent is steepest descent with respect to the Euclidean norm . Similarly , a version of sign gradient descent arises as steepest descent with respect to the maximum norm , ‖z‖∞ : = maxi |zi| . Namely , xt − αt‖∇ft‖1 sign ( ∇ft ) ∈ arg min x∈Rd ( ft + 〈∇ft , x− xt〉+ 1 2αt ‖x− xt‖2∞ ) . ( 3 ) This update is equivalent to Eq . ( 1 ) up to the scaling with the gradient norm ‖∇ft‖1 which may be subsumed in the step size.2 Table 1 lists a few more steepest descent methods , which are discussed in more detail in Appendix A.1 . In the following , we will explore steepest descent in more detail by a ) relating it to the concept of smoothness w.r.t . arbitrary norms ; b ) tying its convergence speed to the corresponding smoothness constant ; c ) linking the smoothness constants to properties of the Hessian . 2.1 SMOOTHNESS WITH RESPECT TO ARBITRARY NORMS . Steepest descent is closely connected to the concept of smoothness w.r.t . the relevant norm . We recall that a function f is L-smooth w.r.t . a norm ‖ · ‖ if ‖∇f ( y ) −∇f ( x ) ‖∗ ≤ L‖y − x‖ , ∀x , y ∈ Rd , ( 4 ) where ‖ · ‖∗ denotes the dual norm of ‖ · ‖ , defined as ‖z‖∗ : = max‖x‖≤1〈z , x〉 . Table 1 lists the dual norm pairs for the methods under consideration in this paper . Smoothness directly motivates steepest descent via the following quadratic bound on the function : Lemma 1 . If f isL-smooth w.r.t . ‖·‖ , then f ( y ) ≤ f ( x ) +〈∇f ( x ) , y−x〉+L2 ‖y−x‖ 2 ∀x , y ∈ Rd . 1Also known as generalized or non-Euclidean gradient descent ( Carlson et al. , 2015 ; Kelner et al. , 2014 ) . 2 In this paper we study this version of sign gradient descent to facilitate the analysis in the steepest descent framework . In practice it is more common to use Eq . ( 1 ) with a constant or manually decreasing step size . In Appendix C , we discuss a possible explanation for this discrepancy . Setting x = xt we get f ( y ) ≤ ft + 〈∇ft , y − xt〉 + L2 ‖y − xt‖ 2 which resembles Eq . ( 2 ) and minimizing with respect to y results in a steepest descent update with step size αt = 1/L . ( The proofs of this lemma and all subsequent results are provided in Appendix E. ) Due to the equivalence of norms on Rd , a function that is smooth with respect to one norm is also smooth with respect to any other norm . However , the tightest possible smoothness constant , L : = sup x6=y ‖∇f ( y ) −∇f ( x ) ‖∗ ‖y − x‖ , ( 5 ) will differ depending on the choice of norm . In the following , when we say f is L-smooth w.r.t . ‖·‖ , we will always assume L to be given by Eq . ( 5 ) . As we will see next , this constant will govern the convergence speed of the corresponding steepest descent method . 2.2 CONVERGENCE OF STEEPEST DESCENT . We now recall existing convergence results for steepest descent methods based on the smoothness constants w.r.t . arbitrary norms . The assumption of smoothness is crucial since it guarantees an improvement in function value : Lemma 2 . If f is L-smooth w.r.t . ‖·‖ , then a single steepest descent step with step size 1/L satisfies f ( x+ ) ≤ f ( x ) − 1 2L ‖∇f ( x ) ‖2∗ . ( 6 ) Without additional assumptions , this yields O ( 1/T ) convergence to a first-order stationary point : Proposition 1 . If f is L-smooth w.r.t . ‖ · ‖ , then steepest descent ( Eq . 2 ) with αt ≡ 1/L satisfies 1 T T−1∑ t=0 ‖∇ft‖2∗ ≤ 2L ( f0 − f ? ) T . ( 7 ) By nature of Proposition 1 , all steepest descent methods will enjoy the same rate of convergence . But Lemma 2 and Proposition 1 differ across different steepest descent methods in ( i ) the smoothness constant , and ( ii ) the norm in which the gradient magnitude is measured . These two aspects will play a role when we compare sign gradient descent and gradient descent in Section 4 . Appendix A contains stronger convergence results under additional assumptions , in particular linear convergence under the Polyak-Łojasiewicz condition . We omit these results here as they are not of primary interest to our discussion and , in fact , hide the dependence on the gradient norm which we see clearly in Lemma 2 and Proposition 1 . 2.3 SMOOTHNESS AS A BOUND ON THE HESSIAN . The definition of smoothness in Eq . ( 4 ) is unwieldy . For the Euclidean norm , a more intuitive characterization is widely known : a twice-differentiable function f isL2-smooth w.r.t . the Euclidean norm if and only if ‖∇2f ( x ) ‖2 ≤ L2 for all x ∈ Rd . Here , ‖ · ‖2 for matrices denotes the spectral norm , given by the largest-magnitude eigenvalue for symmetric matrices . In the following proposition we show that smoothness with respect to any other norm likewise arises from a bound on the Hessian , but in different matrix norms . In Section 3 , we will use this to better understand the smoothness constant w.r.t . the maximum norm , which critically affects the performance of signGD . Proposition 2 . For any norm ‖ · ‖ on Rd , we define the matrix norm ‖H‖ : = sup ‖x‖≤1 ‖Hx‖∗ . ( 8 ) Then , a twice-differentiable f is L-smooth w.r.t . ‖ · ‖ if and only if ‖∇2f ( x ) ‖ ≤ L for all x . The proof can be found in Appendix E ; we are not aware of prior works showing this for the general case . For the Euclidean norm , Proposition 2 gives back the familiar spectral norm bound . λ m a x = 2 .0 L∞ = 3.00 L∞ = 3.34 L∞ = 3.87 λ m a x = 4 .0 L∞ = 5.00 L∞ = 6.03 L∞ = 7.60 3 UNDERSTANDING THE L∞-SMOOTHNESS CONSTANT . We have seen in Section 2.2 that the smoothness constant w.r.t . a norm is a crucial driver of the convergence speed of the associated steepest descent method . While we have a good intuition for Euclidean smoothness—an upper bound on the eigenvalues of the Hessian—this is somewhat lacking for smoothness w.r.t . the maximum norm . Our goal in this section is to understand which properties of the objective function affect the constantL∞ . The Euclidean smoothness constantL2 will thereby serve as a reference point . We assume f to be L2-smooth w.r.t . ‖ · ‖2 and L∞-smooth w.r.t . ‖ · ‖∞ in the “ tight ” sense of Eq . ( 5 ) . Basic inequalities between the norms yield ( see Appendix E ) L2 ≤ L∞ ≤ dL2 . ( 9 ) The smoothness constant governing the convergence speed of sign gradient descent will always be larger than the one pertinent to gradient descent . This does not mean that signGD is always worse than GD . In addition to being a bound , the smoothness constant is only one factor influencing the convergence speed ; Lemma 2 shows that the dual gradient norm plays a role as well . As we will discuss in Section 4 , this gradient norm is larger for signGD which can make up for the disadvantage of a larger smoothness constant . In this section we will characterize conditions under which L∞ is “ not too bad ” —in particular much smaller than its worst case of dL2—such that signGD can be competitive with gradient descent . Our discussion will use the Hessian-based formulation of the smoothness constants ( Proposition 2 ) . We assume f to be twice-differentiable . As mentioned before , the Euclidean smoothness constant is simply given by the largest-magnitude eigenvalue . The max-norm smoothness constant is determined by L∞ = sup x∈Rd ‖∇2f ( x ) ‖∞,1 , ‖H‖∞,1 : = sup ‖x‖∞≤1 ‖Hx‖1 . ( 10 ) Unfortunately , computing ‖H‖∞,1 is NP-hard ( Rohn , 2000 ) . However , we can obtain some insight through the following upper bound . Proposition 3 . LetH ∈ Rd×d be nonzero , positive semi-definite with eigenvalues λ1 , . . . , λd . Then ‖H‖∞,1 ≤ ρdiag ( H ) −1 ∑ i λi with ρdiag ( H ) : = ∑ i |Hii|∑ i , j |Hij | . ( 11 ) The quantity ρdiag ( H ) ∈ [ d−1 , 1 ] measures the percentage of diagonal mass of the Hessian and thus relates to the “ axis-alignment ” of the objective . Indeed , diagonal matrices satisfy ρdiag = 1 and their eigenvectors coincide with the coordinate axes . Proposition 3 thus indicates that L∞ will depend both on the “ axis-alignment ” of the Hessian as well as its eigenvalues . This is in stark contrast to gradient descent , for which the relevant matrix norm , ‖H‖2 , is invariant to rotations . Figure 1 provides a two-dimensional illustration . Using Proposition 3 , we can now answer our question : Under what conditions is L∞ “ not too bad ” compared to L2 ? Firstly , L∞ benefits from high ρdiag , i.e. , an approximately diagonal Hessian or axis-aligned objective . Secondly , the sum of eigenvalues should be much smaller than its worst case of dλmax or , equivalently , the average eigenvalue should be much smaller than the maximal one : λ̄ : = 1d ∑ i λi λmax . The second condition will be met if the spectrum is dominated by a small number of “ outlier eigenvalues ” that far exceed the bulk of eigenvalues . Notably , both the relative axis-alignment and the skewed spectrum have been observed in neural network training ( Adolphs et al. , 2019 ; Chaudhari et al. , 2017 ; Ghorbani et al. , 2019 ) . The discussion in the preceding paragraphs was based on an upper bound , since ‖H‖∞,1 is generally intractable . Section 5 will feature experiments on quadratics of moderate dimension , where ‖H‖∞,1 can be computed to confirm these findings for the exact L∞ .

Summary: This paper studies sign gradient descent in the framework of steepest descent with respect to a general norm. The authors show the connection to a smoothness with respect to a general norm, which is further related to a bound on the Hessian. Based on the interpretation of steepest descent, the authors compare the behavior of sign gradient descent with the gradient descent, and show that the relative behavior depends on both the percentage of diagonal ? and the spread of eigenvalues. Some experimental results are reported to verify the theory.