Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity

1 INTRODUCTION . We study optimization algorithms for neural network training and aim to resolve the mystery of why adaptive methods converge fast . Specifically , we study gradient-based methods for minimizing a differentiable nonconvex function f : Rd → R , where f ( x ) can potentially be stochastic , i.e. , f ( x ) = Eξ [ F ( x , ξ ) ] . Such choices of f cover a wide range of problems in machine learning , and their study motivates a vast body of current optimization literature . A widely used ( and canonical ) approach for minimizing f is the ( stochastic ) gradient descent ( GD ) algorithm . Despite its simple form , GD often achieves superior empirical ( Wilson et al. , 2017 ) performances and theoretical ( Carmon et al. , 2017 ) guarantees . However , in many tasks such as reinforcement learning and natural language processing ( NLP ) , adaptive gradient methods ( e.g. , Adagrad ( Duchi et al. , 2011 ) , ADAM ( Kingma and Ba , 2014 ) , and RMSProp ( Tieleman and Hinton , 2012 ) ) outperform SGD . Despite their superior empirical performance , our understanding of the fast convergence of adaptive methods is limited . Previous analysis has shown that adaptive methods are more robust to variation in hyper-parameters ( Ward et al. , 2018 ) and adapt to sparse gradients ( Duchi et al. , 2011 ) ( a more detailed literature review is in Appendix A ) . However , in practice , the gradient updates are dense , and even after extensively tuning the SGD hyperparameters , it still converges much slower than adaptive methods in NLP tasks . We analyze the convergence of clipped gradient descent and provide an explanation for its fast convergence . Even though gradient clipping is a standard practice in tasks such as language models ( e.g . Merity et al. , 2018 ; Gehring et al. , 2017 ; Peters et al. , 2018 ) , it lacks a firm theoretical grounding . Goodfellow et al . ( 2016 ) ; Pascanu et al . ( 2013 ; 2012 ) discuss the gradient explosion problem in recurrent models and consider clipping as an intuitive work around . We formalize this intuition and prove that clipped GD can converge arbitrarily faster than fixed-step gradient descent . This result is shown to hold under a novel smoothness condition that is strictly weaker than the standard Lipschitzgradient assumption pervasive in the literature . Hence our analysis captures many functions that are not globally Lipschitz smooth . Importantly , the proposed smoothness condition is derived on the basis of extensive NLP training experiments , which are precisely the same type of experiments for which adaptive gradient methods empirically perform superior to gradient methods . By identifying a a new smoothness condition through experiments and then using it to analyze the convergence of adaptively-scaled methods , we reduce the following gap between theory and practice . On one hand , powerful techniques such as Nesterov ’ s momentum and variance reduction theoretically accelerate convex and nonconvex optimization . But , at least for now , they seem to have limited applicability in deep learning ( Defazio and Bottou , 2018 ) . On the other hand , some widely used techniques ( e.g. , heavy-ball momentum , adaptivity ) lack theoretical acceleration guarantees . We suspect that a major reason here is the misalignment of the theoretical assumptions with practice . Our work demonstrates that the concept of acceleration critically relies on the problem assumptions and that the standard global Lipschitz-gradient condition may not hold in the case of some applications and thus must be relaxed to admit a wider class of objective functions . 1.1 CONTRIBUTIONS . In light of the above background , the main contributions of this paper are the following : Inspired and supported by neural network training experiments , we introduce a new smoothness condition that allows the local smoothness constant to increase with the gradient norm . This condition is strictly weaker than the pervasive Lipschitz-gradient assumption . We provide a convergence rate for clipped GD under our smoothness assumption ( Theorem 3 ) . We prove an upper-bound ( Theorem 6 ) and a lower-bound ( Theorem 4 ) on the convergence rate of GD under our relaxed smoothness assumption . The lower-bound demonstrates that GD with fixed step size can be arbitrarily slower than clipped GD . We provide upper bounds for stochastic clipped GD ( Theorem 7 ) and SGD ( Theorem 8 ) . Again , stochastic clipped GD can be arbitrarily faster than SGD with a fixed step size . We support our proposed theory with realistic neural network experiments . First , in the state of art LSTM language modeling ( LM ) setting , we observe the function smoothness has a strong correlation with gradient norm ( see Figure 2 ) . This aligns with the known fact that gradient clipping accelerates LM more effectively compared to computer vision ( CV ) tasks . Second , our experiments in CV and LM demonstrate that clipping accelerates training error convergence and allows the training trajectory to cross non-smooth regions of the loss landscape . Furthermore , gradient clipping can also achieve good generalization performance even in image classification ( e.g. , 95.2 % test accuracy in 200 epochs for ResNet20 on Cifar10 ) . Please see Section 5 for more details . 2 A NEW RELAXED SMOOTHNESS CONDITION . In this section , we motivate and develop a relaxed smoothness condition that is weaker ( and thus , more general ) than the usual global Lipschitz smoothness assumption . We start with the traditional definition of smoothness . 2.1 FUNCTION SMOOTHNESS ( LIPSCHITZ GRADIENTS ) . Recall that f denotes the objective function that we want to minimize . We say that f is L-smooth if ‖∇f ( x ) −∇f ( y ) ‖ ≤ L‖x− y‖ , for all x , y ∈ Rd . ( 1 ) For twice differentiable functions , condition ( 1 ) is equivalent to ‖∇2f ( x ) ‖ ≤ L , ∀x ∈ Rd . This smoothness condition enables many important theoretical results . For example , Carmon et al . ( 2017 ) show that GD with h = 1/L is up to a constant optimal for optimizing smooth nonconvex functions . But the usual L-smoothness assumption ( 1 ) also has its limitations . Assuming existence of a global constant L that upper bounds the variation of the gradient is very restrictive . For example , simple polynomials such as f ( x ) = x3 break the assumption . One workaround is to assume that L exists in a compact region , and either prove that the iterates do not escape the region or run projectionbased algorithms . However , such assumptions can make L very large and slow down the theoretical convergence rate . In Section 4 , we will show that a slow rate is unavoidable for gradient descent with fixed step size , whereas clipped gradient descent can greatly improve the dependency on L. The above limitations force fixed-step gradient descent ( which is tailored for Lipschitz smooth functions ) to converge slowly in many tasks . In Figure 1 , we plot the estimated function smoothness at different iterations during training neural networks . We find that function smoothness varies greatly at different iterations . From Figure 1 , we further find that local smoothness positively correlates with the full gradient norm , especially in the language modeling experiment . A natural question is : Can we find a fine-grained smoothness condition under which we can design theoretically and empirically fast algorithms at the same time ? To answer this question , we introduce the relaxed smoothness condition in the next section , which is developed on the basis of extensive experiments— Figure 1 provides an illustrative example . 2.2 A NEW RELAXED SMOOTHNESS CONDITION . We observe strong positive correlation between function smoothness and gradient norm in language modeling experiments ( Figure 1 ( a ) ) . This observation leads us to propose the following smoothness condition that allows local smoothness to grow with function gradients . Definition 1 . A second order differentiable function f is ( L0 , L1 ) -smooth if ‖∇2f ( x ) ‖ ≤ L0 + L1‖∇f ( x ) ‖ . ( 2 ) Definition 1 strictly relaxes the usual ( and widely used ) L-smoothness . There are two ways to interpret the relaxation : First , when we focus on a compact region , we can balance the constants L0 and L1 such that L0 L while L1 L. Second , there exist functions that are ( L0 , L1 ) -smooth globally , but not L-smooth . Hence the constant L for L-smoothness gets larger as the compact set increases but L0 and L1 stay fixed . An example is given in Lemma 2 . Remark 1 . It is worth noting that we do not need the Hessian operator norm and gradient norm to necessarily satisfy the linear relation ( 2 ) . As long as these norms are positively correlated , gradient clipping can be shown to achieve faster rate than fixed step size gradient descent . We use the linear relationship ( 2 ) for simplicity of exposition . Lemma 2 . Let f be the univariate polynomial f ( x ) = ∑d i=1 aix i . When d ≥ 3 , then f is ( L0 , L1 ) smooth for some L0 and L1 but not L-smooth . Proof . The first claim follows from limx→∞ ∣∣∣ f ′ ( x ) f ′′ ( x ) ∣∣∣ = limx→−∞ ∣∣∣ f ′ ( x ) f ′′ ( x ) ∣∣∣ =∞ . The second claim follows by the unboundedness of f ′′ ( x ) . 2.3 SMOOTHNESS IN NEURAL NETWORKS . We saw that our smoothness condition relaxes the traditional smoothness assumption and is motivated empirically ( Figure 1 ) . Below we develop some intuition for this phenomenon . We conjecture that the proposed positive correlation results from the common components in expressions of the gradient and the Hessian . We illustrate the reasoning behind this conjecture by considering an ` - layer linear network with quadratic loss—a similar computation also holds for nonlinear networks . The L2 regression loss of a deep linear network is L ( Y , f ( X ) ) : = ‖Y − W ` · · ·W1X‖2 , where Y denotes labels , X denotes the input data matrix , and Wi denotes the weights in the ith layer . By ( Lemma 4.3 Kawaguchi , 2016 ) , we know that ∇vec ( wi ) L ( Y , f ( X ) ) = ( ( W ` · · ·Wi+1 ) ⊗ ( Wi−1 · · ·W2W1X ) T ) T vec ( f ( X ) − Y ) , where vec ( · ) flattens a matrix in Rm×n into a vector in Rmn ; ⊗ denotes the Kronecker product . For constants i , j such that ` ≥ j > i > 0 , the second order derivative ∇vec ( wj ) ∇vec ( wi ) L ( Y , f ( X ) ) = ( ( W ` · · ·Wi+1 ) ⊗ ( Wi−1 · · ·W2W1X ) T ) T ( ( W ` · · ·Wj+1 ) ⊗ ( Wj−1 · · ·W2W1X ) T ) + ( ( Wj−1 · · ·Wi+1 ) ⊗ ( Wi−1 · · ·W2W1X ) ) ( I ⊗ ( ( f ( X ) − Y ) W ` · · ·Wj+1 ) ) . When j = i , the second term equals 0 . Based on the above expressions , we notice that the gradient norm and Hessian norm may be positively correlated due to the following two observations . First , the gradient and the Hessian share many components such as the matrix product of weights across layers . Second , if one naively upper bounds the norm using Cauchy-Schwarz , then both upperbounds would be monotonically increasing with respect to ‖Wi‖ and ‖f ( X ) − Y ‖ .

Summary: This paper applies new assumption on smoothness that assume the norm of Hessian is bounded by a scalar plus norm of gradient. The traditional smoothness is only bounded with a scalar, the proposed assumption is more relaxed because now the norm of Hessian can grow with the norm of gradient. Under this assumption, the authors show clipped gradient converges faster than gradient for general nonconvex problems. The authors provide insights on why the proposed assumption is good for describing neural networks, and empirically verify the assumption with ResNet on CIFAR and LSTM on PTB.