AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods

1 INTRODUCTION . First-order optimization algorithms with adaptive learning rate play an important role in deep learning due to their efficiency in solving large-scale optimization problems . Denote gt ∈ Rn as the gradient of loss function f with respect to its parameters θ ∈ Rn at timestep t , then the general updating rule of these algorithms can be written as follows ( Reddi et al. , 2018 ) : θt+1 = θt − αt√ vt mt . ( 1 ) In the above equation , mt , φ ( g1 , . . . , gt ) ∈ Rn is a function of the historical gradients ; vt , ψ ( g1 , . . . , gt ) ∈ Rn+ is an n-dimension vector with non-negative elements , which adapts the learning rate for the n elements in gt respectively ; αt is the base learning rate ; and αt√vt is the adaptive step size for mt . One common choice of φ ( g1 , . . . , gt ) is the exponential moving average of the gradients used in Momentum ( Qian , 1999 ) and Adam ( Kingma & Ba , 2014 ) , which helps alleviate gradient oscillations . The commonly-used ψ ( g1 , . . . , gt ) in deep learning community is the exponential moving average of squared gradients , such as Adadelta ( Zeiler , 2012 ) , RMSProp ( Tieleman & Hinton , 2012 ) , Adam ( Kingma & Ba , 2014 ) and Nadam ( Dozat , 2016 ) . Adam ( Kingma & Ba , 2014 ) is a typical adaptive learning rate method , which assembles the idea of using exponential moving average of first and second moments and bias correction . In general , Adam is robust and efficient in both dense and sparse gradient cases , and is popular in deep learning research . However , Adam is shown not being able to converge to optimal solution in certain cases . Reddi et al . ( 2018 ) point out that the key issue in the convergence proof of Adam lies in the quantity Γt , ( √vt αt − √ vt−1 αt−1 ) , ( 2 ) ∗Equally contributed . which is assumed to be positive , but unfortunately , such an assumption does not always hold in Adam . They provide a set of counterexamples and demonstrate that the violation of positiveness of Γt will lead to undesirable convergence behavior in Adam . Reddi et al . ( 2018 ) then propose two variants , AMSGrad and AdamNC , to address the issue by keeping Γt positive . Specifically , AMSGrad defines v̂t as the historical maximum of vt , i.e. , v̂t = max { vi } ti=1 , and replaces vt with v̂t to keep vt non-decreasing and therefore forces Γt to be positive ; while AdamNC forces vt to have “ long-term memory ” of past gradients and calculates vt as their average to make it stable . Though these two algorithms solve the non-convergence problem of Adam to a certain extent , they turn out to be inefficient in practice : they have to maintain a very large vt once a large gradient appears , and a large vt decreases the adaptive learning rate αt√vt and slows down the training process . In this paper , we provide a new insight into adaptive learning rate methods , which brings a new perspective on solving the non-convergence issue of Adam . Specifically , in Section 3 , we study the counterexamples provided by Reddi et al . ( 2018 ) via analyzing the accumulated step size of each gradient gt . We observe that in the common adaptive learning rate methods , a large gradient tends to have a relatively small step size , while a small gradient is likely to have a relatively large step size . We show that the unbalanced step sizes stem from the inappropriate positive correlation between vt and gt , and we argue that this is the fundamental cause of the non-convergence issue of Adam . In Section 4 , we further prove that decorrelating vt and gt leads to equal and unbiased expected step size for each gradient , thus solving the non-convergence issue of Adam . We subsequently propose AdaShift , a decorrelated variant of adaptive learning rate methods , which achieves decorrelation between vt and gt by calculating vt using temporally shifted gradients . Finally , in Section 5 , we study the performance of our proposed AdaShift , and demonstrate that it solves the non-convergence issue of Adam , while still maintaining a decent performance compared with Adam in terms of both training speed and generalization . 2 PRELIMINARIES . Adam . In Adam , mt and vt are defined as the exponential moving average of gt and g2t : mt = β1mt−1 + ( 1− β1 ) gt and vt = β2vt−1 + ( 1− β2 ) g2t , ( 3 ) where β1 ∈ [ 0 , 1 ) and β2 ∈ [ 0 , 1 ) are the exponential decay rates for mt and vt , respectively , with m0 = 0 and v0 = 0 . They can also be written as : mt = ( 1− β1 ) t∑ i=1 βt−i1 gi and vt = ( 1− β2 ) t∑ i=1 βt−i2 g 2 i . ( 4 ) To avoid the bias in the estimation of the expected value at the initial timesteps , Kingma & Ba ( 2014 ) propose to apply bias correction to mt and vt . Using mt as instance , it works as follows : mt = ( 1− β1 ) ∑t i=1 β t−i 1 gi ( 1− β1 ) ∑t i=1 β t−i 1 = ∑t i=1 β t−i 1 gi∑t i=1 β t−i 1 = ( 1− β1 ) ∑t i=1 β t−i 1 gi 1− βt1 . ( 5 ) Online optimization problem . An online optimization problem consists of a sequence of cost functions f1 ( θ ) , . . . , ft ( θ ) , . . . , fT ( θ ) , where the optimizer predicts the parameter θt at each timestep t and evaluate it on an unknown cost function ft ( θ ) . The performance of the optimizer is usually evaluated by regret R ( T ) , ∑T t=1 [ ft ( θt ) − ft ( θ∗ ) ] , which is the sum of the difference between the online prediction ft ( θt ) and the best fixed-point parameter prediction ft ( θ∗ ) for all the previous steps , where θ∗ = arg minθ∈ϑ ∑T t=1 ft ( θ ) is the best fixed-point parameter from a feasible set ϑ. Counterexamples . Reddi et al . ( 2018 ) highlight that for any fixed β1 and β2 , there exists an online optimization problem where Adam has non-zero average regret , i.e. , Adam does not converge to optimal solution . The counterexamples in the sequential version are given as follows : ft ( θ ) = { Cθ , if t mod d = 1 ; −θ , otherwise , ( 6 ) where C is a relatively large constant and d is the length of an epoch . In Equation 6 , most gradients of ft ( θ ) with respect to θ are −1 , but the large positive gradient C at the beginning of each epoch makes the overall gradient of each epoch positive , which means that one should decrease θt to minimize the loss . However , according to ( Reddi et al. , 2018 ) , the accumulated update of θ in Adam under some circumstance is opposite ( i.e. , θt is increased ) , thus Adam can not converge in such case . Reddi et al . ( 2018 ) argue that the reason of the non-convergence of Adam lies in that the positive assumption of Γt , ( √ vt/αt − √ vt−1/αt−1 ) does not always hold in Adam . The counterexamples are also extended to stochastic cases in ( Reddi et al. , 2018 ) , where a finite set of cost functions appear in a stochastic order . Compared with sequential online optimization counterexample , the stochastic version is more general and closer to the practical situation . For the simplest one dimensional case , at each timestep t , the function ft ( θ ) is chosen as i.i.d . : ft ( θ ) = { Cθ , with probability p = 1+δC+1 ; −θ , with probability 1− p = C−δC+1 , ( 7 ) where δ is a small positive constant that is smaller than C. The expected cost function of the above problem is F ( θ ) = 1+δC+1Cθ − C−δ C+1θ = δθ , therefore , one should decrease θ to minimize the loss . Reddi et al . ( 2018 ) prove that when C is large enough , the expectation of accumulated parameter update in Adam is positive and results in increasing θ . Basic Solutions Reddi et al . ( 2018 ) propose maintaining the strict positiveness of Γt as solution , for example , keeping vt non-decreasing or using increasing β2 . In fact , keeping Γt positive is not the only way to guarantee the convergence of Adam . Another important observation is that for any fixed sequential online optimization problem with infinitely repeating epochs ( e.g. , Equation 6 ) , Adam will converge as long as β1 is large enough . Formally , we have the following theorem : Theorem 1 ( The influence of β1 ) . For any fixed sequential online convex optimization problem with infinitely repeating of finite length epochs ( d is the length of an epoch ) , if ∃G ∈ R such that ‖∇ft ( θ ) ‖∞ ≤ G and ∃T ∈ N , ∃ 2 > 1 > 0 such that 1 < αt√vtG 2 < 2 holds for all t > T , then , for any fixed β2 ∈ [ 0 , 1 ) , there exists a β1 ∈ [ 0 , 1 ) such that Adam has average regret ≤ 2 ; The intuition behind Theorem 1 is that , if β1 → 1 , then mt → ∑d i=1 gi/d , i.e. , mt approaches the average gradient of an epoch , according to Equation 5 . Therefore , no matter what the adaptive learning rate αt/ √ vt is , Adam will always converge along the correct direction . 3 THE CAUSE OF NON-CONVERGENCE : UNBALANCED STEP SIZE . In this section , we study the non-convergence issue by analyzing the counterexamples provided by Reddi et al . ( 2018 ) . We show that the fundamental problem of common adaptive learning rate methods is that : vt is positively correlated to the scale of gradient gt , which results in a small step size αt/ √ vt for a large gradient , and a large step size for a small gradient . We argue that such an unbalanced step size is the cause of non-convergence . We will first define net update factor for the analysis of the accumulated influence of each gradient gt , then apply the net update factor to study the behaviors of Adam using Equation 6 as an example . The argument will be extended to the stochastic online optimization problem and general cases . 3.1 NET UPDATE FACTOR . When β1 6= 0 , due to the exponential moving effect of mt , the influence of gt exists in all of its following timesteps . For timestep i ( i ≥ t ) , the weight of gt is ( 1− β1 ) βi−t1 . We accordingly define a new tool for our analysis : the net update net ( gt ) of each gradient gt , which is its accumulated influence on the entire optimization process : net ( gt ) , ∞∑ i=t αi√ vi [ ( 1− β1 ) βi−t1 gt ] = k ( gt ) · gt , where k ( gt ) = ∞∑ i=t αi√ vi ( 1− β1 ) βi−t1 , ( 8 ) and we call k ( gt ) the net update factor of gt , which is the equivalent accumulated step size for gradient gt . Note that k ( gt ) depends on { vi } ∞i=t , and in Adam , if β1 6= 0 , then all elements in { vi } ∞i=t are related to gt . Therefore , k ( gt ) is a function of gt . It is worth noticing that in Momentum method , vt is equivalently set as 1 . Therefore , we have k ( gt ) = αt and net ( gt ) = αtgt , which means that the accumulated influence of each gradient gt in Momentum is the same as vanilla SGD ( Stochastic Gradient Decent ) . Hence , the convergence of Momentum is similar to vanilla SGD . However , in adaptive learning rate methods , vt is function over the past gradients , which makes its convergence nontrivial .

Summary: Based on an extensive argument acoordig to which Adam potential failures are due to the positive correlation between gradient and moment estimation, the authors propose Adashift, a method in which temporal shift (and more surprisingly 'spatial' shift, ie mixing of parameters) is used to ensure that moment estimation is less correlated with gradient, ensuring convergence of Adashift in pathological cases, without the efficiency cost of simpler method such as AMSGrad. An extensive analysis of a pathological counter example, introduced in Reddi et al. 2018 is analysed, before the algorithm presentation and experimental validation. Experiments shows that the algorithm has equivalent speed as Adam and sometimes false local minima, resulting in better training error, and potentially better test error.