Acutum: When Generalization Meets Adaptability

1 INTRODUCTION . With the rapid development of neural network architectures ( Goodfellow et al. , 2016 ) , training algorithms have attracted much more attention in the modern machine learning community . Due to the network size and data amount increasing dramatically , calculating the full gradient of data and implementing the full gradient descent ( GD ) become computationally expensive . Therefore , stochastic gradient descent ( SGD ) ( Robbins & Monro , 1951 ) becomes the most practical optimization method for training deep neural networks ( DNNs ) . In each iteration , SGD samples mini-batch data and computes the gradient corresponding to the mini-batch . Although SGD is computationally affordable , it needs a mechanism of fine-tune the learning rate , e.g. , linear decay or exponential decay , to converge efficiently . In fact , it is pretty brittle to tune the learning rate dynamically with SGD . To release the learning rate tuning burden of SGD and accelerate its convergence , several adaptive variants of SGD were proposed , including Adagrad ( Duchi et al. , 2011 ) , Adadelta ( Zeiler , 2012 ) , RMSProp ( Hinton et al. , 2012 ) , Adam ( Kingma & Ba , 2014 ) , etc . The descent direction is elementwise automatically adapted by the first moment and the second moment with an exponential average of gradients . Such a descent direction has achieved great improvements in practice , while the fundamental interpretation is still unclear . Besides , the theoretical regret analysis of these online learning algorithms has become completed gradually in convex objective settings . Due to the fast decay of exponential moving average , Adam can not converge even in simple convex cases as shown in ( Reddi et al. , 2019 ) . Amsgrad ( Reddi et al. , 2019 ) addressed this issue by keeping an extra non-decreasing sequence to buffer the fast decay . On the other hand , although adaptive methods are easy to use and fast to converge , the generalization results can not be as good as SGD ( Wilson et al. , 2017 ) . Various works ( Chen & Gu , 2018 ; Luo et al. , 2019 ) have been proposed to make algorithms not only converge faster but also has good generalization . However , these algorithms still inherit the second moment following Adam . Different from existing works , we provide a novel view for this problem by revisiting adaptive algorithms Through the view of direction angle between the descent direction and the momentum , we are able to demonstrate a fresh understanding of momentum . In summary , we propose a new algorithm , called acute adaptive momentum ( Acutum ) , to find adaptive gradients with only the first moment . To remove the second moments safely , we investigate their properties by reformulating the proximal subproblem of Adagrad . The main observation is that second moments essentially penalize the projection of the current descent direction on the exponential moving average of previous gradients . The new intuition allows us to remove second moments in Adam without any loss of performance . To the best of our knowledge , it is the first time to remove the second moments for adaptive gradient methods . As shown by our experiment , the proposed algorithm ( Acutum ) outperforms Adam/Amsgrad in wall-clock time while almost preserving the generalization ability of SGD . In particular , we make the following contributions : • We propose a new optimization algorithm called Acutum , which doesn ’ t need to compute the second moments . To the best of our knowledge , it is the first time to obtain the adaptive gradients with only the first moments . The intuition behind our algorithm is to remove second moments by attaching their purpose to the first moment through Adagrad ’ s proximal subproblem inspiration . • We provide a regret convergence analysis of Acutum on the convex setting , based on the analysis of ( Kingma & Ba , 2014 ; Reddi et al. , 2019 ; Chen & Gu , 2018 ) , and prove a datadependent regret bound , complementing our intuition . • We also provide various experiments about our proposed Acutum on training modern deep models . The experimental results empirically show that Acutum takes the best from both Adam-type methods and SGD . Notations . In the rest of this paper , we assume any decision variables θ ∈ Rd where d denotes the dimension of parameters . We denote the l2 norm of a given vector θ by ‖θ‖ = √∑d i=1 θ 2 i . With slightly abuse of notation , we write arithmetic symbols as element-wise operations for vectors , e.g. , a2 = [ a21 , a 2 2 , . . . ] T , a/b = [ a1/b1 , a2/b2 , . . . ] T . We denote by bxc the greatest integer less than or equal to the real number x . Given any integers x , y , where y 6= 0 , we denote by x ( mod y ) the remainder of the Euclidean division of x by y . In the optimization setting , we denote by fi ( · ) the loss function when we feed the i-th training data into our model , we also denote fS ( · ) when we feed a collection of samples , i.e. , fS ( · ) : = ∑ i∈S fi ( · ) . Given any point θ ∈ Rd and any convex set X ⊆ Rd , we denote by ΠX ( θ ) the projection of θ onto X , i.e. , arg minµ∈X ‖µ− θ‖ . 2 RELATED WORK . The increasing size of deep neural networks and the amount of train data have dramatically induced the difficulty of training neural networks . The huge parameter size makes the landscape of neural networks more non-convex , and the amount of train data requires more computation resources to calculate the gradient . Therefore , all the facts urge researchers to design faster optimization algorithms for deep neural networks , while maintaining a reasonable generalization ability . SGD-momentum is widely used in training large-scale neural networks while the excellent generalization ability of SGD is vulnerable to its learning rate . Meanwhile , researchers began to focus on the design of adaptive gradient methods for a fast and simple optimizer . Adagrad proposed in ( Duchi et al. , 2011 ) introduced the second moment to obtain a self-adaptive learning rate , and hence freed researchers ’ hands of parameter tuning . The update rules of Adagrad can be formulated as follows : θt+1 = θt − αt gt√ vt , where gt denotes the stochastic gradient , vt is the accumulation of gradient ’ s second moments , i.e. , vt = 1 t ∑t τ=1 g 2 τ , and αt is the decreasing learning rate with αt = O ( 1/ √ t ) . Theoretically , Ada- grad improves the convergence of regret from O ( √ d/ √ T ) to O ( 1/ √ T ) for the convex objectives with sparse gradients . However , in practice , people realize that adaptive gradient of Adagrad , i.e. , gt/ √ vt goes to zero very quickly due to vt accumulating to large number quickly as the algorithm proceeds . To make it through , RMSProp ( Hinton et al. , 2012 ) introduced the exponential decay in the second moment to control the accumulation speed of second moment in Adagrad . Furthermore , ( Kingma & Ba , 2014 ) incorporated the momentum into RMSProp and proposed Adam . The detailed procedure of Adam can be formulated as : θt+1 = θt − αt mt√ vt , ( 1 ) where mt is the exponential decay of momentum , i.e. , mt = ∑t τ=1 ( 1 − β1 ) β t−τ 1 gτ , and vt is the exponential decay of second moment , i.e. , vt = ∑t τ=1 ( 1− β2 ) β t−τ 2 g 2 τ . The faster convergence , robust hyper-parameters and good performance on CV and NLP ’ s tasks make Adam become one of the most successful optimizers these years . However , ( Reddi et al. , 2019 ) showed Adam does not converge in certain convex cases , and proposed Amsgrad to correct the direction of Adam with imposing an increasing sequence of the second moment . Although convergence , in theory , is fixed and the training speed is still fast , the generalization ability of adaptive algorithms is still worse than SGD with momentum . Thus , there are a lot of studies to improve the generalization performance of Adam types algorithms . Almost all of these works try to make some connections between Adam and SGD-momentum such that the proposed algorithm converges faster like Adam while generalizing better as SGD . For example , ( Zhang et al. , 2017 ) proposed a normalized direction-preserving Adam ( ND-Adam ) . To make the direction of the gradients preserved , the authors adjusted the learning rate to each weight vector as a whole , instead of each individual weight . It could be thought as an adjustment of learning rate dynamically in each iteration , instead of the learning rate decay . ( Loshchilov & Hutter , 2017 ) proposed to modify Adam to decay all weights the same regardless of the gradient variances . ( Keskar & Socher , 2017 ) proposed to improve the generalization performance by investigating a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate . The experimental results show that it obtains a better generalization with the convergence speed sacrificing . With a similar insight , ( Luo et al. , 2019 ) introduced the element-wise clipping technique to the second moment and proposed AdaBound , which allows the algorithm behavior switched from a clipped Adam to SGD-momentum as the algorithm proceeds . Besides , ( Chen & Gu , 2018 ) proposed Padam which tried to find the balanced state between Adam and SGD-momentum through tuning the order of the second moment in the update paradigm . All the aforementioned methods try to find the connection between Adam and SGD-momentum , thus all these algorithms take the second moment adaptation as a grant . The difference between previous works and ours is that we suggest to remove the second moment and propose an angle based algorithm , by revisiting the original idea of second moment adaptation . Such intuition could be of independent interest in the field of the optimization community . 3 THE ACUTE ADAPTIVE MOMENTUM ALGORITHM . In this section , we first revisit existing adaptive algorithms from a novel point of view , then introduce the intuition and provide the algorithm procedure of our method Acutum . The performance guarantees established in Section 4 and evaluation results in Section 5 validate our insights . From the above section , one may notice that almost all the adaptive methods utilize second moments to adjust the individual magnitude of the updates . However , the efficient calculation of second moments , see Equation ( 1 ) , is only proposed in Adagrad ( Duchi et al. , 2011 ) . Let ’ s take a look at the original update paradigm of Adagrad : θt+1 = θt − αtG−1/2t gt , ( 2 ) where Gt = ∑t τ=1 gτg T τ , and gτ denotes the stochastic gradient calculated at iteration τ . Recall that in each iteration , SGD seeks the optimal solution of a subproblem constructed from a quadratic approximation of the objective . Similarly , we can write the update procedure of Adagrad in Equation ( 2 ) as θt+1 = arg min θ ( θ − θt ) T gt︸ ︷︷ ︸ T1 + 1 2αt ( θ − θt ) T G1/2t ( θ − θt ) ︸ ︷︷ ︸ T2 ( 3 ) where the first-order penalty T1 leads the parameter θt to the opposite direction to the current gradient gt . Note that Adagrad can adaptively adjust each individual dimension of θt through introducing Gt in the quadratic term T2 , while SGD can not . If we substitute the definition of Gt back into Equation ( 3 ) , T2 can be formulated as : T2 = 1 2αt ( θ − θt ) T G1/2t ( θ − θt ) = 1 2αt ( θ − θt ) T ( t∑ τ=1 gτg T τ ) 1/2 ( θ − θt ) . ( 4 ) By Young ’ s inequality ( Young , 1912 ) , we can obtain an upper bound of T2 as follows : T2 ≤ 1 8αt ‖θ − θt‖2︸ ︷︷ ︸ T3 + 1 2αt t∑ τ=1 ∥∥∥ ( θ − θt ) T gτ∥∥∥2︸ ︷︷ ︸ T4 . ( 5 ) Combining Equation ( 5 ) and Equation ( 3 ) , we can approximate the update rule of Adagrad as follows : θt+1 ≈ arg min θ ( θ − θt ) T gt︸ ︷︷ ︸ T1 + 1 8αt ‖θ − θt‖2︸ ︷︷ ︸ T3 + 1 2αt t∑ τ=1 ∥∥∥ ( θ − θt ) T gτ∥∥∥2︸ ︷︷ ︸ T4 . We can think the upper bound of the Adagrad subproblem as the combination of the standard first and second order penalty as T1 and T3 , with a penalty for the projections of current descent θ − θt on the previous gradients gτ , i.e. , T4 . With these observations in mind , a natural question comes out : how can we understand T4 in neural network training tasks or in general online learning tasks ? Before answering this question , we first describe the practical usage of training samples . The training procedure of DNNs can be stated as follows . The algorithm randomly shuffles the whole training dataset A and partitions A into mini-batch of equal size { A0 , A1 , . . . , Ap−1 } , then the algorithm is fed with the sample in a fixed order , say , first A0 , then A1 , and so on . The whole procedure repeats after each pass of the whole dataset . To make our description more clear , we take the first pass as an example . Let ∇fAt ( θt ) be the gradient calculated on iteration t by using the sample in subset At , where 0 ≤ t < p due to the first pass . Note that the loss function is the sum of whole samples , e.g. , 1n ∑n i=1 fi ( θ ) , if we utilize ∇fAi ( θt ) to directly update parameters like SGD , e.g. , θt+1 = θt − α∇fAt ( θt ) , the batch loss∑ i∈At fi ( θ ) will decrease since it aligns with the opposite direction of its gradient , e.g. , ( θt+1 − θt ) T∇fAt ( θt ) = −‖∇fAt ( θt ) ‖2 < 0 . However , for loss corresponding to the sample not in At , it is highly possible that ( θt+1 − θt ) T∇fAi ( θt ) > 0 . On other words , only using −∇fAi ( θt ) as update direction will decrease the loss function corresponding to At but increase the total loss function except At . Ideally , if there exists a direction θt+1−θt which is orthogonal to∇fAi ( θt ) for any i 6= t ( mod p ) , and forms an acute angle with ∇fAt ( θt ) , then θt+1 can guarantee a sufficient descent for the loss function corresponding to At while not increase the loss function outside At . Then a natural update of θt+1 to achieve the above goal can be obtained by solving the following sub-problem : arg min θ c2 ‖θ − θt‖2 + c1 ( θ − θt ) T gt s.t . ( θ − θt ) T ∇fAi ( θt ) = 0 , i 6= t ( mod p ) , ( 6 ) In practice , however , the computational cost of computing ∇fAi ( θt ) for all i is expensive in each iteration . As a natural solution , we can approximate ∇fAi ( θt ) with previous gradients fAi ( θi ) , which is exactly gi calculated in Adagrad in iteration i before iteration t , where i 6= t ( mod p ) . Hence a soft margin variant of Equation ( 6 ) can be formulated as θt+1 = arg min θ c2,0 ‖θ − θt‖2 + c2,1 t∑ τ=1 ∥∥∥ ( θ − θt ) T gτ∥∥∥2 + c1 ( θ − θt ) T gt , ( 7 ) which is exactly the formulation in Equation ( 5 ) and Equation ( 3 ) ! Thus , one can reasonably consider that Adagrad decreases the cumulative loss by guaranteeing a descent on the current batch loss while does not increase previous batch loss . Although the approximation of ∇fAi ( θt ) using previous gradients gi in Adagrad performs well in practice , there is still a large accuracy gap . For example , let gt−1 and gt−2 denote the gradients ∇fAi−1 ( θt−1 ) and ∇fAi−2 ( θt−2 ) , respectively . It is often the case that the approximation error between∇fAi−1 ( θt ) and gt−1 is smaller than that between∇fAi−2 ( θt ) and gt−2 , assuming the objective is local smoothness1 . In other words , the approximation accuracy of gτ to ∇fAi ( θt ) decays as τ decreases . Inspired by this observation , we substitute the arithmetic average in Equation ( 7 ) to exponential moving average and reassign the coefficients of gτ like Adam/momentum methods , then Equation ( 7 ) becomes θt+1 = arg min θ c2,0 ‖θ − θt‖2 + c2,1 t∑ τ=1 ∥∥∥ ( 1− β ) βt−τ ( θ − θt ) T gτ∥∥∥2 + c1 ( θ − θt ) T gt . This is what RMSProp actually does . Therefore , we obtain the same optimization intuition of RMSProp as Adagrad but add weight of the approximate gradient according to its approximation accuracy . Enlightened by the above analysis , we are ready to propose a new adaptive momentum method . Let mt denote the approximated gradient of the previous batch loss at current parameters . With the following observation : ( i ) if θt − θt+1 forms acute angles with both gt and mt , rather than penalizing the projection like what Adagrad and RMSProp do , we can obtain both descent on current and previous batches ; ( ii ) to handle the case when the estimation of previous gradients using mt is not accurate , we expect the current gradient gt to dominate the descent direction , which is denoted as m̂t . It can be guaranteed by requiring the projection length of m̂t on gt is less than the length of gt if properly regularizing mt . Then , we propose a new iterative subproblem in the following : θt+1 = arg min θ c2,0 ‖θ − θt‖2 + c1 ( θ − θt ) T ( gt + ‖gt‖ m̂t ‖m̂t‖ ) , where m̂t+1 = gt + ‖gt‖ m̂t ‖m̂t‖ . ( 8 ) With such formulation , the following properties hold : • We ensure the above two observations , specifically , with the update of momentum m̂t , we can guarantee that ‖m̂t+1‖ = O ( ‖gt‖ ) . • We attach the properties of the second moment to the first-order operation , inspired by Adagrad proximal subproblem . The proposed algorithm Acutum is summarized in Algorithm 1 . Algorithm 1 Acute Adaptive Momentum method ( Acutum ) 1 : Input : initial point θ0 ∈ X ; step size { αt } , momentum parameters { βt } 2 : set m̂0 = 0 3 : for t = 1 to T do 4 : gt = ∇ft ( θt ) 5 : m̂t = gt + β1 , t ‖gt‖ / ( ‖m̂t−1‖+ ) · m̂t−1 6 : θt+1 = ΠX ( θt − αt · m̂t ) 7 : end for 8 : Return : xT+1 Note that , at step 5 we use to prevent precision overflow caused by a tiny ‖m̂t−1‖ . β1 , t is the momentum weight . We specify its value in the theoretical analysis in Section 4 . In practice , we can simply set β1 , t to be 1 . Compared with other adaptive gradient methods , Acutum has fewer hyper-parameters , and is much easier to tune deep models . Besides , it is also computation-friendly ( for both time and space complexities ) since the algorithm does not need second moments . 1The local smoothness assumption is widely used in the convergence analysis for convex and non-convex objectives , see ( Nesterov , 1998 ; Johnson & Zhang , 2013 ) .

Summary: This paper proposes new stochastic optimization methods to achieve a fast convergence of adaptive SGD and preserve the generalization ability of SGD. The idea is to let the search direction opposite to the gradient at the current batch of examples and a bit orthogonal to previous batch of examples. The algorithm is easy to implement and backed up with regret bounds. Several experimental results are also reported to verify the effectiveness of the algorithm.