SAdam: A Variant of Adam for Strongly Convex Functions

vexity can be utilized to further improve the performance remains an open problem . In this paper , we give an affirmative answer by developing a variant of Adam ( referred to as SAdam ) which achieves a data-dependent O ( log T ) regret bound for strongly convex functions . The essential idea is to maintain a faster decaying yet under controlled step size for exploiting strong convexity . In addition , under a special configuration of hyperparameters , our SAdam reduces to SC-RMSprop , a recently proposed variant of RMSprop for strongly convex functions , for which we provide the first data-dependent logarithmic regret bound . Empirical results on optimizing strongly convex functions and training deep networks demonstrate the effectiveness of our method . 1 INTRODUCTION . Online Convex Optimization ( OCO ) is a well-established learning framework which has both theoretical and practical appeals ( Shalev-Shwartz et al. , 2012 ) . It is performed in a sequence of consecutive rounds : In each round t , firstly a learner chooses a decision xt from a convex set D ⊆ Rd , at the same time , an adversary reveals a loss function ft ( · ) : D 7→ R , and consequently the learner suffers a loss ft ( xt ) . The goal is to minimize regret , defined as the difference between the cumulative loss of the learner and that of the best decision in hindsight ( Hazan et al. , 2016 ) : R ( T ) : = T∑ t=1 ft ( xt ) −min x∈D T∑ t=1 ft ( x ) . The most classic algorithm for OCO is Online Gradient Descent ( OGD ) ( Zinkevich , 2003 ) , which attains an O ( √ T ) regret . OGD iteratively performs descent step towards gradient direction with a predetermined step size , which is oblivious to the characteristics of the data being observed . As a result , its regret bound is data-independent , and can not benefit from the structure of data . To address this limitation , various of adaptive gradient methods , such as Adagrad ( Duchi et al. , 2011 ) , RMSprop ( Tieleman & Hinton , 2012 ) and Adadelta ( Zeiler , 2012 ) have been proposed to exploit the geometry of historical data . Among them , Adam ( Kingma & Ba , 2015 ) , which dynamically adjusts the step size and the update direction by exponential average of the past gradients , has been extensively popular and successfully applied to many applications ( Xu et al. , 2015 ; Gregor et al. , 2015 ; Kiros et al. , 2015 ; Denkowski & Neubig , 2017 ; Bahar et al. , 2017 ) . Despite the outstanding performance , Reddi et al . ( 2018 ) pointed out that Adam suffers the non-convergence issue , and developed two modified versions , namely AMSgrad and AdamNC . These variants are equipped with data-dependent regret bounds , which are O ( √ T ) in the worst case and become tighter when gradients are sparse 1 . ∗Lijun Zhang is the corresponding author . 1We note that , as very recently pointed out by Tran et al . ( 2019 ) , there still exists a minor theoretical flaw in the analysis of Reddi et al . ( 2018 ) , and such issue in fact appears in many of recent variants of AMSgrad/AdamNC . In this paper , we provide a simple way to fix this problem . The details can be found in Appendix G. While the theoretical behavior of Adam in convex cases becomes clear , it remains an open problem whether strong convexity can be exploited to achieve better performance . Such property arises , for instance , in support vector machines as well as other regularized learning problems , and it is wellknown that the vanilla OGD with an appropriately chosen step size enjoys a much better O ( log T ) regret bound for strongly convex functions ( Hazan et al. , 2007 ) . In this paper , we propose a variant of Adam adapted to strongly convex functions , referred to as SAdam . Our algorithm follows the general framework of Adam , yet keeping a faster decaying step size controlled by time-variant heperparameters to exploit strong convexity . Theoretical analysis demonstrates that SAdam achieves a data-dependent O ( log T ) regret bound for strongly convex functions , which means that it converges faster than AMSgrad and AdamNC in such cases , and also enjoys a huge gain in the face of sparse gradients . Furthermore , under a special configuration of heperparameters , the proposed algorithm reduces to the SC-RMSprop ( Mukkamala & Hein , 2017 ) , which is a variant of RMSprop algorithm for strongly convex functions . We provide an alternative proof for SC-RMSprop , and establish the first data-dependent logarithmic regret bound . Finally , we evaluate the proposed algorithm on strongly convex problems as well as deep networks , and the empirical results demonstrate the effectiveness of our method . Notation . Throughout the paper , we use lower case bold face letters to denote vectors , lower case letters to denote scalars , and upper case letters to denote matrices . We use ‖ · ‖ to denote the ` 2-norm and ‖ · ‖∞ the infinite norm . For a positive definite matrix H ∈ Rd×d , the weighted ` 2-norm is defined by ‖x‖2H = x > Hx . The H-weighted projection ΠHD ( x ) of x onto D is defined by ΠHD ( x ) = arg miny∈D ‖y− x‖2H . We use gt to denote the gradient of ft ( · ) at xt . For vector sequence { vt } Tt=1 , we denote the i-th element of vt by vt , i . For diagonal matrix sequence { Mt } Tt=1 , we use mt , i to denote the i-th element in the diagonal of Mt . We use g1 : t , i = [ g1 , i , · · · , gt , i ] to denote the vector obtained by concatenating the i-th element of the gradient sequence { gt } Tt=1 . 2 RELATED WORK . In this section , we briefly review related work in online convex and strongly convex optimization . In the literature , most studies are devoted to the minimization of regret for convex functions . Under the assumptions that the infinite norm of gradients and the diameter of the decision set are bounded , OGD with step size on the order of O ( 1/ √ t ) ( referred to as convex OGD ) achieves a data-independent O ( d √ T ) regret ( Zinkevich , 2003 ) , where d is the dimension . To conduct more informative updates , Duchi et al . ( 2011 ) introduce Adagrad algorithm , which adjusts the step size of OGD in a perdimension basis according to the geometry of the past gradients . In particular , the diagonal version of the algorithm updates decisions as xt+1 = xt − α√ t Vt −1/2gt ( 1 ) where α > 0 is a constant factor , Vt is a d× d diagonal matrix , and ∀i ∈ [ d ] , vt , i = ( ∑t j=1 g 2 j , i ) /t is the arithmetic average of the square of the i-th elements of the past gradients . Intuitively , while the step size of Adagrad , i.e. , ( α/ √ t ) V −1/2 t , decreases generally on the order of O ( 1/ √ t ) as that in convex OGD , the additional matrix V −1/2t will automatically increase step sizes for sparse dimensions in order to seize the infrequent yet valuable information therein . For convex functions , Adagrad enjoys an O ( ∑d i=1 ‖g1 : T , i‖ ) regret , which is O ( d √ T ) in the worst case and becomes tighter when gradients are sparse . Although Adagrad works well in sparse cases , its performance has been found to deteriorate when gradients are dense due to the rapid decay of the step size since it uses all the past gradients in the update ( Zeiler , 2012 ) . To tackle this issue , Tieleman & Hinton ( 2012 ) propose RMSprop , which alters the arithmetic average procedure with Exponential Moving Average ( EMA ) , i.e. , Vt = βVt−1 + ( 1− β ) diag ( gtg > t ) where β ∈ [ 0 , 1 ] is a hyperparameter , and diag ( · ) denotes extracting the diagonal matrix . In this way , the weights assigned to past gradients decay exponentially so that the reliance of the update is essentially limited to recent few gradients . Since the invention of RMSprop , many EMA variants of Adagrad have been developed ( Zeiler , 2012 ; Kingma & Ba , 2015 ; Dozat , 2016 ) . One of the most popular algorithms is Adam ( Kingma & Ba , 2015 ) , where the first-order momentum acceleration , shown in ( 2 ) , is incorporated into RMSprop to boost the performance : ĝt = β1ĝt−1 + ( 1− β1 ) gt ( 2 ) Vt = β2Vt−1 + ( 1− β2 ) diag ( gtg > t ) ( 3 ) xt+1 = xt − α√ t Vt −1/2ĝt ( 4 ) While it has been successfully applied to various practical applications , a recent study by Reddi et al . ( 2018 ) shows that Adam could fail to converge to the optimal decision even in some simple one-dimensional convex scenarios due to the potential rapid fluctuation of the step size . To resolve this issue , they design two modified versions of Adam . The first one is AMSgrad , V̂t = max { V̂t−1 , Vt } xt+1 = xt − α√ t V̂ −1/2 t ĝt where an additional element-wise maximization procedure is employed before the update of xt to ensure a stable step size . The other is AdamNC , where the framework of Adam remains unchanged , yet a time-variant β2 ( i.e. , β2t ) is adopted to keep the step size under control . Theoretically , the two algorithms achieve data-dependentO ( √ T ∑d i=1 vT , i+ ∑d i=1 ‖g1 : T , i‖ log T ) andO ( √ T ∑d i=1 vT , i+∑d i=1 ‖g1 : T , i‖ ) regrets respectively . In the worst case , they suffer O ( d √ T log T ) and O ( d √ T ) regrets respectively , and enjoy a huge gain when data is sparse . Note that the aforementioned algorithms are mainly analysed in general convex settings and suffer at least O ( d √ T ) regret in the worst case . For online strongly convex optimization , the classical OGD with step size proportional to O ( 1/t ) ( referred to as strongly convex OGD ) achieves a dataindependent O ( d log T ) regret ( Hazan et al. , 2007 ) . Inspired by this , Mukkamala & Hein ( 2017 ) modify the update rule of Adagrad in ( 1 ) as follows xt+1 = xt − α t Vt −1gt so that the step size decays approximately on the order of O ( 1/t ) , which is similar to that in strongly convex OGD . The new algorithm , named SC-Adagrad , is proved to enjoy a data-dependent regret bound of O ( ∑d i=1 log ( ‖g1 : T , i‖2 ) ) , which is O ( d log T ) in the worst case . They further extend this idea to RMSprop , and propose an algorithm named SC-RMSprop . However , as pointed out in Section 3 , their regret bound for SC-RMSprop is in fact data-independent , and in this paper we provide the first data-dependent regret bound for this algorithm . Very recently , several modifications of Adam adapted to non-convex settings have been developed ( Chen et al. , 2019 ; Basu et al. , 2018 ; Zhang et al. , 2018 ; Shazeer & Stern , 2018 ) . However , to our knowledge , none of these algorithms are particularly designed for strongly convex functions , nor enjoy a logarithmic regret bound .

Summary: In the setting of online convex optimization, this paper investigates the question of whether adaptive gradient methods can achieve “data dependent” logarithmic regret bounds when the class of loss functions is strongly convex. To this end, the authors propose a variant of Adam - called SAdam - which indeed satisfies such a desired bound. Importantly, SAdam is an extension of SC-RMSprop (a variant of RMSprop) for which a “data independent” logarithmic bound was found. Experiments on optimizing strongly convex functions and training deep networks show that SAdam outperforms other adaptive gradient methods (and SGD).