Optimistic Adaptive Acceleration for Optimization

1 INTRODUCTION . Nowadays deep learning has been very successful in numerous applications , from robotics ( e.g. , Levine et al . ( 2017 ) ) , computer vision ( e.g. , He et al . ( 2016 ) ; Goodfellow et al . ( 2014 ) ) , reinforcement learning ( e.g. , Mnih et al . ( 2013 ) ) , to natural language processing ( e.g. , Graves et al . ( 2013 ) ) . A common goal in these applications is learning quickly . It becomes a desired goal due to the presence of big data and/or the use of large neural nets . To accelerate the process , there are variety of training algorithms proposed in recent years , such as AMSGRAD ( Reddi et al . ( 2018 ) ) , ADAM ( Kingma & Ba ( 2015 ) ) , RMSPROP ( Tieleman & Hinton ( 2012 ) ) , ADADELTA ( Zeiler ( 2012 ) ) , and NADAM ( Dozat ( 2016 ) ) , etc . All the prevalent algorithms for training deep nets mentioned above combine two ideas : the idea of adaptivity from ADAGRAD ( Duchi et al . ( 2011 ) ; McMahan & Streeter ( 2010 ) ) and the idea of momentum from NESTEROV ’ S METHOD ( Nesterov ( 2004 ) ) or HEAVY BALL method ( Polyak ( 1964 ) ) . ADAGRAD is an online learning algorithm that works well compared to the standard online gradient descent when the gradient is sparse . Its update has a notable feature : the effective learning rate is different for each dimension , depending on the magnitude of gradient in each dimension , which might help in exploiting the geometry of data and leading to a better update . On the other hand , NESTEROV ’ S METHOD or HEAVY BALL Method ( Polyak ( 1964 ) ) is an accelerated optimization algorithm whose update not only depends on the current iterate and current gradient but also depends on the past gradients ( i.e. , momentum ) . State-of-the-art algorithms like AMSGRAD ( Reddi et al . ( 2018 ) ) and ADAM ( Kingma & Ba ( 2015 ) ) leverage the ideas to accelerate training neural nets . In this paper , we propose an algorithm that goes further than the hybrid of the adaptivity and momentum approach . Our algorithm is inspired by OPTIMISTIC ONLINE LEARNING ( see e.g . Chiang et al . ( 2012 ) ; Rakhlin & Sridharan ( 2013a ; b ) ; Syrgkanis et al . ( 2015 ) ; Abernethy et al . ( 2018 ) ) . OPTIMISTIC ONLINE LEARNING considers that a good guess of the loss function in each round is available and plays an action by utilizing the guess . By exploiting the guess , algorithms in OPTIMISTIC ONLINE LEARNING can enjoy a smaller regret than the ones without exploiting the guess . We combine the OPTIMISTIC ONLINE LEARNING idea with the adaptivity and the momentum ideas to design a new algorithm — OPTIMISTIC-AMSGRAD . We also provide a theoretical analysis of OPTIMISTIC-AMSGRAD . The proposed algorithm not only adapts to the informative dimensions , exhibits momentum , but also exploits a good guess of the next gradient to facilitate acceleration . We conduct experiments and show that OPTIMISTIC-AMSGRAD improves AMSGRAD in terms of various measures : training loss , testing loss , and classification accuracy on training/testing data over epochs . 2 PRELIMINARIES . We begin by providing some background in online learning , as it will be the main tool to design and analyze our proposed algorithm . We follow the notation in the literature of adaptive optimization ( Kingma & Ba ( 2015 ) ; Reddi et al . ( 2018 ) ) . For any vector u , v ∈ Rd , u/v represents element-wise division , u2 represents element-wise square , √ u represents element-wise square-root . We denote g1 : T [ i ] as the sum of the ith element of T vectors g1 , g2 , . . . , gT ∈ Rd . 2.1 A BRIEF REVIEW OF ONLINE LEARNING AND OPTIMISTIC ONLINE LEARNING . The standard setup of online learning is that , in each round t , an online learner selects an action wt ∈ K ⊆ Rd , then the learner observes ` t ( · ) and suffers loss ` t ( wt ) after the learner commits the action . The goal of the learner is minimizing the regret , RegretT ( { wt } ) : = ∑T t=1 ` t ( wt ) − ∑T t=1 ` t ( w ∗ ) , which is the cumulative loss of the learner minus the cumulative loss of some benchmark w∗ ∈ K. The idea of OPTIMISTIC ONLINE LEARNING ( e.g. , Chiang et al . ( 2012 ) ; Rakhlin & Sridharan ( 2013a ; b ) ; Syrgkanis et al . ( 2015 ) ; Abernethy et al . ( 2018 ) ) is as follows . Suppose that , in each round t , the learner has a good guess mt ( · ) of the loss function ` t ( · ) before playing an action wt . Then , the learner should exploit the guess mt ( · ) to choose an action wt since mt ( · ) is close to the true loss function ` t ( · ) . 1 For example , Syrgkanis et al . ( 2015 ) proposes an optimistic-variant of FOLLOW-THE-REGULARIZED-LEADER ( FTRL ) . FTRL ( see e.g . Hazan ( 2016 ) ) is an online learning algorithm whose update is wt = arg minw∈K〈w , Lt−1〉+ 1ηR ( w ) , where η is a parameter , R ( · ) is a 1-strongly convex function with respect to a norm ( ‖ · ‖ ) on the constraint setK , and Lt−1 : = ∑t−1 s=1 gs is the cumulative sum of gradient vectors of the loss functions ( i.e. , gs : = ∇ ` s ( ws ) ) up to but not including t. FTRL has regret at most O ( √∑T t=1 ‖gt‖∗ ) . On the other hand , OPTIMISTIC-FTRL ( Syrgkanis et al . ( 2015 ) ) has the update wt = arg minw∈K〈w , Lt−1 +mt〉+ 1ηR ( w ) , where mt is the learner ’ s guess of the gradient vector gt : = ∇ ` t ( wt ) . Under the assumption that loss functions are convex , the regret of OPTIMISTIC-FTRL is at most O ( √∑T t=1 ‖gt −mt‖∗ ) , which can be much smaller than the regret of FTRL if mt is close to gt . Consequently , OPTIMISTIC-FTRL can achieve better performance than FTRL . On the other hand , if mt is far from gt , then the regret of OPTIMISTIC-FTRL would be only a constant factor worse than that of its non-optimistic counterpart . In Section 4 , we will provide a strategy to obtain mt . At the moment , we just would like to use this example of FTRL to emphasize the importance of leveraging a good guess mt for updating wt , in order to achieve a faster convergence rate ( or equivalently , small regret ) . We will have a similar argument when we compare OPTIMISTIC-AMSGRAD and AMSGRAD . 2.2 ADAM AND AMSGRAD . ADAM ( Kingma & Ba ( 2015 ) ) is a popular algorithm for training deep nets . It combines the momentum idea ( Polyak ( 1964 ) ) with the idea of ADAGRAD ( Duchi et al . ( 2011 ) ) , which has effective different learning rates for different dimensions . The effective learning rate of ADAGRAD in iteration t for a dimension j is proportional to the inverse of √ Σts=1gs [ j ] 2 , where gs [ j ] is the jth element of the gradient vector gs in time s. This adaptive learning rate might help for accelerating the convergence when the gradient vector is sparse ( Duchi et al . ( 2011 ) ) . However , when applying ADAGRAD to train deep nets , it is observed that the learning rate might decay too fast ( Kingma & Ba ( 2015 ) ) . Therefore , Kingma & Ba ( 2015 ) propose using a moving average of gradients divided by the square root of the second moment of the moving average ( element-wise fashion ) , for updating the model parameter w ( i.e. , lines 5,6 and 8 of Algorithm 1 ) . Yet , ADAM ( Kingma & Ba ( 2015 ) ) fails at 1Imagine that if the learner would had been known ` t ( · ) before committing its action , then it would exploit the knowledge to determine its action and consequently minimizes the regret . Algorithm 1 AMSGRAD ( Reddi et al . ( 2018 ) ) 1 : Required : parameter β1 , β2 , and ηt . 2 : Init : w1 ∈ K ⊆ Rd and v̂0 = v0 = 1 ∈ Rd . 3 : for t = 1 to T do 4 : Get mini-batch stochastic gradient vector gt at wt . 5 : θt = β1θt−1 + ( 1− β1 ) gt . 6 : vt = β2vt−1 + ( 1− β2 ) g2t . 7 : v̂t = max ( v̂t−1 , vt ) . 8 : wt+1 = wt − ηt θt√ v̂t . ( element-wise division ) 9 : end for some online convex optimization problems . AMSGRAD ( Reddi et al . ( 2018 ) ) fixes the issue . The algorithm of AMSGRAD is shown in Algorithm 1 . The difference between ADAM and AMSGRAD lies on line 7 of Algorithm 1 . ADAM does not have the max operation on line 7 ( i.e. , v̂t = vt for ADAM ) while AMSGRAD adds the operation to guarantee a non-increasing learning rate , ηt√ v̂t , which helps for the convergence ( i.e. , average regret RegretTT → 0 ) . For the parameters of AMSGRAD , it is suggested that β1 = 0.9 and β2 = 0.99 . 3 OPTIMISTIC-AMSGRAD . Algorithm 2 OPTIMISTIC-AMSGRAD 1 : Required : parameter β1 , β2 , , and ηt . 2 : Init : w1 = w−1/2 ∈ K ⊆ Rd and v̂0 = v0 = 1 ∈ Rd . 3 : for t = 1 to T do 4 : Get mini-batch stochastic gradient vector gt at wt . 5 : θt = β1θt−1 + ( 1− β1 ) gt . 6 : vt = β2vt−1 + ( 1− β2 ) ( gt −mt ) 2 . 7 : v̂t = max ( v̂t−1 , vt ) . 8 : wt+ 1 2 = ΠK [ wt− 1 2 − ηt θt√v̂t ] . 9 : wt+1 = ΠK [ wt+ 1 2 − ηt+1 ht+1√v̂t ] , where ht+1 : = β1θt−1 + ( 1− β1 ) mt+1 and mt+1 is the guess of gt+1 . 10 : end for We propose a new optimization algorithm , OPTIMISTIC-AMSGRAD , shown in Algorithm 2 . In each iteration , the learner computes a gradient vector gt : = ∇ ` t ( wt ) at wt ( line 4 ) , then it maintains an exponential moving average of θt ∈ Rd ( line 5 ) and vt ∈ Rd ( line 6 ) , which is followed by the max operation to obtain v̂t ∈ Rd ( line 7 ) . The learner also updates an auxiliary variable wt+ 12 ∈ K ( line 8 ) . It uses the auxiliary variable to update and commit wt+1 ( line 9 ) , which exploits the guess mt+1 of gt+1 to get wt+1 . As the learner ’ s action set is K ⊆ Rd , we adopt the notation ΠK [ · ] for the projection to K if needed . We see that OPTIMISTIC-AMSGRAD has three properties : • Adaptive learning rate of each dimension as ADAGRAD ( Duchi et al . ( 2011 ) ) . ( line 6 , line 8 and line 9 ) • Exponentially moving average of the past gradients as NESTEROV ’ S METHOD ( Nesterov ( 2004 ) ) and the HEAVY-BALL method ( Polyak ( 1964 ) ) . ( line 5 ) • Optimistic update that exploits a good guess of the next gradient vector as optimistic online learning algorithms ( e.g . Chiang et al . ( 2012 ) ; Rakhlin & Sridharan ( 2013a ; b ) ; Syrgkanis et al . ( 2015 ) ) . ( line 9 ) The first property helps for acceleration when the gradient has a sparse structure . The second one is from the well-recognized idea of momentum which can also help for acceleration . The last one , perhaps less known outside the ONLINE LEARNING community , can actually lead to acceleration when the prediction of the next gradient is good . This property will be elaborated in the following subsection in which we provide the theoretical analysis of OPTIMISTIC-AMSGRAD . Observe that the proposed algorithm does not reduce to AMSGRAD when mt = 0 . Furthermore , if K = Rd ( unconstrained case ) , one might want to combine line 8 and line 9 and get a single line as wt+1 = wt− 12 − ηt θt√ v̂t − ηt+1 ht+1√v̂t . Yet , based on this expression , we see that wt+1 is updated from wt− 12 instead of wt . Therefore , while OPTIMISTIC-AMSGRAD looks like just doing an additional update compared to AMSGRAD , the difference of the updates is subtle . In the following analysis , we show that the interleaving actually leads to certain cancellation in the regret bound .

Summary: This paper proposes an online optimization method called Optimistic-AMSGrad, which combines two existing methods: (i) AMSGrad (Reddi et al 2018) and (ii) optimistic online learning where the prediction step is done with the extrapolation algorithm by Scieur et al 2016. The authors do a good job of presenting the method (by introducing the background in proper order), the paper seems self-contained and cites the relevant literature. The regret analysis of the proposed algorithm is provided, where the obtained regret can be smaller than AMSGrad depending on whether or not the guess of the gradient and the gradient are close.