On the Marginal Regret Bound Minimization of Adaptive Methods

1 INTRODUCTION . Accelerating the convergence speed of optimization algorithms is one main concern of the machine learning community . After stochastic gradient descent ( SGD ) was introduced , quite a few variants of SGD have become popular , such as momentum ( Polyak , 1964 ) and AdaGrad ( Duchi et al. , 2011 ) . Instead of directly moving parameters in the negative direction of the gradient , AdaGrad proposed to scale the gradient by a matrix , which was the matrix in the proximal function of the composite mirror descent rule ( Duchi et al. , 2011 ) . The diagonal version of AdaGrad designed this matrix to be the square root of the global average of the squared gradients . Duchi et al . ( 2011 ) proved that this algorithm could be faster than SGD when the gradients were sparse . However , AdaGrad ’ s performance is known to deteriorate when the gradients are dense , especially in high dimensional problems such as deep learning ( Reddi et al. , 2018 ) . To tackle this issue , many new algorithms were proposed to boost the performances of AdaGrad . Most of these algorithms focused on changing the design of the matrix in the proximal function . For example , RMSProp ( Tieleman & Hinton , 2012 ) and Adam ( Kingma & Ba , 2015 ) changed the global average design in AdaGrad to the exponential moving average . However , Reddi et al . ( 2018 ) proved that such a modification had convergence issues in the presence of high frequency noises and added a max operation to the matrix of Adam , leading to the AMSGrad algorithm . Other modifications , such as Padam ( Chen & Gu , 2018 ) , AdaShift ( Zhou et al. , 2019 ) , NosAdam ( Huang et al. , 2019 ) , and Radam ( Liu et al. , 2019 ) , were based on various designs of this matrix as well . However , all aforementioned works did not improve the convergence rate of AdaGrad and simply supported their designs using experiments and synthetic examples . A theoretical foundation for the design of this matrix that improves the convergence and guides future adaptive algorithms is very much needed . In this work , we bring new insights to the design of the matrix in the proximal function . In particular , our major contributions in this paper are listed as follows • We propose a new motivation for designing the proximal function in adaptive algorithms . Specifically , we have found a marginally optimal design , which is the best matrix at each time step through minimizing the marginal increment of the regret bound . • Based on our proposal of marginal regret bound minimization , we create a new class of adaptive algorithms , named as AMX . We prove theoretically that AMX can converge with a regret bound of size Õ ( √ τ ) , where τ is smaller than T . Such a regret bound is potentially much smaller than those of common adaptive algorithms and can make AMX converge much faster than any existing adaptive algorithms , depending on τ . In the worst case , we show it is at least as fast as AMSGrad and AdaGrad under the same assumptions • We evaluate AMX ’ s empirical performance on different tasks in deep learning . All experi- ments show our algorithm can converge fast and achieve good testing performances . 2 BACKGROUND . Notation : We denote the set of all positive definite matrices in Rd×d by S+d . For any two vectors a , b ∈ Rd , we use √ a for element-wise square root , a2 for element-wise square , |a| for element-wise absolute value , a/b for element-wise division , and max ( a , b ) for element-wise maximum between a and b . We also frequently use the notation g1 : T , i = [ g1 , i , g2 , i , · · · , gT , i ] , i.e . the vector of all the i-th elements of vectors g1 , g2 , · · · gT . For a vector a , we use diag ( a ) to represent the diagonal matrix whose diagonal entries are a . For two functions f ( t ) , g ( t ) , f ( t ) = o ( g ( t ) ) means f ( t ) /g ( t ) → 0 as t goes to infinity . We use Õ ( · ) to omit logarithm terms in big-O notations . We say a space X has a bounded diameter D∞ if ‖x− y‖∞ ≤ D∞ , ∀x , y ∈ X . Online Learning Framework . We choose the online learning framework to analyze all the algorithms in this paper . In this framework , an algorithm picks a new xt ∈ X according to its update rule at each iteration t , where X ⊆ Rd is the set of feasible values of xt . The composite loss function ft + φ is then revealed , where φ is the regularization function that controls the complexity of x and ft can be considered as the instantaneous loss at t. In the convex setting , ft and φ are both convex functions . The regularized regret function is defined with respect to an optimal predictor x∗ as R ( T ) = T∑ t=1 ft ( xt ) − ft ( x∗ ) + φ ( xt ) − φ ( x∗ ) . Our goal is to find algorithms that ensures a sub-linear regret , i.e . R ( T ) = o ( T ) , which means that the average regret converges to zero . For example , online gradient descent is proved to have a regret of O ( √ dT ) ( Zinkevich , 2003 ) , where d is the dimension size of X . Note that stochastic optimization and online learning are basically interchangeable ( Cesa-Bianchi et al. , 2004 ) . Therefore , we will refer to online algorithms and their stochastic counterparts using the same names . For example , we will use stochastic gradient descent ( SGD ) to represent online gradient descent as it is more well-known . Composite Mirror Descent Setup . In this paper , we will revisit the general composite mirror descent method ( Duchi et al. , 2010b ) used in the creation of the first adaptive algorithm , AdaGrad , to bring new insights into adaptive methods . Such a general framework is preferred because it covers a wide range of algorithms , including both SGD and all the adaptive methods , and thus simplifies the discussions . The composite mirror descent rule at the time step t+ 1 is to solve for xt+1 = argminx∈X { αt〈gt , x〉+ αtφ ( x ) +Bψt ( x , xt ) } , ( 1 ) where gt is the gradient , φ ( x ) is the regularization function in the dual space , and αt is the step size . Also , ψt is a strongly convex and differentiable function , named as the proximal function and Bψt ( x , xt ) is the Bregman divergence associated with ψt defined as Bψt ( x , y ) = ψt ( x ) − ψt ( y ) − 〈∇ψt ( y ) , x− y〉 . The general update rule ( 1 ) is mostly determined by the function ψt . We first observe that it becomes the projected SGD algorithm when ψt ( x ) = xTx and φ ( x ) = 0. xt+1 = argminx∈X { αt〈gt , x〉+ ‖x− xt‖22 } = ΠX ( xt − αtgt ) , ( SGD ) where ΠX ( x ) = argminy∈X ‖x− y‖2 is the projection operation that ensures the updated parameter is in the original space . On the other hand , adaptive algorithms choose different proximal functions ψt ( x ) = 〈x , Htx〉 , where Ht can be any full or diagonal symmetric positive definite matrix . xt+1 = argminx∈X { αt〈gt , x〉+ αtφ ( x ) + 〈x− xt , Ht ( x− xt ) 〉 } , ( Adaptive ) Another popular representation of adaptive algorithms is the generalized projection rule xt+1 = ΠX , Ht ( xt − αtH−1t gt ) , where ΠX , Ht ( x ) = argminy∈X ‖H 1/2 t ( x− y ) ‖2 , which is used in a lot of recent literature such as Reddi et al . ( 2018 ) ; Huang et al . ( 2019 ) . We show that these two rules are actually equivalent when φ ( x ) = 0 in the Appendix A.1 , so that the regret bounds found in different representations can be generalized . A few recent works have shown that adaptive algorithms work well with special designs of the step sizes αt ( Choi et al. , 2019 ; Vaswani et al. , 2020 ) . In this work , we choose the more standard αt = α/ √ t as it is used in most analysis . Also , we restrain our discussions to diagonal matrix proximal functions in the main text , i.e . Ht = diag ( ht ) , ht ∈ Rd . Discussions on extending our results to full matrix proximal functions are provided in Appendix B . Different Designs of the Proximal Function . Recently , researchers have proposed numerous designs of Ht = diag ( ht ) , such as AdaGrad ( Duchi et al. , 2011 ) , Adam ( Kingma & Ba , 2015 ) , AMSGrad ( Reddi et al. , 2018 ) and NosAdam ( Huang et al. , 2019 ) , to name a few . It ’ s impossible to go over all the proposed designs in this section so we choose the two most famous designs to review . The first adaptive algorithm , AdaGrad , used the square root of the average of past gradient squares as the diagonal ht of the matrix in the proximal function ( Duchi et al. , 2011 ) , i.e . ht = ( ∑t i=1 g 2 i t ) 1/2 ( AdaGrad ) Normally , a small constant is added to ht at each iteration . Some recent work have shown that tuning this constant can benefit the performance of adaptive algorithms ( Zaheer et al. , 2018 ; Savarese et al. , 2019 ) . However , in this work , we assume it is small and fixed for simplicity , as it is originally designed to compute the pseudo-inverse , or equivalently , avoid division by zero in the generalized projected descent . Kingma & Ba ( 2015 ) proposed Adam to replace the simple average by exponential moving average in AdaGrad , but Reddi et al . ( 2018 ) showed that there was a mistake in Adam ’ s convergence analysis , which lead to divergence of Adam even in simple convex problems . They therefore proposed the following simple modification of Adam to ensure its convergence . ht = maxt { ∑t i=1 ( 1− β2 ) β t−i 2 g 2 i t ) 1/2 } ( AMSGrad ) where β2 ∈ ( 0 , 1 ) is a constant . We propose the following theorem that generalizes the regret bounds for most of the designs of the diagonal matrix proximal function . Theorem 2.1 Let the sequence { xt } be defined by the update rule ( 1 ) and for any x∗ , denote D2t , ∞ = ‖xt − x∗‖2∞ . When ψt ( x ) = 〈x , Htx〉 , where Ht = diag ( ht,1 , ht,2 , · · · , ht , d ) , assume without loss of generality that φ ( x1 ) = 0 , H0 = 0 , if ( ht , i/αt ) ≥ ( ht−1 , i/αt−1 ) , then R ( T ) ≤ T∑ t=1 d∑ i=1 ( ht , i αt − ht−1 , i αt−1 ) D2t , ∞ + T∑ t=1 d∑ i=1 αtg 2 t , i 2ht , i . ( 2 ) The proof is relegated to Appendix A.3 . The above regret bound is suitable for any designs of ht that satisfy the constraint condition ( ht , i/αt ) ≥ ( ht−1 , i/αt−1 ) . Such a condition is crucial because if it is unsatisfied , the regret R ( T ) might diverge . In fact , the divergence of Adam in simple convex problems results from not satisfying this constraint , which is proved by Reddi et al . ( 2018 ) . With αt = α/ √ t in Theorem 2.1 , most of recent adaptive algorithms have a regret bound of the following form ( Duchi et al. , 2011 ; Reddi et al. , 2018 ; Huang et al. , 2019 ; Luo et al. , 2019 ) . R ( T ) ≤ C1 √ T d∑ i=1 hT , i + f ( T ) d∑ i=1 ‖g1 : T , i‖2 + C2 ( 3 ) where C1 , C2 are constants and f ( T ) = o ( √ T ) . 1 These algorithms are supposed to converge faster than SGD when the gradients are sparse or small , i.e . when ∑d i=1 hT , i √ d and ∑d i=1 ‖g1 : T , i‖2 √ dT . However , all existing regret bounds are still O ( √ T ) , which makes it hard to compare different proximal functions . Whether the best proximal function exists and whether the O ( √ T ) regret bound can be further improved still remain open questions . 1The regret bound of AdaGrad given in the original paper was O ( ‖g1 : T , i‖2 ) because Duchi et al . ( 2011 ) used a constant learning rate αt = α and hence ht = ∑t i=1 g 2 i . When changed into the same setting where αt = α/ √ t , ht becomes ∑t i=1 g 2 i /t and the regret also has the form of ( 3 ) , shown in Reddi et al . ( 2018 ) .

Summary: This paper introduces a new online convex optimization algorithm that operates in via the reduction to online linear optimization in which the regret is bounded by $\sum_{t=1}^T \langle g_t, x_t - u\rangle$ where $g_t$ is the gradient of the t^th loss at $x_t$. The algorithm is based on online mirror descent with non-decreasing quadratic regularizers $x^\top H_t x$, using the update $x_{t+1} = argmin_x \langle g_t, x \rangle + (x-x_t)^\top H_t(x-x_t)$ (or, equivalently using the terminology in the paper, $argmin \langle g_t, x_t\rangle /\sqrt(t) + (x-x_t)^\top H_t(x-x_t)$ where we replace $H_t$ by $H_t/\sqrt{t}$. The analysis is restricted to diagonal $H_t$, for which we can break the regret into a sum of $d$ 1-dimensional problems, so it suffices to do the analysis in the scalar case. The idea is to break out the standard analysis of mirror descent regret as the sum over all t of $D^2(H_t - H_{t-1}) + g_t^2 H_t^{-1}$, where $D$ is the $\ell_\infty$ diameter of the domain, and then choose $H_t$ to minimize each of these terms greedily subject to the non-decreasing condition. A regret bound is provided for this algorithm that achieves worst-case $\sqrt{T}$ regret, but in cases in which the gradients are small, the regret is much better. By employing this strategy on a per-coordinate basis one can obtain an adagrad-esque regret bound.