On the Dynamic Regret of Online Multiple Mirror Descent

1 INTRODUCTION . Online optimization refers to the design of sequential decisions where system parameters and cost functions vary with time . It has applications to various classes of problems , such as object tracking ( Shahrampour & Jadbabaie , 2017 ) , networking ( Shi et al. , 2018 ) , cloud computing ( Lin et al. , 2012 ) , and classification ( Crammer et al. , 2006 ) . It is also an important tool in the development of algorithms for reinforcement learning ( Yuan & Lamperski , 2017 ) and deep learning ( Mnih et al. , 2015 ) . In this work , we consider online convex optimization , which can be formulated as a discrete-time sequential learning process as follows . At each round t , the learner first makes a decision xt ∈ X , where X is a convex set representing the solution space . The learner then receives a convex cost function ft ( x ) : X → R and suffers the corresponding cost of ft ( xt ) associated with the submitted decision . The goal of the online learner is to minimize the total accrued cost over a finite number of rounds , denoted by T . For performance evaluation , prior studies on online learning often focus on the static regret , defined as the difference between the learner ’ s accumulated cost and that of an optimal fixed offline decision , which is made in hindsight with knowledge of ft ( · ) for all t : RegsT = T∑ t=1 ft ( xt ) −min x∈X T∑ t=1 ft ( x ) . A successful online algorithm closes the gap between the online decisions and the offline counterpart when normalized by T , i.e. , sustaining sublinear static regret in T . In the literature , there are various online algorithms ( Zinkevich , 2003 ; Cesa-Bianchi & Lugosi , 2006 ; Hazan et al. , 2006 ; Duchi et al. , 2010 ; Shalev-Shwartz , 2012 ) that guarantee a sublinear bound on the static regret . However , algorithms that guarantee performance close to that of a static decision may still perform poorly in dynamic settings . Consequently , the static regret fails to accurately reflect the quality of decisions in many practical scenarios . Therefore , the dynamic regret has become a popular metric in recent works ( Besbes et al. , 2015 ; Mokhtari et al. , 2016 ; Yang et al. , 2016 ; Zhang et al. , 2017 ) , which allows a dynamic sequence of comparison targets and is defined by RegdT = T∑ t=1 ft ( xt ) − T∑ t=1 ft ( x ∗ t ) , where x∗t = argminx∈X ft ( x ) is a minimizer of the cost at round t. It is well-known that the online optimization problem may be intractable in a dynamic setting , due to arbitrary fluctuation in the cost functions . Hence , achieving a sublinear bound on the dynamic regret may be impossible . However , it is possible to upper bound the dynamic regret in terms of certain regularity measures . One of the measures to represent regularity is the path length , defined by CT = T∑ t=2 ‖x∗t − x∗t−1‖ , ( 1 ) which illustrates the accumulative variation in the minimizer sequence . For instance , the dynamic regret of online gradient descent for convex cost functions can be bounded by O ( √ T ( 1 + CT ) ) ( Zinkevich , 2003 ) . 1 For strongly convex functions , the dynamic regret of online gradient descent can be reduced to O ( CT ) ( Mokhtari et al. , 2016 ) . When the cost functions are smooth and strongly convex , by allowing the learner to make multiple queries to the gradient of the cost functions , the regret bound can be further improved to O ( min ( CT , ST ) ) , where ST represents the squared path length , defined by ST = T∑ t=2 ‖x∗t − x∗t−1‖2 , ( 2 ) which can be smaller than the path length when the distance between successive minimizers is small . All the aforementioned studies require the cost functions to be Lipschitz continuous . However , there are many commonly used cost functions , e.g. , the quadratic function that do not meet the Lipschitz condition . In addition , the above works rely on measuring distances using Euclidean norms , which hinders the projection step in gradient descent update for some constraint sets , e.g. , probability simplex ( Duchi , 2018 ) . Besides gradient descent , mirror descent is another well-known technique of online convex optimization ( Hall & Willett , 2015 ; Jadbabaie et al. , 2015 ) . Mirror descent uses the Bregman divergence , which generalizes the Euclidean norm used in the projection step of gradient descent , thus acquiring expanded applicability to a broader range of problems . In addition , the Bregman divergence is only mildly dependent on the dimension of decision variables ( Beck & Teboulle , 2003 ; Nemirovsky & Yudin , 1983 ) , so that mirror descent is optimal among first-order methods when the decision variables have high dimensions ( Duchi et al. , 2010 ) . In this work we focus on the mirror descent approach . In previous works on online mirror descent , the learner queries the gradient of each cost function only once , and performs one step of mirror descent to update its decision ( Hall & Willett , 2015 ; Shahrampour & Jadbabaie , 2017 ) . In this case , the dynamic regret has an upper bound of order O ( √ T ( 1 + CT ) ) , which is the same as that of online gradient descent in ( Zinkevich , 2003 ) . In this work , we investigate whether it is possible to improve the dynamic regret when the learner performs multiple mirror descent steps in each online round , while relaxing the Lipschitz continuity condition on the cost functions . To this end , we analyze the performance of the Online Multiple Mirror Descent ( OMMD ) algorithm , which uses multiple steps of mirror descent per online round . When the cost functions are smooth and strongly convex , we show that the upper bound on the dynamic regret can be reduced from 1A more general definition of the dynamic regret was introduced in ( Zinkevich , 2003 ) , which allows comparison against an arbitrary sequence { ut } Tt=1 . We note that the regret bounds developed in ( Zinkevich , 2003 ) also hold for the specific case of ut = x∗t . O ( √ T ( 1 + CT ) ) to O ( min ( CT , ST , GT ) ) , where GT represent the sum squared gradients , i.e. , GT = T∑ t=1 ‖∇ft ( xt ) ‖2∗ , ( 3 ) where ‖.‖∗ denotes the dual norm . The sum squared gradients GT can be smaller than both the path length and squared path length , especially when the cost functions fluctuate drastically over time . In contrast to the aforementioned works , our analysis does not require the cost functions to be Lipschitz continuous . Furthermore , our numerical experiments suggest substantially reduced dynamic regret compared with the best known alternatives , including single-step dynamic mirror descent ( Hall & Willett , 2015 ) , online multiple gradient descent ( Zhang et al. , 2017 ) , and online gradient descent Zinkevich ( 2003 ) . 2 ONLINE MULTIPLE MIRROR DESCENT . In this section , we describe OMMD and discuss how the learner can improve the dynamic regret by performing multiple mirror descent steps per round . Before delving into the details , we proceed by stating several definitions and standard assumptions . 2.1 PRELIMINARIES . Definition 1 : The Bregman divergence with respect to the regularization function r ( · ) is defined as Dr ( x , y ) = r ( x ) − r ( y ) − 〈∇r ( y ) , x− y〉 . The Bregman divergence is a general distance-measuring function , which contains the Euclidean norm and the Kullback-Leibler divergence as two special cases . Using the Bregman divergence , a generalized definition of strong convexity is given in ( ShalevShwartz & Singer , 2007 ) . Definition 2 : A convex function f ( · ) is λ-strongly convex with respect to a convex and differentiable function r ( · ) if f ( y ) + 〈∇f ( y ) , x− y〉+ λDr ( x , y ) ≤ f ( x ) , ∀x , y ∈ X . Following many prior studies on mirror descent , we assume that the cost functions are λ-strongly convex , where the above generalized strong convexity definition is used . We further assume that the cost functions are L-smooth , and the regularization function r ( · ) is Lr-smooth and 1-strongly convex with respect to some norm ( refer to App . A for definitions ) . We note that these are standard assumptions commonly used in the literature after the group of studies began by ( Hazan et al. , 2006 ; Shalev-Shwartz & Singer , 2007 ) , to provide stronger regret bounds by constraining the curvature of cost functions . We further make a standard assumption that the Bregman divergence is Lipschitz continuous as follows : |Dr ( x , z ) −Dr ( y , z ) | ≤ K‖x− y‖ , ∀x , y , z ∈ X , where K is a positive constant . We note that this condition is much milder than the condition of Lipschitz continuous cost functions required in ( Zhang et al. , 2017 ; Mokhtari et al. , 2016 ; Hall & Willett , 2015 ) . There is a notable weakness in such bounds . Since the sequence of cost functions are revealed to the learner , the learner has no control over it . If these cost functions happen to not meet the Lipschitz condition , earlier analyses that require this condition become inapplicable . In this work , we do not require the cost functions to be Lipschitz continuous . Instead , we move the Lipschitz continuity condition from the cost functions to the Bregman divergence to broaden the application of our work . The main benefit of this is that the regularization function and the corresponding Bregman divergence is within the control of the learner . The learner can carefully design this regularization function to satisfy the Lipschitz continuity of the associated Bregman divergence with a small factor . For example , in the particular case of the KL divergence , which is obtained by the choosing negative entropy as the regularization function , on the set X = { x| ∑d i=1 xi = 1 ; xi ≥ 1 D } , the constant K is of O ( logD ) . Other examples of many widely used Bregman divergences that satisfy this condition are given in ( Bauschke & Borwein , 2001 ) . Algorithm 1 Online Multiple Mirror Descent Input : Arbitrary initialization of x1 ∈ X ; step size α ; time horizon T . Output : Sequence of decisions { xt : 1 ≤ t ≤ T } . 1 : for t = 1 , 2 , . . . , T do 2 : submit xt ∈ X and receive ft ( · ) 3 : set y1t = xt 4 : for i = 1 , 2 , . . . , M do 5 : yi+1t = argminy∈X { 〈∇ft ( yit ) , y〉+ 1αDr ( y , y i t ) } 6 : end for 7 : set xt+1 = yM+1t 8 : end for

Summary: This work derives a new upper bound on the dynamic regret for online convex optimization in the restricted setting where the comparison sequence is made up of the minimizers x^_1,...,x*_T of the loss sequence. There are three main parameters that control regret in this case: the path length C_T = ||x*_2-x*_1||+...+||x*_T-x*_{T-1}||. The squared path length C_T = ||x*_2-x*_1||^2+...+||x*_T-x*_{T-1}||^2. And the sum G_T of squared loss gradient norms evaluated at x*_1,...,x*_T.