Revisiting Smoothed Online Learning

4 λ -competitive for λ-quadratic growth functions , both of which improve state-of-the-art results significantly . Moreover , when the hitting cost is both convex and λ-quadratic growth , we reduce the competitive ratio to 1 + 2√ λ by minimizing the weighted sum of the hitting cost and the switching cost . To bound the dynamic regret with switching cost , we follow the standard setting of online convex optimization , in which the hitting cost is convex but hidden from the learner before making predictions . We modify Ader , an existing algorithm designed for dynamic regret , slightly to take into account the switching cost when measuring the performance . The proposed algorithm , named as Smoothed Ader , attains an optimal O ( √ T ( 1 + PT ) ) bound for dynamic regret with switching cost , where PT is the path-length of the comparator sequence . Furthermore , if the hitting cost is accessible in the beginning of each round , we obtain a similar guarantee without the bounded gradient condition , and establish an Ω ( √ T ( 1 + PT ) ) lower bound to confirm the optimality . 1 Introduction . Online learning is the process of making a sequence of predictions given knowledge of the answer to previous tasks and possibly additional information [ Shalev-Shwartz , 2011 ] . While the traditional online learning aims to make the prediction as accurate as possible , in this paper , we study smoothed online learning ( SOL ) , where the online learner incurs a switching cost for changing its predictions between rounds [ Cesa-Bianchi et al. , 2013 ] . SOL has received lots of attention recently because in many real-world applications , a change of action usually brings some additional cost . Examples include the dynamic right-sizing for data centers [ Lin et al. , 2011 ] , geographical load balancing [ Lin et al. , 2012 ] , real-time electricity pricing [ Kim and Giannakis , 2014 ] , video streaming [ Joseph and de Veciana , 2012 ] , spatiotemporal sequence prediction [ Kim et al. , 2015 ] , multi-timescale control [ Goel et al. , 2017 ] , and thermal management [ Zanini et al. , 2010 ] . Specifically , SOL is performed in a sequence of consecutive rounds , where at round t the learner is asked to select a point xt from the decision set X , and suffers a hitting cost ft ( xt ) . Depending on the performance metric , the learner may be allowed to observe ft ( · ) when making decisions , which is different from the traditional online learning in which ft ( · ) is revealed to the learner after submitting the decision [ Cesa-Bianchi and Lugosi , 2006 ] . Additionally , the learner also incurs a switching cost m ( xt , xt−1 ) for changing decisions between successive rounds . The switching cost m ( xt , xt−1 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . could be any distance function , such as the ` 2-norm distance ‖xt − xt−1‖ and the squared ` 2-norm distance ‖xt − xt−1‖2/2 [ Goel et al. , 2019 ] . In the literature , there are two performance metrics for SOL : competitive ratio and dynamic regret with switching cost . Competitive ratio is popular in the community of online algorithms [ Borodin and El-Yaniv , 1998 ] . It is defined as the worst-case ratio of the total cost incurred by the online learner and the offline optimal cost : ∑T t=1 ( ft ( xt ) +m ( xt , xt−1 ) ) minu0 , u1 , ... , uT∈X ∑T t=1 ( ft ( ut ) +m ( ut , ut−1 ) ) . ( 1 ) When focusing on the competitive ratio , the learner can observe ft ( · ) before picking xt . The problem is still nontrivial due to the coupling created by the switching cost . On the other hand , dynamic regret with switching cost is a generalization of dynamic regret—a popular performance metric in the community of online learning [ Zinkevich , 2003 ] . It is defined as the difference between the total cost incurred by the online learner and that of an arbitrary comparator sequence u0 , u1 , . . . , uT ∈ X : T∑ t=1 ( ft ( xt ) +m ( xt , xt−1 ) ) − T∑ t=1 ( ft ( ut ) +m ( ut , ut−1 ) ) . ( 2 ) Different from previous work [ Chen et al. , 2018 , Goel et al. , 2019 ] , we did not introduce the minimization operation over u0 , u1 , . . . , uT in ( 2 ) . The reason is that we want to bound ( 2 ) by certain regularities of the comparator sequence , such as the path-length PT ( u0 , u1 , . . . , uT ) = T∑ t=1 ‖ut − ut−1‖ . ( 3 ) When focusing on ( 2 ) , ft ( · ) is generally hidden from the learner before submitting xt . The conditions for bounding the two metrics are very different , so we study competitive ratio and dynamic regret with switching cost separately . To bound the two metrics simultaneously , we refer to Andrew et al . [ 2013 ] and Daniely and Mansour [ 2019 ] , especially the meta-algorithm in the latter work . This paper follows the line of research stemmed from online balanced descent ( OBD ) [ Chen et al. , 2018 , Goel and Wierman , 2019 ] . The key idea of OBD is to find an appropriate balance between the hitting cost and the switching cost through iterative projections . It has been shown that OBD and its variants are able to exploit the analytical properties of the hitting cost ( e.g. , polyhedral , strongly convex ) to derive dimension-free competitive ratio . At this point , it would be natural to ask why not use the greedy algorithm , which minimizes the weighted sum of the hitting cost and the switching cost in each round , i.e. , min x∈X ft ( x ) + γm ( x , xt−1 ) ( 4 ) to balance the two costs , where γ ≥ 0 is the trade-off parameter . We note that the greedy algorithm is usually treated as the baseline in competitive analysis [ Borodin and El-Yaniv , 1998 ] , but its usage for smoothed online learning is quite limited . One result is given by Goel et al . [ 2019 ] , who demonstrate that the greedy algorithm as a special case of Regularized OBD ( R-OBD ) , is optimal for strongly convex functions . Besides , Lin et al . [ 2020 ] have analyzed the greedy algorithm with γ = 0 , named as the naive approach below , for polyhedral functions and quadratic growth functions . In this paper , we make the following contributions towards understanding the greedy algorithm . • For α-polyhedral functions , the competitive ratio of the naive approach is max ( 1 , 2α ) , which is a significant improvement over the 3 + 8α competitive ratio of OBD [ Chen et al. , 2018 ] and the 1 + 2α ratio proved by Lin et al . [ 2020 , Lemma 1 ] . When α > 2 , the ratio becomes 1 , indicating that the naive approach is optimal in this scenario . • For λ-quadratic growth functions , the competitive ratio of the naive algorithm is 1 + 4λ , which matches the lower bound of this algorithm [ Goel et al. , 2019 , Theorem 5 ] , and is better than the max ( 1 + 6λ , 4 ) ratio obtained by Lin et al . [ 2020 , Lemma 1 ] . • If the hitting cost is both convex and λ-quadratic growth , the greedy algorithm with γ > 0 attains a 1+ 2√ λ competitive ratio , which demonstrates the advantage of taking the switching cost into considerations . Our 1 + 2√ λ ratio is on the same order as Greedy OBD [ Goel et al. , 2019 , Theorem 3 ] but with much smaller constants . • Our analysis of the naive approach and the greedy algorithm is very simple . In contrast , both OBD and Greedy OBD rely on intricate geometric arguments . While both OBD and R-OBD are equipped with sublinear dynamic regret with switching cost , they are unsatisfactory in the following aspects : • The regret of OBD depends on an upper bound of the path-length instead of the path-length itself [ Chen et al. , 2018 , Corollary 11 ] , making it nonadaptive . • The regret of R-OBD is adaptive but it uses the squared ` 2-norm to measure the switching cost , which may not be suitable for general convex functions [ Goel et al. , 2019 ] .1 • Both OBD and R-OBD observe ft ( · ) before selecting xt , which violates the convention of online learning . To avoid the above limitations , we demonstrate that a small change of Ader [ Zhang et al. , 2018a ] , which is an existing algorithm designed for dynamic regret , is sufficient to minimize the dynamic regret with switching cost under the setting of online convex optimization [ Shalev-Shwartz , 2011 ] . Ader runs multiple online gradient descent ( OGD ) [ Zinkevich , 2003 ] with different step sizes as expert-algorithms , and uses Hedge [ Freund and Schapire , 1997 ] as the meta-algorithm to aggregate predictions from experts . The only modification is to incorporate the switching cost into the loss of Hedge . The proposed algorithm , named as Smoothed Ader ( SAder ) , attains the optimal O ( √ T ( 1 + PT ) ) dynamic regret , where PT is the path-length defined in ( 3 ) . Thus , our regret bound is adaptive because it automatically becomes small when the comparators change slowly . Finally , we also investigate the case that the hitting cost is available before predictions , and establish a similar result without the bounded gradient condition . To this end , we design a lookahead version of SAder , which chooses the greedy algorithm in ( 4 ) as the expert and utilizes the cost of the current round in Hedge . To show the optimality of this algorithm , we further establish an Ω ( √ T ( 1 + PT ) ) lower bound under the lookahead setting . 2 Related work . This section reviews related work on smoothed online learning ( SOL ) and dynamic regret . 2.1 Smoothed online learning . SOL has been investigated under the setting of multi-armed bandits [ Agrawal et al. , 1990 , Guha and Munagala , 2009 , Dekel et al. , 2014 , Koren et al. , 2017a , b ] , prediction with expert advice [ CesaBianchi et al. , 2013 ] , and online convex optimization [ Lin et al. , 2011 , Bansal et al. , 2015 , Zhao et al. , 2020b ] . In the following , we mainly discuss smoothed online convex optimization ( SOCO ) . The early works on SOCO focus on designing competitive algorithms in the low-dimensional setting [ Lin et al. , 2011 ] . In particular , Bansal et al . [ 2015 ] show that for SOCO on the real line , the competitive ratio can be upper bounded by 2 , which is proved to be optimal [ Antoniadis and Schewior , 2018 ] . They also establish a competitive ratio of 3 under the memoryless setting . In the study of SOCO , it is common to assume that the learner has access to predictions of future hitting costs , and several algorithms [ Lin et al. , 2012 , Chen et al. , 2015 , 2016 , Li et al. , 2018 , Li and Li , 2020 ] have been developed based on receding horizon control ( RHC ) [ Kwon and Han , 2005 ] . In fact , the greedy algorithm in ( 4 ) can be treated as a variant of RHC . However , previous results for RHC are limited to special problems , and ( 4 ) remains under-explored . For example , when the learner can observe the next W hitting costs , Li et al . [ 2018 ] demonstrate that both the competitive ratio and the dynamic regret with switching cost decay exponentially fast with W . But their analysis relies on very strong conditions , including strong convexity and smoothness . One milestone is the online balanced descent ( OBD ) [ Chen et al. , 2018 ] , which has dimensionfree competitive ratio even when the learner can only observe the hitting cost of the current round . Specifically , OBD iteratively projects the previous point onto a carefully chosen level set of the hitting cost so as to balance the switching cost and the hitting cost . When the hitting cost is α-polyhedral and convex , and the switching cost is the ` 2-norm distance , OBD attains a 3 + 8α competitive ratio . Furthermore , OBD can also be tuned to control the dynamic regret with switching cost . Let L be 1We usually assume the convex function is Lipschitz continuous , and in this case choosing the ` 2-norm distance as the switching cost makes it on the same order as the hitting cost . an upper bound of the path-length of the comparator sequence , i.e. , PT ( u0 , u1 , . . . , uT ) ≤ L. Chen et al . [ 2018 , Corollary 11 ] have proved that T∑ t=1 ( ft ( xt ) + ‖xt − xt−1‖ ) − min PT ( u0 , u1 , ... , uT ) ≤L T∑ t=1 ( ft ( ut ) + ‖ut − ut−1‖ ) = O ( √ TL ) ( 5 ) leading to sublinear regret when L = o ( T ) . However , the upper bound is nonadaptive because it depends on L instead of the actual path-length PT . Later , Goel and Wierman [ 2019 ] demonstrate that OBD is 3 + O ( 1λ ) -competitive for λ-strongly convex functions , when the switching cost is set to be the squared ` 2-norm distance . In a subsequent work , Goel et al . [ 2019 , Theorem 4 ] propose Regularized OBD ( R-OBD ) , which improves the competitive ratio to 12 + 1 2 √ 1 + 4λ , matching the lower bound of strongly convex functions exactly [ Goel et al. , 2019 , Theorem 1 ] . R-OBD includes ( 4 ) as a special case , which also enjoys the optimal competitive ratio for strongly convex functions . Furthermore , Goel et al . [ 2019 , Theorem 6 ] have analyzed the dynamic regret of R-OBD and the following result can be extracted from that paper T∑ t=1 ( ft ( xt ) + 1 2 ‖xt − xt−1‖2 ) − T∑ t=1 ( ft ( ut ) + 1 2 ‖ut − ut−1‖2 ) = O √√√√T T∑ t=1 ‖ut − ut−1‖2 Compared with ( 5 ) , this bound is adaptive because the upper bound depends on the switching cost of the comparator sequence . However , it chooses the squared ` 2-norm as the switching cost , which may not be suitable for general convex functions . When the hitting cost is both quasiconvex and λquadratic growth , Goel et al . [ 2019 , Theorem 3 ] have demonstrated that their Greedy OBD algorithm attains an O ( 1/ √ λ ) competitive ratio , as λ→ 0 . Lin et al . [ 2020 ] have analyzed the naive approach which ignores the switching cost and simply minimizes the hitting cost in each round , i.e. , the greedy algorithm with γ = 0 . It is a bit surprising that this naive approach is 1 + 2α -competitive for α-polyhedral functions and max ( 1 + 6 λ , 4 ) -competitive for λ-quadratic growth functions , without any convexity assumption [ Lin et al. , 2020 , Lemma 1 ] . Argue et al . [ 2020a ] have investigated a hybrid setting in which the hitting cost is both λ-strongly convex and H-smooth , but the switching cost is the ` 2-norm distance instead of the squared one . They develop Constrained OBD , and establish a 4 + 4 √ 2H/λ competitive ratio . However , their analysis relies on a strong condition that the hitting cost is non-negative over the whole space , i.e. , minx∈Rd ft ( x ) = 0 , as opposed to the usual condition minx∈X ft ( x ) = 0 . Finally , we note that SOCO is closely related to convex body chasing ( CBC ) [ Friedman and Linial , 1993 , Antoniadis et al. , 2016 , Bansal et al. , 2018 , Argue et al. , 2019 , Bubeck et al. , 2019 , 2020 ] . In this problem , the online learner receives a sequence of convex bodies X1 , . . . , XT ⊆ Rd and must select one point from each body , and the goal is to minimize the total movement between consecutive output points . Apparently , we can treat CBC as a special case of SOCO by defining the hitting cost ft ( · ) as the indicator function of Xt , which means that the domains of hitting costs are allowed to be different . On the other hand , we can also formulate a d-dimensional SOCO problem as a d+ 1-dimensional CBC problem [ Lin et al. , 2020 , Proposition 1 ] . For the general setting of CBC , the competitive ratio exhibits a polynomial dependence on the dimensionality , and the state-of-the-art result is O ( min ( d , √ d log T ) ) [ Argue et al. , 2020b , Sellke , 2020 ] , which nearly match the Ω ( √ d ) lower bound [ Friedman and Linial , 1993 ] . Our paper aims to derive dimensionality-independent competitive ratios and sublinear dynamic regret for SOCO , under appropriate conditions .

Summary: This paper studies smoothed online learning and considers two performance metrics: the competitive ratio and the dynamic regret. For the competitive ratio part, the authors study the greedy algorithm which minimizes the weighted sum of the hitting costs and the switching cost. They improve the existing competitive ratio results by some constant factors in three different settings. For the dynamic regret part, the authors show that a small modification of Ader, SAder, can attain the same dynamic regret bound as the case without switching costs. They also show that if the agent can observe next step's hitting cost before making decisions, the Lookahead SAder algorithm can achieve the optimal dynamic regret bound without the bounded gradient assumption.  Update: I partly agree with the points made in the response. So I adjust the score to 6.