Revisiting Smoothed Online Learning

4 λ -competitive for λ-quadratic growth functions , both of which improve state-of-the-art results significantly . Moreover , when the hitting cost is both convex and λ-quadratic growth , we reduce the competitive ratio to 1 + 2√ λ by minimizing the weighted sum of the hitting cost and the switching cost . To bound the dynamic regret with switching cost , we follow the standard setting of online convex optimization , in which the hitting cost is convex but hidden from the learner before making predictions . We modify Ader , an existing algorithm designed for dynamic regret , slightly to take into account the switching cost when measuring the performance . The proposed algorithm , named as Smoothed Ader , attains an optimal O ( √ T ( 1 + PT ) ) bound for dynamic regret with switching cost , where PT is the path-length of the comparator sequence . Furthermore , if the hitting cost is accessible in the beginning of each round , we obtain a similar guarantee without the bounded gradient condition , and establish an Ω ( √ T ( 1 + PT ) ) lower bound to confirm the optimality . 1 Introduction . Online learning is the process of making a sequence of predictions given knowledge of the answer to previous tasks and possibly additional information [ Shalev-Shwartz , 2011 ] . While the traditional online learning aims to make the prediction as accurate as possible , in this paper , we study smoothed online learning ( SOL ) , where the online learner incurs a switching cost for changing its predictions between rounds [ Cesa-Bianchi et al. , 2013 ] . SOL has received lots of attention recently because in many real-world applications , a change of action usually brings some additional cost . Examples include the dynamic right-sizing for data centers [ Lin et al. , 2011 ] , geographical load balancing [ Lin et al. , 2012 ] , real-time electricity pricing [ Kim and Giannakis , 2014 ] , video streaming [ Joseph and de Veciana , 2012 ] , spatiotemporal sequence prediction [ Kim et al. , 2015 ] , multi-timescale control [ Goel et al. , 2017 ] , and thermal management [ Zanini et al. , 2010 ] . Specifically , SOL is performed in a sequence of consecutive rounds , where at round t the learner is asked to select a point xt from the decision set X , and suffers a hitting cost ft ( xt ) . Depending on the performance metric , the learner may be allowed to observe ft ( · ) when making decisions , which is different from the traditional online learning in which ft ( · ) is revealed to the learner after submitting the decision [ Cesa-Bianchi and Lugosi , 2006 ] . Additionally , the learner also incurs a switching cost m ( xt , xt−1 ) for changing decisions between successive rounds . The switching cost m ( xt , xt−1 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . could be any distance function , such as the ` 2-norm distance ‖xt − xt−1‖ and the squared ` 2-norm distance ‖xt − xt−1‖2/2 [ Goel et al. , 2019 ] . In the literature , there are two performance metrics for SOL : competitive ratio and dynamic regret with switching cost . Competitive ratio is popular in the community of online algorithms [ Borodin and El-Yaniv , 1998 ] . It is defined as the worst-case ratio of the total cost incurred by the online learner and the offline optimal cost : ∑T t=1 ( ft ( xt ) +m ( xt , xt−1 ) ) minu0 , u1 , ... , uT∈X ∑T t=1 ( ft ( ut ) +m ( ut , ut−1 ) ) . ( 1 ) When focusing on the competitive ratio , the learner can observe ft ( · ) before picking xt . The problem is still nontrivial due to the coupling created by the switching cost . On the other hand , dynamic regret with switching cost is a generalization of dynamic regret—a popular performance metric in the community of online learning [ Zinkevich , 2003 ] . It is defined as the difference between the total cost incurred by the online learner and that of an arbitrary comparator sequence u0 , u1 , . . . , uT ∈ X : T∑ t=1 ( ft ( xt ) +m ( xt , xt−1 ) ) − T∑ t=1 ( ft ( ut ) +m ( ut , ut−1 ) ) . ( 2 ) Different from previous work [ Chen et al. , 2018 , Goel et al. , 2019 ] , we did not introduce the minimization operation over u0 , u1 , . . . , uT in ( 2 ) . The reason is that we want to bound ( 2 ) by certain regularities of the comparator sequence , such as the path-length PT ( u0 , u1 , . . . , uT ) = T∑ t=1 ‖ut − ut−1‖ . ( 3 ) When focusing on ( 2 ) , ft ( · ) is generally hidden from the learner before submitting xt . The conditions for bounding the two metrics are very different , so we study competitive ratio and dynamic regret with switching cost separately . To bound the two metrics simultaneously , we refer to Andrew et al . [ 2013 ] and Daniely and Mansour [ 2019 ] , especially the meta-algorithm in the latter work . This paper follows the line of research stemmed from online balanced descent ( OBD ) [ Chen et al. , 2018 , Goel and Wierman , 2019 ] . The key idea of OBD is to find an appropriate balance between the hitting cost and the switching cost through iterative projections . It has been shown that OBD and its variants are able to exploit the analytical properties of the hitting cost ( e.g. , polyhedral , strongly convex ) to derive dimension-free competitive ratio . At this point , it would be natural to ask why not use the greedy algorithm , which minimizes the weighted sum of the hitting cost and the switching cost in each round , i.e. , min x∈X ft ( x ) + γm ( x , xt−1 ) ( 4 ) to balance the two costs , where γ ≥ 0 is the trade-off parameter . We note that the greedy algorithm is usually treated as the baseline in competitive analysis [ Borodin and El-Yaniv , 1998 ] , but its usage for smoothed online learning is quite limited . One result is given by Goel et al . [ 2019 ] , who demonstrate that the greedy algorithm as a special case of Regularized OBD ( R-OBD ) , is optimal for strongly convex functions . Besides , Lin et al . [ 2020 ] have analyzed the greedy algorithm with γ = 0 , named as the naive approach below , for polyhedral functions and quadratic growth functions . In this paper , we make the following contributions towards understanding the greedy algorithm . • For α-polyhedral functions , the competitive ratio of the naive approach is max ( 1 , 2α ) , which is a significant improvement over the 3 + 8α competitive ratio of OBD [ Chen et al. , 2018 ] and the 1 + 2α ratio proved by Lin et al . [ 2020 , Lemma 1 ] . When α > 2 , the ratio becomes 1 , indicating that the naive approach is optimal in this scenario . • For λ-quadratic growth functions , the competitive ratio of the naive algorithm is 1 + 4λ , which matches the lower bound of this algorithm [ Goel et al. , 2019 , Theorem 5 ] , and is better than the max ( 1 + 6λ , 4 ) ratio obtained by Lin et al . [ 2020 , Lemma 1 ] . • If the hitting cost is both convex and λ-quadratic growth , the greedy algorithm with γ > 0 attains a 1+ 2√ λ competitive ratio , which demonstrates the advantage of taking the switching cost into considerations . Our 1 + 2√ λ ratio is on the same order as Greedy OBD [ Goel et al. , 2019 , Theorem 3 ] but with much smaller constants . • Our analysis of the naive approach and the greedy algorithm is very simple . In contrast , both OBD and Greedy OBD rely on intricate geometric arguments . While both OBD and R-OBD are equipped with sublinear dynamic regret with switching cost , they are unsatisfactory in the following aspects : • The regret of OBD depends on an upper bound of the path-length instead of the path-length itself [ Chen et al. , 2018 , Corollary 11 ] , making it nonadaptive . • The regret of R-OBD is adaptive but it uses the squared ` 2-norm to measure the switching cost , which may not be suitable for general convex functions [ Goel et al. , 2019 ] .1 • Both OBD and R-OBD observe ft ( · ) before selecting xt , which violates the convention of online learning . To avoid the above limitations , we demonstrate that a small change of Ader [ Zhang et al. , 2018a ] , which is an existing algorithm designed for dynamic regret , is sufficient to minimize the dynamic regret with switching cost under the setting of online convex optimization [ Shalev-Shwartz , 2011 ] . Ader runs multiple online gradient descent ( OGD ) [ Zinkevich , 2003 ] with different step sizes as expert-algorithms , and uses Hedge [ Freund and Schapire , 1997 ] as the meta-algorithm to aggregate predictions from experts . The only modification is to incorporate the switching cost into the loss of Hedge . The proposed algorithm , named as Smoothed Ader ( SAder ) , attains the optimal O ( √ T ( 1 + PT ) ) dynamic regret , where PT is the path-length defined in ( 3 ) . Thus , our regret bound is adaptive because it automatically becomes small when the comparators change slowly . Finally , we also investigate the case that the hitting cost is available before predictions , and establish a similar result without the bounded gradient condition . To this end , we design a lookahead version of SAder , which chooses the greedy algorithm in ( 4 ) as the expert and utilizes the cost of the current round in Hedge . To show the optimality of this algorithm , we further establish an Ω ( √ T ( 1 + PT ) ) lower bound under the lookahead setting . 2 Related work . This section reviews related work on smoothed online learning ( SOL ) and dynamic regret . 2.1 Smoothed online learning . SOL has been investigated under the setting of multi-armed bandits [ Agrawal et al. , 1990 , Guha and Munagala , 2009 , Dekel et al. , 2014 , Koren et al. , 2017a , b ] , prediction with expert advice [ CesaBianchi et al. , 2013 ] , and online convex optimization [ Lin et al. , 2011 , Bansal et al. , 2015 , Zhao et al. , 2020b ] . In the following , we mainly discuss smoothed online convex optimization ( SOCO ) . The early works on SOCO focus on designing competitive algorithms in the low-dimensional setting [ Lin et al. , 2011 ] . In particular , Bansal et al . [ 2015 ] show that for SOCO on the real line , the competitive ratio can be upper bounded by 2 , which is proved to be optimal [ Antoniadis and Schewior , 2018 ] . They also establish a competitive ratio of 3 under the memoryless setting . In the study of SOCO , it is common to assume that the learner has access to predictions of future hitting costs , and several algorithms [ Lin et al. , 2012 , Chen et al. , 2015 , 2016 , Li et al. , 2018 , Li and Li , 2020 ] have been developed based on receding horizon control ( RHC ) [ Kwon and Han , 2005 ] . In fact , the greedy algorithm in ( 4 ) can be treated as a variant of RHC . However , previous results for RHC are limited to special problems , and ( 4 ) remains under-explored . For example , when the learner can observe the next W hitting costs , Li et al . [ 2018 ] demonstrate that both the competitive ratio and the dynamic regret with switching cost decay exponentially fast with W . But their analysis relies on very strong conditions , including strong convexity and smoothness . One milestone is the online balanced descent ( OBD ) [ Chen et al. , 2018 ] , which has dimensionfree competitive ratio even when the learner can only observe the hitting cost of the current round . Specifically , OBD iteratively projects the previous point onto a carefully chosen level set of the hitting cost so as to balance the switching cost and the hitting cost . When the hitting cost is α-polyhedral and convex , and the switching cost is the ` 2-norm distance , OBD attains a 3 + 8α competitive ratio . Furthermore , OBD can also be tuned to control the dynamic regret with switching cost . Let L be 1We usually assume the convex function is Lipschitz continuous , and in this case choosing the ` 2-norm distance as the switching cost makes it on the same order as the hitting cost . an upper bound of the path-length of the comparator sequence , i.e. , PT ( u0 , u1 , . . . , uT ) ≤ L. Chen et al . [ 2018 , Corollary 11 ] have proved that T∑ t=1 ( ft ( xt ) + ‖xt − xt−1‖ ) − min PT ( u0 , u1 , ... , uT ) ≤L T∑ t=1 ( ft ( ut ) + ‖ut − ut−1‖ ) = O ( √ TL ) ( 5 ) leading to sublinear regret when L = o ( T ) . However , the upper bound is nonadaptive because it depends on L instead of the actual path-length PT . Later , Goel and Wierman [ 2019 ] demonstrate that OBD is 3 + O ( 1λ ) -competitive for λ-strongly convex functions , when the switching cost is set to be the squared ` 2-norm distance . In a subsequent work , Goel et al . [ 2019 , Theorem 4 ] propose Regularized OBD ( R-OBD ) , which improves the competitive ratio to 12 + 1 2 √ 1 + 4λ , matching the lower bound of strongly convex functions exactly [ Goel et al. , 2019 , Theorem 1 ] . R-OBD includes ( 4 ) as a special case , which also enjoys the optimal competitive ratio for strongly convex functions . Furthermore , Goel et al . [ 2019 , Theorem 6 ] have analyzed the dynamic regret of R-OBD and the following result can be extracted from that paper T∑ t=1 ( ft ( xt ) + 1 2 ‖xt − xt−1‖2 ) − T∑ t=1 ( ft ( ut ) + 1 2 ‖ut − ut−1‖2 ) = O √√√√T T∑ t=1 ‖ut − ut−1‖2 Compared with ( 5 ) , this bound is adaptive because the upper bound depends on the switching cost of the comparator sequence . However , it chooses the squared ` 2-norm as the switching cost , which may not be suitable for general convex functions . When the hitting cost is both quasiconvex and λquadratic growth , Goel et al . [ 2019 , Theorem 3 ] have demonstrated that their Greedy OBD algorithm attains an O ( 1/ √ λ ) competitive ratio , as λ→ 0 . Lin et al . [ 2020 ] have analyzed the naive approach which ignores the switching cost and simply minimizes the hitting cost in each round , i.e. , the greedy algorithm with γ = 0 . It is a bit surprising that this naive approach is 1 + 2α -competitive for α-polyhedral functions and max ( 1 + 6 λ , 4 ) -competitive for λ-quadratic growth functions , without any convexity assumption [ Lin et al. , 2020 , Lemma 1 ] . Argue et al . [ 2020a ] have investigated a hybrid setting in which the hitting cost is both λ-strongly convex and H-smooth , but the switching cost is the ` 2-norm distance instead of the squared one . They develop Constrained OBD , and establish a 4 + 4 √ 2H/λ competitive ratio . However , their analysis relies on a strong condition that the hitting cost is non-negative over the whole space , i.e. , minx∈Rd ft ( x ) = 0 , as opposed to the usual condition minx∈X ft ( x ) = 0 . Finally , we note that SOCO is closely related to convex body chasing ( CBC ) [ Friedman and Linial , 1993 , Antoniadis et al. , 2016 , Bansal et al. , 2018 , Argue et al. , 2019 , Bubeck et al. , 2019 , 2020 ] . In this problem , the online learner receives a sequence of convex bodies X1 , . . . , XT ⊆ Rd and must select one point from each body , and the goal is to minimize the total movement between consecutive output points . Apparently , we can treat CBC as a special case of SOCO by defining the hitting cost ft ( · ) as the indicator function of Xt , which means that the domains of hitting costs are allowed to be different . On the other hand , we can also formulate a d-dimensional SOCO problem as a d+ 1-dimensional CBC problem [ Lin et al. , 2020 , Proposition 1 ] . For the general setting of CBC , the competitive ratio exhibits a polynomial dependence on the dimensionality , and the state-of-the-art result is O ( min ( d , √ d log T ) ) [ Argue et al. , 2020b , Sellke , 2020 ] , which nearly match the Ω ( √ d ) lower bound [ Friedman and Linial , 1993 ] . Our paper aims to derive dimensionality-independent competitive ratios and sublinear dynamic regret for SOCO , under appropriate conditions .

Summary: This paper studies smoothed online learning by targeting two performance metrics: competitive ratio and dynamic regret with switching cost. The paper first improves the competitive ratio of the naive approach and the greedy algorithm. Then the paper proposes a novel algorithm called SAder which achieves optimal dynamic regret with switching cost under the lookahead setting.