Online Convex Optimization with Continuous Switching Constraint

p T ) when the switching cost budget S = ⌦ ( p T ) , and ⌦ ( min { T/S , T } ) when S = O ( p T ) , where T is the time horizon . The essential idea is to carefully design an adaptive adversary , who can adjust the loss function according to the cumulative switching cost of the player incurred so far based on the orthogonal technique . We then develop a simple gradient-based algorithm which enjoys the minimax optimal regret bound . Finally , we show that , for strongly convex functions , the regret bound can be improved to O ( log T ) for S = ⌦ ( log T ) , and O ( min { T/ exp ( S ) + S , T } ) for S = O ( log T ) . 1 Introduction . Online convex optimization ( OCO ) is a fundamental framework for studying sequential decision making problems ( Shalev-Shwartz , 2011 ) . Its protocol can be seen as a game between a player and an adversary : In each round t , firstly , the player selects an action wt from a convex set D ✓ Rd . After submitting the answer , a loss function ft : D 7 ! R is revealed , and the player suffers a loss ft ( wt ) . The goal is to minimize the regret : R = TX t=1 ft ( wt ) min w2D TX t=1 ft ( w ) , ( 1 ) which is the difference between the cumulative loss of the player and that of the best action in hindsight . Over the past decades , the problem of OCO has been extensively studied , yielding various algorithms and theoretical guarantees ( Hazan , 2016 ; Orabona , 2019 ) . However , most of the existing approaches allow the player to switch her action freely during the learning process . As a result , these methods become unsuitable for many real-life scenarios , such as the online shortest paths problem ( Koolen et al. , 2010 ) , and portfolio management ( Dekel et al. , 2014 ; Vittori et al. , 2020 ) , where the switching of actions brings extra cost , and the budget for the overall switching cost is strictly ⇤Lijun Zhang is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . constrained . To address this problem , recent advances in OCO introduced the switching-constrained problem ( Altschuler & Talwar , 2018 ; Chen et al. , 2020 ) , where a hard constraint is imposed to the number of the player ’ s action shifts , i.e. , TX t=2 { wt 6= wt 1 } K , ( 2 ) and the goal is to minimize regret under a fixed budget K. For this problem , Chen et al . ( 2020 ) have shown that , given any K , we could precisely control the the overall switching cost in ( 2 ) , while achieving a minimax regret bound of order ⇥ ( T/ p K ) . One limitation of ( 2 ) is that it treats different amounts of changes between wt 1 and wt equally , since the binary function is used as the penalty for action shifts . However , as observed by many practical applications , e.g. , thermal management ( Zanini et al. , 2010 ) , video streaming ( Joseph & de Veciana , 2012 ) and multi-timescale control ( Goel et al. , 2017 ) , the price paid for large and small action changes are not the same . Specifically , for these scenarios , the switching cost between two consecutive rounds is typically characterized by a ` 2-norm function , i.e. , kwt wt 1k . Motivated by this observation , in this paper , we introduce a novel OCO setting , named OCO with continuous switching constraint ( OCO-CSC ) , where the player needs to choose actions under a hard constraint on the overall ` 2-norm switching cost , i.e. , TX t=2 kwt wt 1k S , ( 3 ) where S is a budget given by the environment . The main advantage of OCO-CSC is that , equipped with ( 3 ) , we could have a more delicate control on the overall switching cost compared to the binary constraint in ( 2 ) . For the proposed problem , we firstly observe that , an O ( T/ p S ) regret bound can be achieved by using the method proposed for the switching-constrained OCO ( Chen et al. , 2020 ) under a proper configuration of K. However , this bound is not tight , since there is a large gap from the lower bound established in this paper . Specifically , we provide a lower bound of order ⌦ ( p T ) when S = ⌦ ( p T ) , and ⌦ ( min { TS , T } ) when S = O ( p T ) . Our basic framework for constructing the lower bound follows the classical linear game ( Abernethy et al. , 2008 ) , while we adopt a novel continuous-constraint-related dynamic policy for the adversary , which allows it to adaptively change the loss function according to the player ’ s cumulative switching costs . Furthermore , we prove that the classical online gradient descent ( OGD ) with an appropriately chosen step size is able to obtain the matching upper bound . These results demonstrate that there is a phase transition phenomenon between large and small switching budget regimes , which is in sharp contrast to the switchingconstrained setting , where the minimax bound always decreases with ⇥ ( 1/ p K ) . Finally , we propose a variant of OGD for -strongly convex functions , which can achieve an O ( log T ) regret bound when S = ⌦ ( log T ) , and an O ( T/ exp ( S ) + S ) regret bound when S = O ( log T ) . 2 Related Work . In this section , we briefly review related work on online convex optimization . 2.1 Classical OCO . The framework of OCO is established by the seminal work of Zinkevich ( 2003 ) . For general convex functions , Zinkevich ( 2003 ) shows that online gradient descent ( OGD ) with step size on the order of O ( 1/ p t ) enjoys an O ( p T ) regret bound . For -strongly convex functions , Hazan et al . ( 2007 ) prove that OGD with step size of order O ( 1/ [ t ] ) achieves an O ( log T ) regret bound . Both bounds have been proved to be minimax optimal ( Abernethy et al. , 2008 ) . For exponentially concave functions , the state-of-the-art algorithm is online Newton step ( Hazan et al. , 2007 ) , which enjoys an O ( d log T ) regret bound , where d is the dimensionality . 2.2 Switching-constrained OCO . One related line of research is the switching-constrained setting , where the player is only allowed to change her action no more than K times . This setting has been studied in various online learning scenarios , such as prediction with expert advice ( Altschuler & Talwar , 2018 ) and bandits problems ( Simchi-Levi & Xu , 2019 ; Dong et al. , 2020 ; Ruan et al. , 2021 ) . In this paper , we focus on online convex optimization . Jaghargh et al . ( 2019 ) firstly consider this problem , and develop a novel online algorithm based on the Poison Process , which can achieve an expected regret of order O ( T 3/2/E [ K ] ) for any given expected switching budget E [ K ] . Therefore , the regret will become sublinear for E [ K ] = o ( p T ) . Later , Chen et al . ( 2020 ) propose a variant of the classical OGD based on the mini-batch approach . Specifically , the algorithm averagely divides the time horizon into K intervals , and only update the decision at the end of each interval based on OGD . The show that the simple algorithm enjoys an O ( T/ p K ) regret bound for any given budget K. They also prove that this result is minimax optimal by establishing a matching ⌦ ( T/ p K ) lower bound . We note that , when the action set is bounded ( i.e. , maxw1 , w22D kw1 w2k D ) , since TX t=2 kwt wt 1k DK , we could set K = bS/Dc to satisfy ( 3 ) and immediately obtain an O ( T/ p S ) regret for OCOCSC , but there is still a large gap from the lower bound we provide in this paper . Very recently , a concurrent work ( Sherman & Koren , 2021 ) considered switching-constrained OCO under oblivious adversary setting , and derived ⇥ ( T/S ) bounds . However , we note that : ( i ) lower bound for the switching-constraint OCO does not translate to a lower bound for our continuous switching cost setting ; and ( ii ) their upper bound for oblivious only holds in expectation , and has an undesirablep d factor . 2.3 OCO with Ramp Constraints . Another related setting is OCO with ramp constraints , which is studied by Badiei et al . ( 2015 ) . In this setting , at each round , the player must choose an action satisfying the following inequality : |wt , i wt 1 , i| Xi , ( 4 ) where wt , i denotes the i-th dimension of wt , and Xi is a constant factor . The constraint in ( 4 ) limits the player ’ s action switching in a per-round and per-dimension level . This is very different from the constraint we proposed in ( 3 ) , which mainly focus on the long-term and overall switching cost . Moreover , we note that , Badiei et al . ( 2015 ) assume the player could get access to a sequence of future loss functions before choosing wt , while in this paper we follow the classical OCO framework in which the player can only make use of the historical data . 2.4 OCO with Long-term Constraints . Our proposed problem is also related to OCO with long-term constraints ( Mahdavi et al. , 2012 ; Jenatton et al. , 2016 ; Yu et al. , 2017 ) , where the action set is written as m convex constraints , i.e. , D = { w 2 Rd : gi ( w ) 0 , i 2 [ m ] } , ( 5 ) and we only require these constraints to be satisfied in the long term , i.e. , PT t=1 gi ( wt ) 0 , i 2 [ m ] . The goal is to minimize regret while keeping PT t=1 gi ( wt ) small . We note that , in this setting , the action set is expressed by the constraint , which is in contrast to OCO-CSC , where the constraint and the decision set D are independent . Moreover , the constraint in OCO-CSC is time-variant and decided by the historical decisions , while the constraint in ( 5 ) is static ( or stochastic , considered by Yu et al. , 2017 ) . Recently , several work start to investigate OCO with long-term and time-variant constraints , but this task is proved to be impossible in general ( Mannor et al. , 2009 ) . Therefore , existing studies have to consider more restricted settings , such as weaker definitions of regret ( Neely & Yu , 2017 ; Liakopoulos et al. , 2019 ; Yi et al. , 2020 ; Valls et al. , 2020 ) .

Summary: The authors study online convex optimization with an L2 switching budget (S) on the actions chosen by the online algorithm. The authors propose a modification of the orthogonal technqiue of Abernathy et al to incorporate the switching constraint and propose a new lower bound on the regret achievable for this problem. The authors show a phase transition in the achievable regret for different values of S. The authors show that by setting the step size appropriately, vanilla OGD can match the lower bounds they present.