Online Convex Optimization with Continuous Switching Constraint

p T ) when the switching cost budget S = ⌦ ( p T ) , and ⌦ ( min { T/S , T } ) when S = O ( p T ) , where T is the time horizon . The essential idea is to carefully design an adaptive adversary , who can adjust the loss function according to the cumulative switching cost of the player incurred so far based on the orthogonal technique . We then develop a simple gradient-based algorithm which enjoys the minimax optimal regret bound . Finally , we show that , for strongly convex functions , the regret bound can be improved to O ( log T ) for S = ⌦ ( log T ) , and O ( min { T/ exp ( S ) + S , T } ) for S = O ( log T ) . 1 Introduction . Online convex optimization ( OCO ) is a fundamental framework for studying sequential decision making problems ( Shalev-Shwartz , 2011 ) . Its protocol can be seen as a game between a player and an adversary : In each round t , firstly , the player selects an action wt from a convex set D ✓ Rd . After submitting the answer , a loss function ft : D 7 ! R is revealed , and the player suffers a loss ft ( wt ) . The goal is to minimize the regret : R = TX t=1 ft ( wt ) min w2D TX t=1 ft ( w ) , ( 1 ) which is the difference between the cumulative loss of the player and that of the best action in hindsight . Over the past decades , the problem of OCO has been extensively studied , yielding various algorithms and theoretical guarantees ( Hazan , 2016 ; Orabona , 2019 ) . However , most of the existing approaches allow the player to switch her action freely during the learning process . As a result , these methods become unsuitable for many real-life scenarios , such as the online shortest paths problem ( Koolen et al. , 2010 ) , and portfolio management ( Dekel et al. , 2014 ; Vittori et al. , 2020 ) , where the switching of actions brings extra cost , and the budget for the overall switching cost is strictly ⇤Lijun Zhang is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . constrained . To address this problem , recent advances in OCO introduced the switching-constrained problem ( Altschuler & Talwar , 2018 ; Chen et al. , 2020 ) , where a hard constraint is imposed to the number of the player ’ s action shifts , i.e. , TX t=2 { wt 6= wt 1 } K , ( 2 ) and the goal is to minimize regret under a fixed budget K. For this problem , Chen et al . ( 2020 ) have shown that , given any K , we could precisely control the the overall switching cost in ( 2 ) , while achieving a minimax regret bound of order ⇥ ( T/ p K ) . One limitation of ( 2 ) is that it treats different amounts of changes between wt 1 and wt equally , since the binary function is used as the penalty for action shifts . However , as observed by many practical applications , e.g. , thermal management ( Zanini et al. , 2010 ) , video streaming ( Joseph & de Veciana , 2012 ) and multi-timescale control ( Goel et al. , 2017 ) , the price paid for large and small action changes are not the same . Specifically , for these scenarios , the switching cost between two consecutive rounds is typically characterized by a ` 2-norm function , i.e. , kwt wt 1k . Motivated by this observation , in this paper , we introduce a novel OCO setting , named OCO with continuous switching constraint ( OCO-CSC ) , where the player needs to choose actions under a hard constraint on the overall ` 2-norm switching cost , i.e. , TX t=2 kwt wt 1k S , ( 3 ) where S is a budget given by the environment . The main advantage of OCO-CSC is that , equipped with ( 3 ) , we could have a more delicate control on the overall switching cost compared to the binary constraint in ( 2 ) . For the proposed problem , we firstly observe that , an O ( T/ p S ) regret bound can be achieved by using the method proposed for the switching-constrained OCO ( Chen et al. , 2020 ) under a proper configuration of K. However , this bound is not tight , since there is a large gap from the lower bound established in this paper . Specifically , we provide a lower bound of order ⌦ ( p T ) when S = ⌦ ( p T ) , and ⌦ ( min { TS , T } ) when S = O ( p T ) . Our basic framework for constructing the lower bound follows the classical linear game ( Abernethy et al. , 2008 ) , while we adopt a novel continuous-constraint-related dynamic policy for the adversary , which allows it to adaptively change the loss function according to the player ’ s cumulative switching costs . Furthermore , we prove that the classical online gradient descent ( OGD ) with an appropriately chosen step size is able to obtain the matching upper bound . These results demonstrate that there is a phase transition phenomenon between large and small switching budget regimes , which is in sharp contrast to the switchingconstrained setting , where the minimax bound always decreases with ⇥ ( 1/ p K ) . Finally , we propose a variant of OGD for -strongly convex functions , which can achieve an O ( log T ) regret bound when S = ⌦ ( log T ) , and an O ( T/ exp ( S ) + S ) regret bound when S = O ( log T ) . 2 Related Work . In this section , we briefly review related work on online convex optimization . 2.1 Classical OCO . The framework of OCO is established by the seminal work of Zinkevich ( 2003 ) . For general convex functions , Zinkevich ( 2003 ) shows that online gradient descent ( OGD ) with step size on the order of O ( 1/ p t ) enjoys an O ( p T ) regret bound . For -strongly convex functions , Hazan et al . ( 2007 ) prove that OGD with step size of order O ( 1/ [ t ] ) achieves an O ( log T ) regret bound . Both bounds have been proved to be minimax optimal ( Abernethy et al. , 2008 ) . For exponentially concave functions , the state-of-the-art algorithm is online Newton step ( Hazan et al. , 2007 ) , which enjoys an O ( d log T ) regret bound , where d is the dimensionality . 2.2 Switching-constrained OCO . One related line of research is the switching-constrained setting , where the player is only allowed to change her action no more than K times . This setting has been studied in various online learning scenarios , such as prediction with expert advice ( Altschuler & Talwar , 2018 ) and bandits problems ( Simchi-Levi & Xu , 2019 ; Dong et al. , 2020 ; Ruan et al. , 2021 ) . In this paper , we focus on online convex optimization . Jaghargh et al . ( 2019 ) firstly consider this problem , and develop a novel online algorithm based on the Poison Process , which can achieve an expected regret of order O ( T 3/2/E [ K ] ) for any given expected switching budget E [ K ] . Therefore , the regret will become sublinear for E [ K ] = o ( p T ) . Later , Chen et al . ( 2020 ) propose a variant of the classical OGD based on the mini-batch approach . Specifically , the algorithm averagely divides the time horizon into K intervals , and only update the decision at the end of each interval based on OGD . The show that the simple algorithm enjoys an O ( T/ p K ) regret bound for any given budget K. They also prove that this result is minimax optimal by establishing a matching ⌦ ( T/ p K ) lower bound . We note that , when the action set is bounded ( i.e. , maxw1 , w22D kw1 w2k D ) , since TX t=2 kwt wt 1k DK , we could set K = bS/Dc to satisfy ( 3 ) and immediately obtain an O ( T/ p S ) regret for OCOCSC , but there is still a large gap from the lower bound we provide in this paper . Very recently , a concurrent work ( Sherman & Koren , 2021 ) considered switching-constrained OCO under oblivious adversary setting , and derived ⇥ ( T/S ) bounds . However , we note that : ( i ) lower bound for the switching-constraint OCO does not translate to a lower bound for our continuous switching cost setting ; and ( ii ) their upper bound for oblivious only holds in expectation , and has an undesirablep d factor . 2.3 OCO with Ramp Constraints . Another related setting is OCO with ramp constraints , which is studied by Badiei et al . ( 2015 ) . In this setting , at each round , the player must choose an action satisfying the following inequality : |wt , i wt 1 , i| Xi , ( 4 ) where wt , i denotes the i-th dimension of wt , and Xi is a constant factor . The constraint in ( 4 ) limits the player ’ s action switching in a per-round and per-dimension level . This is very different from the constraint we proposed in ( 3 ) , which mainly focus on the long-term and overall switching cost . Moreover , we note that , Badiei et al . ( 2015 ) assume the player could get access to a sequence of future loss functions before choosing wt , while in this paper we follow the classical OCO framework in which the player can only make use of the historical data . 2.4 OCO with Long-term Constraints . Our proposed problem is also related to OCO with long-term constraints ( Mahdavi et al. , 2012 ; Jenatton et al. , 2016 ; Yu et al. , 2017 ) , where the action set is written as m convex constraints , i.e. , D = { w 2 Rd : gi ( w ) 0 , i 2 [ m ] } , ( 5 ) and we only require these constraints to be satisfied in the long term , i.e. , PT t=1 gi ( wt ) 0 , i 2 [ m ] . The goal is to minimize regret while keeping PT t=1 gi ( wt ) small . We note that , in this setting , the action set is expressed by the constraint , which is in contrast to OCO-CSC , where the constraint and the decision set D are independent . Moreover , the constraint in OCO-CSC is time-variant and decided by the historical decisions , while the constraint in ( 5 ) is static ( or stochastic , considered by Yu et al. , 2017 ) . Recently , several work start to investigate OCO with long-term and time-variant constraints , but this task is proved to be impossible in general ( Mannor et al. , 2009 ) . Therefore , existing studies have to consider more restricted settings , such as weaker definitions of regret ( Neely & Yu , 2017 ; Liakopoulos et al. , 2019 ; Yi et al. , 2020 ; Valls et al. , 2020 ) .

Summary: This paper studies online convex optimization with continuous switching constraints. The objective of the online agent is to minimize the static regret while keeping the total switching cost within some budget $S$. Under some standard assumptions, the authors establish a lower bound of the regret for all online algorithms by designing the adversary’s policy. They also show that the classical online gradient descent algorithm can match this lower bound if the step sizes are chosen appropriately. In addition, under stronger assumptions on the loss functions, the authors show the regret can be further improved to $O(\log{T})$.