Distributed Online Optimization with Long-Term Constraints

√ T log ( T ) ) . These re- gret scalings match those obtained by state-of-the-art algorithms and fundamental limits in the corresponding centralized online optimization problem ( for both convex and strongly convex loss functions ) . In the case of bandit feedback , the proposed algorithms achieve a regret and constraint violation in O ( Tmax { c,1−c/3 } ) and O ( T 1−c/2 ) for any c ∈ ( 0 , 1 ) . We numerically illustrate the performance of our algorithms for the particular case of distributed online regularized linear regression problems . 1 INTRODUCTION . The Online Convex Optimization ( OCO ) paradigm Hazan ( 2016 ) has recently become prominent in various areas of machine learning where the environment sequentially generating data is too complex to be efficiently modeled . OCO portrays optimization as a process , and applies a robust and sequential optimization approach where one learns from experiences as time evolves . Specifically , under the OCO framework , at each time-step the learner commits to a decision and suffers from a loss , a convex function of the decision . The successive loss functions are unknown beforehand and may vary arbitrarily over time . At the end of each step , the loss function may be revealed ( a scenario referred to as full information ) . Alternatively , the experienced loss only might be available ( bandit feedback ) . The objective of the decision maker is to minimize the loss accumulated over time . The performance of an algorithm in OCO is assessed through the notion of regret , comparing the accumulated loss under the algorithm and that achieved by an Oracle always selecting the best fixed decision . In case of full information feedback , it is known that the best possible regret scales in O ( √ T ) ( resp . O ( log T ) ) for convex ( resp . strongly convex ) loss functions Zinkevich ( 2003 ) ; Hazan et al . ( 2007 ) ; Abernethy et al . ( 2009 ) . This paper extends the OCO framework to a distributed setting where ( different ) data is collected and processed at N computing units in a network . More precisely , we consider scenarios where in each time-step , each unit i commits to a decision xi ( t ) and then experiences a local loss equal to ` i , t ( xi ( t ) ) . Units update their decision based on previously observed local losses and messages received from neighboring units with the objective of identifying the decision x ? = arg minx ∑T t=1 ∑N i=1 ` i , t ( x ) minimizing the accumulated system-wide loss . Many traditional applications of the centralized OCO framework Hazan ( 2016 ) naturally extend to this distributed setting . As a motivating example , consider the following distributed online spam filtering task ( refer to Hazan ( 2016 ) for a description of the spam filter design problem in a centralized setting ) . In each time-step , each unit i ( here an email server ) receives an email characterized by a vector ai , t ∈ Rd ( according to the “ bag-of-words ” representation ) . Unit i applies for this email a filter represented by a vector xi ( t ) ∈ X where X is convex compact subset of Rd , returns a label f ( a > i , txi ( t ) ) , and experiences a loss equal to ` i , t ( xi ( t ) ) = ( f ( a > i , txi ( t ) ) − yi , t ) 2 where yi , t is the true email label ( -1 for spam or 1 for valid ) . Note that the sequences of loss functions are inherently different at various units because the latter receive different emails . Nevertheless , each unit would ideally wish to identify and apply as fast as possible the filter minimizing the system-wide loss , i.e. , a filter that exploits the knowledge extracted from all emails , including those received at other units . By leveraging this knowledge , each unit would adapt faster to an adversary also modifying in an online manner spam emails . More generally the distributed OCO framework can be applied to networks of learning agents , where each agent wishes to take advantage of what other agents have learnt to speed up and robustify its own learning process . 1.1 THE DOCO ( DISTRIBUTED ONLINE CONVEX OPTIMIZATION ) FRAMEWORK . We describe here our distributed optimization problem in more detail . We consider a network of N computing units described by a sequence of directed graphs Gt = { V , Et } with node set V = { 1 , . . . , N } and edge set Et at time t. Gt represents the communication constraints at the end of time-step t : each unit is allowed to send its decision at time t to its neighbors in Gt . Each unit i ∈ V is associated with a sequence of convex loss functions { ` i , t } Tt=1 , where ` i , t : Rd → R. Optimization process . In each time-step t , each unit i ∈ V selects xi ( t ) ∈ Rd . Then , in case of full information feedback , the loss function ` i , t is revealed to unit i , whereas in case of bandit feedback , the loss ` i , t ( xi ( t ) ) is revealed only . Unit i finally receives vectors , functions of decisions selected by its neighbors in Gt , i.e. , xj ( t ) for j such that ( j , i ) ∈ Et , and updates its decision for the next time-step . Decision constraints . The decisions should be selected in X a convex subset of Rd characterized by a family of inequalities : X = { x ∈ Rd | cs ( x ) ≤ 0 , s = 1 , . . . , p } . Imposing such constrained decisions implies that each unit should be able in each time-step to perform a projection onto X , which can be extremely computationally expensive . To circumvent this difficulty , we adopt the notion of long-term constraints introduced in Mahdavi et al . ( 2012 ) . Specifically , we only impose that the constraints are satisfied in a long run rather than in each time-step , i.e. , that∑T t=1 ∑N i=1 ∑p s=1 cs ( xi ( t ) ) ≤ 0 . This relaxation allows units to violate the constraints by projecting onto a simpler set that contains X . Our results can be modified to account for the actual constraints ( but using projection steps ) . Regrets and cumulative absolute constraint violation . The objective is to design distributed sequential decision selection algorithms so that each unit identifies the decision minimizing the accumulated system-wide loss . The performance of such an algorithm is hence captured by the regrets at the various units . The regret at unit i is : Reg ( i , T ) : = T∑ t=1 N∑ j=1 ` j , t ( xi ( t ) ) − T∑ t=1 N∑ j=1 ` j , t ( x ? ) , ( 1 ) where x ? = arg minx∈X ∑T t=1 ∑N j=1 ` j , t ( x ) . The system-level regret is defined as the worst pos- sible regret at all units : SReg ( T ) , maxi=1 , ... , N Reg ( i , T ) . Now since we allow units to select decisions outside X , the performance of an algorithm is further characterized by the so-called cumulative absolute constraint violation defined by : ( here [ a ] + = max { 0 , a } ) CACV ( T ) : = T∑ t=1 N∑ i=1 p∑ s=1 [ cs ( xi ( t ) ) ] + . ( 2 ) 1.2 MAIN RESULTS . We propose simple distributed algorithms where in each time-step , each unit combines information received from its neighbors to update its decision and its local dual variable . Our algorithms enjoy the following performance guarantees : Full Information feedback . In the case of full information feedback , the proposed algorithms achieve a system-level regret and a cumulative constraint violation in O ( Tmax { 1−c , c } ) and O ( T 1−c/2 ) , respectively and for any c ∈ ( 0 , 1 ) ( c expresses the trade-off between regret and cumulative constraint violation ) . Theses bounds match those of centralized online optimization algorithms in Mahdavi et al . ( 2012 ) ; Jenatton et al . ( 2016 ) ; Yuan & Lamperski ( 2018 ) . When c = 1/2 , we get a regret scaling in O ( √ T ) , which corresponds to the fundamental regret limits for centralized online problems Abernethy et al . ( 2009 ) , which is rather surprising in view of the dynamically changing environment , the decentralized structure of the algorithm , and the presence of the constraints . When the loss functions are strongly convex , we establish improved upper bounds on the regret and cumulative connstraint violation in O ( log ( T ) ) and O ( √ T log ( T ) ) . These bounds generalize to our distributed setting those derived in Yuan & Lamperski ( 2018 ) for centralized problems . Bandit feedback . In the case of bandit feedback , the proposed algorithms achieve a systemlevel regret and a cumulative constraint violation in O ( d2Tmax { 1−c/3 , c } ) and O ( dT 1−c/2 ) , respectively , for any c ∈ ( 0 , 1 ) . For example , when c = 34 , the proposed algorithm attains a regret bound in O ( d2T 3/4 ) . The performance guarantees can be improved to O ( d2T 2/3 log ( T ) ) and O ( d √ T log ( T ) ) in the case of strongly convex losses . 1.3 RELATED WORK . Early work on online convex optimization in a centralized setting include Zinkevich ( 2003 ) ; Flaxman et al . ( 2005 ) . Today we know that a regret in O ( √ T ) is achievable in both full information and bandit feedback , see e.g . Bubeck et al . ( 2017 ) . Projection-free algorithms have been also developed Mahdavi et al . ( 2012 ) ; Jenatton et al . ( 2016 ) ; Yuan & Lamperski ( 2018 ) with regret and cumulative constraint violation inO ( Tmax { c,1−c } ) andO ( T 1−c/2 ) ( c ∈ ( 0 , 1 ) ) in case of full information feedback ( Yuan & Lamperski ( 2018 ) uses the cumulative squared constraint violation ) . Our algorithms achieve the same guarantees in a distributed setting . It is worth zooming into the rich literature on centralized online convex optimization with bandit feedback . In the seminal work Flaxman et al . ( 2005 ) , the authors designed an algorithm with onepoint bandit feedback and regret in O ( d2T 3/4 ) . The work Agarwal et al . ( 2010 ) extended this algorithm to multi-point bandit feedback setting , where multiple points around the decision can be queried for the loss function ; they established O ( d2 √ T ) and O ( d2 log ( T ) ) regret bounds for general convex and strongly convex loss functions , respectively . The work Mahdavi et al . ( 2012 ) studied the online bandit optimization with long-term constraints under two-point bandit feedback for domain . They established O ( √ T ) and O ( d2T 3/4 ) bounds on the regret and the cumulative constraint violations , respectively . In this paper , we design distributed algorithms with one-point bandit feedback only , and with the same regret guarantees as the centralized algorithm in Flaxman et al . ( 2005 ) . Over the last few years , there have been a rising interest for the Distributed OCO framework . Particularly , Shahrampour & Jadbabaie ( 2017 ) ; Lee et al . ( 2017 ) propose distributed algorithms with O ( √ T ) regret , but require an exact projection onto the decision set in each time-step . Zhang et al . ( 2017 ) presents a distributed online conditional gradient algorithm , replacing the projection steps by a much simpler linear optimization steps , but at the expense of worse and sub-optimal regret guarantees , scaling in O ( T 3/4 ) . The other approach to avoid projections is to allow the algorithm to violate the constraints , and has been studied in Yuan et al . ( 2018 ) . The problem studied in Yuan et al . ( 2018 ) is a special case of our problem ( where only one inequality constraint is considered ) , and the regret and cumulative constraint violation guarantees obtained there are much worse than ours . The authors of Li et al . ( 2018 ) ; Yi et al . ( 2019 ) also use the long-term constraints approach to avoid projections , but analyze a very different optimization problem where units have different decision variables , and no consensus among units is required . Finally it is worth mentioning that all the aforementioned papers are restricted to full information feedback . Notation and Terminology . Let ‖x‖ and [ x ] i to denote the Euclidean norm and the ith component of a vector x ∈ Rd , respectively . Let ΠX [ x ] be the Euclidean projection of a vector x onto the set X . Let Rp+ be the nonnegative orthant in Rp : R p + = { x ∈ Rp | [ x ] i ≥ 0 , i = 1 , . . . , p } . Denote the ( i , j ) -th element of a matrix A by [ A ] ij . For a convex function f , a subgradient ( resp . gradient when f is differentiable ) at a point x is denoted by ∂f ( x ) ( resp . ∇f ( x ) ) . Given two positive sequences { at } ∞t=1 and { bt } ∞t=1 , we write at = O ( bt ) if lim supt→∞ at/bt < ∞ .

Summary: This paper considers distributed online convex optimization with long-term constraints, which extends Yuan & Lamperski (2018)’s work to decentralized case with time-varying directed network. The authors propose DOCO frameworks (full-information and one-point bandit feedback) based on augmented Lagrangian functions. They also provide the corresponding regret bounds for both strongly and non-strongly convex cases. The experiments on synthetic data validate the effectiveness of proposed algorithms.