Distributed Bandit Learning: Near-Optimal Regret with Efficient Communication

1 INTRODUCTION . Bandit learning is a central topic in online learning , and has various real-world applications , including clinical trials ( Wang , 1991 ) , model selection ( Maron & Moore , 1994 ) and recommendation systems ( Agarwal et al. , 2009 ; Li et al. , 2010 ; Abe et al. , 2003 ) . In many tasks using bandit algorithms , it is appealing to employ more agents to learn collaboratively and concurrently in order to speed up the learning process . In many other tasks , the sequential decision making is distributed by nature . For instance , multiple spatially separated labs may be working on the same clinical trial . In such distributed applications , communication between agents is critical , but may also be expensive or time-consuming . Another example is a recommendation system deployed on multiple servers to handle high demand . Since the communication between servers may cause service latency , it would be desirable to design communication strategies without communicating too much . This motivates us to consider efficient protocols for distributed learning in bandit problems . A straightforward communication protocol for bandit learning is immediate sharing : each agent shares every new sample immediately with others . Under this scheme , agents can have good collaborative behaviors close to that in a centralized setting . However , the amount of communicated data is directly proportional to the total size of collected samples . When the bandit is played for a long timescale , the cost of communication would render this scheme impractical . A natural question to ask is : How ∗These two authors contributed equally . much communication is actually needed for near-optimal performance ? In this work , we show that the answer is somewhat surprising : The required communication cost has almost no dependence on the time horizon . In this paper , we consider the distributed learning of stochastic multi-armed bandits ( MAB ) and stochastic linear bandits . There are M agents interacting with the same bandit instance in a synchronized fashion . In time steps t = 1 , · · · , T , each agent pulls an arm and observes the associated reward . Between time steps , agents can communicate via a server-agent network . Following the typical formulation of single-agent bandit learning , we consider the task of regret minimization ( Lai et al. , 1987 ; Dani et al. , 2008 ; Bubeck et al. , 2012 ) . The total regret of all agents is used as the performance criterion of a communication protocol . The communication cost is measured by the total amount of data communicated in the network . Our goal is to minimize communication cost while maintaining near-optimal performance , that is , regret comparable to the optimal regret of a single agent in MT interactions with the bandit instance . For multi-armed bandits , we propose the DEMAB protocol , which achieves near-optimal regret . The amount of transmitted data per agent in DEMAB is independent of T , and is logarithmic with respect to other parameters . For linear bandits , we propose the DELB protocol , which achieves near-optimal regret , and has communication cost with at most logarithmic dependence on T . 1.1 PROBLEM SETTING . Communication Model The communication model we consider consists of a server and several agents . Agents can communicate with the server by sending or receiving packets . Each data packet contains integers or real numbers . We define the communication cost of a protocol as the number of integers or real numbers communicated between server and agents1 . We assume that communication between server and agents has zero latency . Note that protocols in our model can be easily adapted to a network without a server , by designating an agent as the server . Distributed Multi-armed Bandits In distributed multi-armed bandits , there are M agents , labeled 1 , ... , M . Each agent is given access to the same stochastic K-armed bandit instance . Each arm k in the instance is associated with a reward distribution Pk . Pk is supported on [ 0 , 1 ] with mean µ ( k ) . Without loss of generality , we assume that arm 1 is the best arm ( i.e . µ ( 1 ) ≥ µ ( k ) , ∀k ∈ [ K ] ) . At each time step t = 1 , 2 , ... , T , each agent i chooses an arm at , i , and receives reward rt , i independently sampled from Pat , i . The goal of the agents is to minimize their total regret , which is defined as REG ( T ) = T∑ t=1 M∑ i=1 ( µ ( 1 ) − µ ( at , i ) ) . Distributed Linear Bandits In distributed linear bandits , the agents are given access to the same d-dimensional stochastic linear bandits instance . In particular , we assume that at time step t , agents are given an action set D ⊆ { x ∈ Rd : ‖x‖2 ≤ 1 } . Agent i chooses action xt , i ∈ D and observes reward yt , i . We assume that the mean of the reward is decided by an unknown parameter θ∗ ∈ Rd : yt , i = x T t , iθ ∗ + ηt , i , where ηt , i ∈ [ −1 , 1 ] are independent and have zero mean . For simplicity , we assume ‖θ∗‖2 ≤ 1 . For distributed linear bandits , the cumulative regret is defined as the sum of individual agent ’ s regrets : REG ( T ) = T∑ t=1 M∑ i=1 ( max x∈D xT θ∗ − xTt , iθ∗ ) . Here , we assume that the action set is fixed . A more general setting considers a time-varying action set Dt . In both cases , algorithms with O ( d √ T log T ) regret have been proposed ( Abbasi-Yadkori et al. , 2011 ) , while a regret lower bound of Ω ( d √ T ) is shown in Dani et al . ( 2008 ) . For both distributed multi-armed bandits and distributed linear bandits , our goal is to use as little communication as possible to achieve near-optimal regret . Since any M -agent protocol running for 1In our protocols , the number of bits each integer or real number uses is only logarithmic w.r.t . instance scale . Using the number of bits as the definition of communication complexity instead will only result in an additional logarithmic factor . The number of communicated bits is analyzed at the end of Appendix C G and H T steps can be simulated by a single-agent bandit algorithm running for MT time steps , the regret of any protocol is lower bounded by the optimal regret of a single-agent algorithm running for MT time steps . Therefore , we consider Õ ( √ MKT ) regret for multi-armed bandits and Õ ( d √ MT ) regret for linear bandits to be near-optimal . We are mainly interested in the case where the time horizon T is the dominant factor ( compared to M or K ) . Unless otherwise stated , we assume that T > max { M log 2M K , M , K } in the multi-armed bandits case and T > M in the linear bandits case . 1.2 OUR CONTRIBUTION . Now we give an overview of our results . In both settings , we present communication-efficient protocols that achieve near-optimal regret . Our results are summarized in Table 1 . Our results are compared with a naive baseline solution called immediate sharing in Table 1 : each agent sends the index of the arm he pulled and the corresponding reward he received to every other agent via the server immediately . This protocol can achieve near-optimal regret for both MAB and linear bandits ( O ( √ MKT ) and Õ ( d √ MT ) ) , but comes with high communication cost ( O ( M2T ) and O ( M2dT ) ) . Distributed MAB For distributed multi-armed bandits , we propose DEMAB ( Distributed Elimination for MAB ) protocol , which achieves near-optimal regret ( O ( √ MKT log ( MK ) ) ) with O ( M log ( MK ) ) communication cost . The communication cost is independent of the number of time steps T and grows only logarithmically w.r.t . the number of arms . We also prove the following lower bound : For any protocol with regret bound o ( M √ KT ) , the expected communication cost is at least Ω ( M ) . That is , in order to achieve near-optimal regret , the communication cost of DEMAB matches our lower bound except for logarithmic factors . Distributed Linear Bandits We propose DELB ( Distributed Elimination for Linear Bandits ) , an elimination based protocol for distributed linear bandits which achieves near-optimal regret bound ( O ( d √ MT log T ) ) with communication cost O ( ( Md+ d log log d ) log T ) . The communication cost of DELB enjoys nearly linear dependence on both M and d , and has at most logarithmic dependence on T . For the more general case where the action set is time-varying , we propose DisLinUCB ( Distributed LinUCB ) protocol , which achieves near-optimal regret with O ( M1.5d3 ) communication cost . 2 RELATED WORK . There has been growing interest in bandits problems with multiple players . One line of research considers the challenging problem of multi-armed bandits with collisions ( Rosenski et al. , 2016 ; Bistritz & Leshem , 2018 ; Kalathil et al. , 2014 ) , in which the reward for an arm is 0 if it is chosen by more than one player . The task is to minimize regret without communication . Their setting is motivated by problems in cognitive radio networks , and is fairly different from ours . Szörényi et al . ( 2013 ) and Korda et al . ( 2016 ) consider the distributed learning of MAB and linear bandits with restriction on the communication network . Motivated by fully decentralized applications , Szörényi et al . ( 2013 ) consider P2P communication networks , where an agent can communicate with only two other agents at each time step . A gossip-based -greedy algorithm is proposed for distributed MAB . Their algorithm achieves a speedup linear in M in terms of error rate , but the communication cost is linear in T . The work of Korda et al . ( 2016 ) uses a gossip protocol for regret minimization in distributed linear bandits . The main difference between their setting and ours is that each agent is only allowed to communicate with one agent at each time step in Korda et al . ( 2016 ) 2 . Their algorithm achieves near-optimal ( O ( d √ MT log T ) ) total regret using O ( Md2T ) communication cost . Another setting in literature concerns about distributed pure exploration in multi-armed bandits ( Hillel et al. , 2013 ; Tao et al. , 2019 ) , where the communication model is the most similar one to ours . These works use elimination based protocols for collaborative exploration , and establish tight bounds for communication-speedup tradeoff . However , their task ( speedup in pure exploration ) is not directly comparable to ours ( i.e . are not reducible to each other ) . Moreover , in Hillel et al . ( 2013 ) ; Tao et al . ( 2019 ) , the number of communication rounds is used as the measure of communication , while we use the amount of transmitted data .

Summary: The authors study a bandit problem where there are multiple agents (say, M) and each of the agents is playing a multi-armed bandit problem for T rounds. The agents can communicate with each other in order to achieve small regret. The problem is to design a strategy for arm-playing and communication so that the agents all combined can achieve a small regret w/o communicating a lot. The authors study this bandit problem in 2 settings: 1. multi-armed bandit setting and 2.  bandit linear optimization. For both these settings, the authors establish elimination style algorithms with communication. upon communication the sub-optimal arms are eliminated and the game continues with the remaining arms.