Cooperative Stochastic Bandits with Asynchronous Agents and Constrained Feedback

1 Introduction . Multi-armed bandits ( MABs ) [ 16 , 50 ] is a remarkably successful online learning framework that has been extensively studied since the 1950s [ 47 ] . It has a broad range of applications including datacenter optimization , web advertising , and recommender systems [ 50 , 31 , 39 , 15 ] . In the basic MAB problem , a learner repeatedly chooses an action ( pulls an arm ) in each round , and collects ( and observes ) the reward associated with the selected arm , but not rewards associated with the unselected arms . The goal of the learner is to maximize the long-term reward collected , and the performance metric is regret , which is the difference between the expected reward for the best arm and that received by the learner . The distributed/multi-agent/multiplayer MAB problem is an extension of the basic MAB that has been studied extensively in different settings [ 43 , 51 , 11 , 36 , 33 , 25 , 45 , 53 , 48 ] . This problem is motivated by several applications such as ( 1 ) large-scale learning systems [ 19 ] in domains such as finance or recommendation systems ; ( 2 ) cooperative search by multiple robots [ 41 , 32 ] ; ( 3 ) the application of wireless cognitive radio [ 17 , 43 , 42 , 9 ] ; and ( 4 ) distributed learning in distributed systems ( e.g. , a set of IoT devices learning about an underlying environment ) [ 46 , 23 , 5 , 12 , 54 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Most prior work on the multi-agent MAB problem assumes that agents have full access to all arms , and hence they solve the same MAB problem with the goal of minimizing the aggregate regret of the agents either in a competition setting [ 3 , 14 , 17 , 53 , 11 , 13 , 43 , 42 , 9 ] where agents receive degraded or no rewards when pulling the same arm , or in a collaboration/cooperation setting [ 45 , 53 , 38 , 37 , 36 , 48 , 44 ] , where agents receive independent rewards when they pull the same arm , and agents can communicate to improve their learning performance . In this work , we focus on the cooperative version of this problem , which has been extensively studied in recent years under different settings . In a thread of work , the agents are considered to be connected together over a communication graph [ 45 , 53 , 38 , 37 , 10 , 24 ] . Among them , [ 53 ] presents a more practical model in which agents are limited in their available communication capacity with others , and also develops the state-of-the-art algorithm which achieves an asymptotically optimal regret . The above basic models have been extended to several other settings such as pure-exploration through cooperation among agents for identifying the best stochastic arm [ 29 ] , cooperative multi-agent bandits with heavy tails [ 25 ] , cooperative kernelised contextual bandits [ 27 ] , linear bandits with safety constraints [ 2 ] , and the case with both honest and malicious agents [ 52 ] . In [ 10 ] , the problem is extended to the case where the underlying topology is time-varying . In another trend , the cooperative bandits have been studied over social networks [ 48 , 51 , 36 ] . Last , in a very recent trend , the stochastic bandits have extended to the federated setting , in which the learning task is distributed among multiple agents [ 56 , 49 , 26 ] . In this paper , we study a heterogeneous version of the cooperative multi-agent stochastic bandit problem with a set A = { 1 , . . . , M } of agents and a set K = { 1 , . . . , K } of arms . Agent j ∈ A has access to a subset Kj ⊆ K of arms . We refer to arms in Kj as local and the remaining arms in K\Kj as external arms . Agents come with different learning capabilities and can pull an arm every 1/θj rounds , with 0 < θj ≤ 1 as the action rate of agent j . The goal is to collaboratively find the optimal arm in K. An agent can pull any arm and receives a reward ; however , it observes the reward only when it pulls a local arm . It instantaneously forwards this observation to all other agents . We call this setup Feedback-constrained Cooperative Multi-agent MAB ( FC-CMA2B ) and formally define it in Section 2 . The FC-CMA2B setting arises naturally in large-scale learning systems that are often geographically distributed for domains such as web search , finance , and recommendation systems . In such largescale systems , it is usually impossible to run a single bandit algorithm . Instead , a distributed set of agents work to solve the problem , each with access to a subset of the action space . However , the entire learning task is integrated and the goal is to find the best global action , e.g. , the most relevant search result in a web search application , or the best possible video recommendation in the YouTube recommendation system among millions of possible recommendations . In the above scenario , learning among different agents may be asynchronous in the sense that each agent has its own action ( decision making ) rate , which is captured by FC-CMA2B . In Appendix A , we provide a few concrete application scenarios that could be captured by FC-CMA2B . The algorithmic challenge in FC-CMA2B originates from a nontrivial tradeoff that an agent must make between pulling a local arm with an observable reward , and pulling an external arm with the expectation of a larger but not observable reward . One might simply run a cooperative UCB algorithm on each agent without any distinction between local and external arms and pull the arm with the highest confidence interval . Then , if the selected arm is local , the agent can broadcast its observation to others . However , this strategy suffers a poor regret . The reason is intuitive . Consider a case where a suboptimal arm is only observable by a slow agent , i.e. , an agent with a very small action rate θj . In such a case , the upper confidence bound of the suboptimal arm will remain large due to the limited observations available only to the slow agent . Since the indexing policy of cooperative UCB suggests pulling the arm with the highest upper confidence bound , then , the suboptimal arm will be continuously pulled by everyone , resulting in a poor regret . We note that [ 55 ] also considers a cooperative bandit setting with heterogeneous agents with a subset of arms and different action rates . However , in [ 55 ] , the goal of each agent is to find the local optimal arm , while in this work , the goal of each agent is to find the global optimal arm . This difference leads to substantially different challenges in the algorithm design and analysis . More specifically , in our work , a foundational challenge is to find an effective cooperative strategy to resolve a dilemma between pulling local vs. external arms . This is not the case in [ 55 ] since the goal is to find the best local action . Contributions . This paper makes the following contributions . Algorithm design using a two-stage learning strategy . To tackle the new dilemma in FC-CMA2B for pulling either local or external arms , a proper bandit algorithm should be able to collect sufficient reward information , which is possible only by pulling local arms . To achieve this , our high-level idea is to distinguish between local and external arms and prioritize the local arms and only switch to external arms when they are clearly better . We implement this idea by proposing AAE-LCB , a two-stage learning strategy , which in the first stage selects local arms based on a carefully-designed active arm elimination ( AAE ) policy , and in the second stage selects external arms based on a lower confidence bound ( LCB ) policy . More specifically , in the first stage of AAE-LCB , each agent pulls only from a dynamically-constructed candidate set of local arms . Then through an AAE process , a local arm is removed from the candidate set when its confidence interval falls below that of at least one other ( local or external ) arm . With this strategy , AAE-LCB makes a sufficient number of observations of local arms , and resolves issues encountered by the cooperative UCB . For an agent whose local arms do not include the optimal arm , the elimination process continues until all local arms are eliminated , and when so , the algorithm enters a second stage , where the agent only pulls external arms . In the second stage , the agent totally relies on external observations that may occur at different rates . Differences in observation rates complicates decision making regarding choice of external arm such that the agent has to balance between well observed arms , and those that are not well observed but likely to be optimal . To tackle this tradeoff , AAE-LCB pulls an external arm with the largest LCB , and the intuition is that an arm with greater LCB is both better observed and more likely to be optimal . Note that selecting based on largest UCB may lead to pulling external arms whose upper confidence bounds are large due to a lack of samples . Regret lower and upper bounds . We analyze the regret of AAE-LCB and show that it achieves a regret of O ( ∑ i : ∆i > 0 Θi/Θi∗ ×K log T/∆i ) , where Θi is the action rate of arm i defined as the sum of the action rates of agents containing arm i and i∗ is the optimal arm . We further establish a regret lower bound for FC-CMA2B and shows AAE-LCB is regret optimal up to a small factor . For comparison , we investigate the regret of a baseline algorithm , AAE-AAE , which pulls external arms based on another AAE process and show that it exhibits poor performance in the presence of agents with low action rates . Specifically , we show that AAE-AAE suffers a regret of Ω ( Θ/Θmin log T ) , where Θ = ∑ j∈A θj and Θmin = mini∈K ∑ j : i∈Kj θj are the aggregate action rate of all agents , and the minimum aggregate action rate of agents containing any arm i , respectively . This result shows that regret can be arbitrarily bad when Θmin , i.e. , the minimum observation rate among all arms ( either optimal or suboptimal ) , is small . Hence , instead of regret depending on the observation rate of the least observable ( and possibly suboptimal ) arms , the regret of AAE-LCB depends on that of the optimal arm . Numerical results . Through brief numerical experiments , we verify our theoretical observations and show that the cooperative extension of UCB and AAE-AAE are either unable to gain sufficient observations for the global optimal arm or vulnerable to the action rate of slow agents with suboptimal arms , while AAE-LCB is robust to both issues and achieves much better performance . 2 Problem Setup . Feedback-constrained Cooperative Multi-agent MAB ( FC-CMA2B ) . We consider a multi-agent stochastic bandit setting with a set A = { 1 , . . . , M } of independent agents existing over the entire time period , and a set K = { 1 , . . . , K } of arms . Associated with arms are mutually independent sequences of i.i.d . Bernoulli rewards with mean 0 ≤ µ ( i ) ≤ 1 , for arm i ∈ K. Agent j ∈ A has full access to a subset Kj ⊆ K , Kj = |Kj | , of arms . We refer to Kj and K \ Kj as the sets of local and external arms of agent j , respectively . Agents are allowed to pull and receive a reward from any arm from K , however , they only receive observations from local arms . That is to say , if the selected arm is external , the agent does not observe the reward even though the reward is allocated . Last , we make no assumption regarding overlaps among sets Kj . In addition to heterogeneity in access to arms , agents are also heterogeneous in their decision making capabilities . Specifically , considering the decision rounds { 1 , . . . , T } , agent j is able to pull an arm every ωj ∈ N+ rounds , i.e. , decision rounds for agent j are t = ωj , 2ωj , . . . , Njωj , where Nj = bT/ωjc . Parameter ωj represents the inter-round gap of agent j . For simplicity of analysis , we define θj : = 1/ωj as the action rate of agent j . Intuitively , the larger θj , the faster the agent j is in pulling arms . The goal of each agent is to learn the global optimal arm . The regret for agent j in FC-CMA2B is RjT : = µ ( i ∗ ) Nj − ∑ t∈ { kωj : k=1 , ... , Nj } xt ( I j t ) , where i∗ is the optimal arm in K , and Ijt ∈ K is the pulled arm by agent j at time t. Agent j receives the reward xt ( I j t ) whose value is observable only if I j t ∈ Kj . The ultimate goal is to minimize the aggregate regret of the agents , i.e. , RT = ∑ j∈AR j T . We note that when two agents select the same arm , they collect stochastically independent rewards , and there is no reward degradation such as occurs in competition settings . We also consider full and truthful cooperation among agents , i.e. , after committing to one decision and receiving a reward from the pulled arm , an agent broadcasts observations to others at no cost and no delay . In the supplementary material , we further consider a more general model with communication delays among agents . Additional notations and terminologies . To facilitate our algorithm design and analysis , we introduce the following notations . Let Ai denote the set of agents whose local arm sets include arm i , Ai : = { j ∈ A : i ∈ Kj } , ∀i ∈ K. By ∆ ( i′ , i ) , we denote the difference in mean rewards of arms i′ and i , i.e. , ∆ ( i′ , i ) : = µ ( i′ ) − µ ( i ) . When i′ = i∗ , we rewrite ∆ ( i∗ , i ) as ∆i , which is known as the suboptimality gap in the basic bandit problem . Last , we define Θ and Θi , i ∈ K , as follows Θ : = ∑ j∈A θj , and Θi : = ∑ j∈Ai θj , where Θi is the action rate of arm i across all agents in Ai . Intuitively , the larger Θi is , the higher the rate at which arm i can be pulled by agents inAi . The action rate Θi plays a key role in characterizing the regret bounds .

Summary: This paper studies an asynchronous distributed multi-arm bandit problem. In this problem, each agent can pull arms at a different rate. In addition, each agent can only observe the reward from a subset of arms. The reward observed by an agent is immediately broadcasted to all other agents. The question is to analyze regret. The paper provides both upper and lower bound results.