Heterogeneous Multi-player Multi-armed Bandits: Closing the Gap and Generalization

1 Introduction . Motivated by the application of cognitive radio ( Anandkumar et al. , 2010 , 2011 ; Gai et al. , 2010 ) , the multi-player version of the multi-armed bandits problem ( MP-MAB ) has sparked significant interests in recent years . MP-MAB takes player interactions into account by having multiple decentralized players simultaneously play the bandit game and interact with each other through arm collisions . Prior MP-MAB studies mostly focus on the homogeneous variant , where the bandit model is assumed to be the same across players ( Liu and Zhao , 2010 ; Rosenski et al. , 2016 ; Besson and Kaufmann , 2018 ) . Recent attentions have shifted towards the more general MP-MAB model with player-dependent bandit instances ( i.e. , the heterogeneous variant ) ( Bistritz and Leshem , 2018 , 2020 ; Tibrewal et al. , 2019 ; Boursier et al. , 2020 ) . However , unlike the homogeneous variant , the current understanding of the heterogeneous setting is still limited . • Recent advances ( Boursier and Perchet , 2019 ; Wang et al. , 2020 ) show that for the homogeneous setting , decentralized MP-MAB algorithms can achieve almost the same performance as centralized ones . However , state-of-the-art results in heterogeneous variants still have significant gaps from 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the centralized performance . It remains an open problem whether a decentralized algorithm can approach the centralized performance for the heterogeneous MP-MAB variant . • All prior MP-MAB works are confined to a linear system reward function : the system reward is the sum of individual outcomes from players . However , practical system objectives are often captured by more complicated nonlinear reward functions , e.g. , the minimal function ( see Section 5.1 ) . In this paper , we make progress in the aforementioned problems for decentralized heterogeneous MPMAB . A novel algorithm called BEACON – Batched Exploration with Adaptive COmmunicatioN – is proposed and analyzed . In particular , this work makes the following contributions . • BEACON introduces several novel ideas to the design of implicit communication and efficient exploration . For the former , a novel adaptive differential implicit communication ( ADC ) scheme is proposed , which can significantly lower the implicit communication loss compared to the state of the art . For the latter , core principles from CUCB ( Chen et al. , 2013 ) are incorporated with a batched exploration design , which leads to both efficient and effective explorations . • For the linear reward function , we rigorously show that regret bounds of BEACON , both problemdependent and problem-independent , not only improve all prior regret analyses but more importantly are capable of approaching the centralized lower bounds , thus answering the aforementioned open problem positively . • We then propose to generalize the study of heterogeneous MP-MAB to general ( nonlinear ) reward functions . BEACON is extended to solve such problems and we show that it achieves a regret of O ( log ( T ) ) , where T is the time horizon . The analysis itself holds important value as it bridges the regret analysis of combinatorial MAB ( CMAB ) and MP-MAB . • BEACON achieves impressive empirical results . It not only outperforms existing decentralized algorithms significantly , but indeed has a comparable performance as the centralized benchmark , hence corroborating the theoretical analysis . Remarkably , BEACON with the linear reward function generally achieves ∼ 6× improvement over the state-of-the-art METC ( Boursier et al. , 2020 ) . 2 Problem Formulation . A decentralized MP-MAB model consists of K arms and M players . As commonly assumed ( Bistritz and Leshem , 2020 ; Boursier et al. , 2020 ) , there are more arms than players , i.e. , M ≤ K , and initially the players have knowledge of K but not M . Furthermore , no explicit communications are allowed among players , which results in a decentralized system . Also , time is assumed to be slotted , and at time step t , each player m ∈ [ M ] chooses and pulls an arm sm ( t ) ∈ [ K ] . The action vector of all players at time t is denoted as S ( t ) : = [ s1 ( t ) , ... , sM ( t ) ] , which is referred to as a “ matching ” for convenience although it is not necessarily one-to-one . Individual Outcomes . For each player m , an outcome1 Ok , m ( t ) is associated with her action of pulling arm sm ( t ) = k at time t , which is defined as Ok , m ( t ) : = Xk , m ( t ) ηk ( S ( t ) ) . ( 1 ) In Eqn . ( 1 ) , Xk , m ( t ) is a random variable of arm utility and ηk ( S ( t ) ) is the no-collision indicator defined by ηk ( S ) : = 1 { |Ck ( S ) | ≤ 1 } with Ck ( S ) : = { n ∈ [ M ] |sn = k } . In other words , if player m is the only player choosing arm k , the outcome is Xk , m ( t ) ; if multiple players choose arm k simultaneously , a collision happens on this arm and the outcome is zero regardless of Xk , m ( t ) . For a certain arm-player pair , i.e. , ( k , m ) , the set of random arm utilities { Xk , m ( t ) } t≥1 is assumed to be sampled independently from an unknown distribution φk , m , which has a bounded support on [ 0 , 1 ] and an unknown expectation E [ Xk , m ( t ) ] = µk , m . In general , these utility distributions are player-dependent , i.e. , µk , m 6= µk , n when m 6= n. Note that despite the time independence among { Xk , m ( t ) } t≥1 for a certain arm-player pair ( k , m ) , correlations can exist among the random utility variables of different arm-player pairs , i.e. , among Xk , m ( t ) for different ( k , m ) pairs . To ease the exposition , we define S = { S = [ s1 , ... , sM ] |sm ∈ [ K ] , ∀m ∈ [ M ] } as the set of all possible matchings S and abbreviate the arm k of player m as arm ( k , m ) . We further denote µ = [ µk , m ] ( k , m ) ∈ [ K ] × [ M ] and µS = [ µsm , m ] m∈ [ M ] for S = [ s1 , ... , sM ] . System Rewards . Besides players ’ individual outcomes , with matching S ( t ) chosen at time t , a random system reward , denoted as V ( S ( t ) , t ) , is collected for the entire system . The most commonly- 1The term “ outcome ” distinguishes players ’ individual rewards from the later introduced system rewards . studied reward function ( Bistritz and Leshem , 2020 ; Boursier et al. , 2020 ) is the sum of outcomes from different players ( referrd to as the linear reward function ) , i.e. , V ( S ( t ) , t ) : = ∑ m∈ [ M ] Osm ( t ) , m ( t ) . With this linear reward function , for matching S , the expected system reward is denoted as Vµ , S : = E [ V ( S , t ) ] = ∑ m∈ [ M ] µsm , mηsm ( S ) under matrix µ . As almost all of the existing MP-MAB literature focus on the linear reward function , we also focus on this case first , but note that the problem formulation presented in this section can be extended to general ( nonlinear ) reward functions in Section 5 . Feedback Model . Different feedback models exist in the MP-MAB literature , and this work focuses on the collision-sensing model ( Bistritz and Leshem , 2018 , 2020 ; Boursier and Perchet , 2019 ; Boursier et al. , 2020 ) . Specifically , player m can access her own outcome Osm ( t ) , m ( t ) and the corresponding no-collision indicator ηsm ( t ) ( S ( t ) ) , but neither the overall reward V ( S ( t ) , t ) nor outcomes of other players . In other words , at time t , player m chooses arm sm ( t ) based on her own history Hm ( t ) = { sm ( τ ) , Osm ( τ ) , m ( τ ) , ηsm ( τ ) ( S ( τ ) ) } 1≤τ≤t−1 . Regret Definition . If µ is known a priori , the optimal choice is the matching that gives the highest expected reward Vµ , ∗ : = maxS∈S Vµ , S . We formally define the regret after T rounds of playing as R ( T ) = TVµ , ∗ − E [ T∑ t=1 V ( S ( t ) , t ) ] , ( 2 ) where the expectation is w.r.t . the randomness of the policy and the environment . One technical novelty worth noting is that this work considers the general case with possibly multiple optimal matchings , instead of the commonly assumed unique one ( Bistritz and Leshem , 2018 , 2020 ) . Multiple optimal matchings might be uncommon for the linear reward function , but often occur under more sophisticated reward functions that will be discussed later , e.g. , the minimal function , and brings substantial difficulties into player coordination . In addition , the proposed BEACON design is also applicable to the homogeneous setting , i.e. , ∀m ∈ [ M ] , µk , m = µk , with some trivial adjustments . 3 The BEACON Algorithm . 3.1 Algorithm Structure and Key Ideas . After the orthogonalization procedure ( Wang et al. , 2020 ) at the beginning of the game , during which each player individually estimates the number of players M and assigns herself of a unique index m ∈ [ M ] , BEACON proceeds in epochs and each epoch consists of two phases : ( implicit ) communication and exploration.2 While similar two-phase structures have been adopted by other heterogeneous MP-MAB algorithms ( Tibrewal et al. , 2019 ; Boursier et al. , 2020 ) , those designs fail to have regrets approaching the centralized lower bound . The challenge in approaching the centralized lower bound is not only designing more efficient implicit communications and explorations , but also connecting them in a way that neither phase dominates the overall regret and both approach the centralized lower bound simultaneously . BEACON precisely achieves these goals , with several key ideas that not only are crucial to closing the regret gap but also hold individual values in MP-MAB research . First , a novel adaptive differential communication ( ADC ) method is proposed , which is fundamental in improving the effectiveness and efficiency of implicit communications . Specifically , ADC drastically reduces the communication cost from up to O ( log ( T ) ) per epoch in state-of-the-art designs ( Boursier et al. , 2020 ) to O ( 1 ) per epoch , which ensures a low communication cost . Second , CUCB principles ( Chen et al. , 2013 ) are incorporated with a batched exploration structure to ensure a low exploration loss ( see Section 8 for more discussions on the relationship between CMAB and MP-MAB ) . CUCB principles address a critical challenge of large amount of matchings in heterogeneous MP-MAB ( i.e. , |S| = KM ) , which hampered prior designs . The batched structure , on the other hand , is carefully embedded and optimized such that the need of communication and exploration is balanced , leading to neither dominating the overall regret . 2Details of the orthogonalization procedure are given in the supplementary material . In addition , by “ exploration phase ” , we mean the time steps in one epoch that are not used for ( implicit ) communications , which actually contain both exploration and exploitation .

Summary: Overall, this paper introduces a new algorithm BEACON for multi-agent multi-armed bandits that closes the gap between the state-of-the-art upper bound and information theoretic lower bound of the regrets by implicit communication via adaptive differential communication protocols and efficient exploration via UCB-based algorithms. In addition to the existing case where the system-level reward is a linear combination of individual rewards, BEACON also tackles the general cases where the combination is non-linear, obtaining a regret bound of O(log T). Experiments show that BEACON significantly outperforms state-of-the-art benchmarks in decentralized bandits.