Heterogeneous Multi-player Multi-armed Bandits: Closing the Gap and Generalization

1 Introduction . Motivated by the application of cognitive radio ( Anandkumar et al. , 2010 , 2011 ; Gai et al. , 2010 ) , the multi-player version of the multi-armed bandits problem ( MP-MAB ) has sparked significant interests in recent years . MP-MAB takes player interactions into account by having multiple decentralized players simultaneously play the bandit game and interact with each other through arm collisions . Prior MP-MAB studies mostly focus on the homogeneous variant , where the bandit model is assumed to be the same across players ( Liu and Zhao , 2010 ; Rosenski et al. , 2016 ; Besson and Kaufmann , 2018 ) . Recent attentions have shifted towards the more general MP-MAB model with player-dependent bandit instances ( i.e. , the heterogeneous variant ) ( Bistritz and Leshem , 2018 , 2020 ; Tibrewal et al. , 2019 ; Boursier et al. , 2020 ) . However , unlike the homogeneous variant , the current understanding of the heterogeneous setting is still limited . • Recent advances ( Boursier and Perchet , 2019 ; Wang et al. , 2020 ) show that for the homogeneous setting , decentralized MP-MAB algorithms can achieve almost the same performance as centralized ones . However , state-of-the-art results in heterogeneous variants still have significant gaps from 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the centralized performance . It remains an open problem whether a decentralized algorithm can approach the centralized performance for the heterogeneous MP-MAB variant . • All prior MP-MAB works are confined to a linear system reward function : the system reward is the sum of individual outcomes from players . However , practical system objectives are often captured by more complicated nonlinear reward functions , e.g. , the minimal function ( see Section 5.1 ) . In this paper , we make progress in the aforementioned problems for decentralized heterogeneous MPMAB . A novel algorithm called BEACON – Batched Exploration with Adaptive COmmunicatioN – is proposed and analyzed . In particular , this work makes the following contributions . • BEACON introduces several novel ideas to the design of implicit communication and efficient exploration . For the former , a novel adaptive differential implicit communication ( ADC ) scheme is proposed , which can significantly lower the implicit communication loss compared to the state of the art . For the latter , core principles from CUCB ( Chen et al. , 2013 ) are incorporated with a batched exploration design , which leads to both efficient and effective explorations . • For the linear reward function , we rigorously show that regret bounds of BEACON , both problemdependent and problem-independent , not only improve all prior regret analyses but more importantly are capable of approaching the centralized lower bounds , thus answering the aforementioned open problem positively . • We then propose to generalize the study of heterogeneous MP-MAB to general ( nonlinear ) reward functions . BEACON is extended to solve such problems and we show that it achieves a regret of O ( log ( T ) ) , where T is the time horizon . The analysis itself holds important value as it bridges the regret analysis of combinatorial MAB ( CMAB ) and MP-MAB . • BEACON achieves impressive empirical results . It not only outperforms existing decentralized algorithms significantly , but indeed has a comparable performance as the centralized benchmark , hence corroborating the theoretical analysis . Remarkably , BEACON with the linear reward function generally achieves ∼ 6× improvement over the state-of-the-art METC ( Boursier et al. , 2020 ) . 2 Problem Formulation . A decentralized MP-MAB model consists of K arms and M players . As commonly assumed ( Bistritz and Leshem , 2020 ; Boursier et al. , 2020 ) , there are more arms than players , i.e. , M ≤ K , and initially the players have knowledge of K but not M . Furthermore , no explicit communications are allowed among players , which results in a decentralized system . Also , time is assumed to be slotted , and at time step t , each player m ∈ [ M ] chooses and pulls an arm sm ( t ) ∈ [ K ] . The action vector of all players at time t is denoted as S ( t ) : = [ s1 ( t ) , ... , sM ( t ) ] , which is referred to as a “ matching ” for convenience although it is not necessarily one-to-one . Individual Outcomes . For each player m , an outcome1 Ok , m ( t ) is associated with her action of pulling arm sm ( t ) = k at time t , which is defined as Ok , m ( t ) : = Xk , m ( t ) ηk ( S ( t ) ) . ( 1 ) In Eqn . ( 1 ) , Xk , m ( t ) is a random variable of arm utility and ηk ( S ( t ) ) is the no-collision indicator defined by ηk ( S ) : = 1 { |Ck ( S ) | ≤ 1 } with Ck ( S ) : = { n ∈ [ M ] |sn = k } . In other words , if player m is the only player choosing arm k , the outcome is Xk , m ( t ) ; if multiple players choose arm k simultaneously , a collision happens on this arm and the outcome is zero regardless of Xk , m ( t ) . For a certain arm-player pair , i.e. , ( k , m ) , the set of random arm utilities { Xk , m ( t ) } t≥1 is assumed to be sampled independently from an unknown distribution φk , m , which has a bounded support on [ 0 , 1 ] and an unknown expectation E [ Xk , m ( t ) ] = µk , m . In general , these utility distributions are player-dependent , i.e. , µk , m 6= µk , n when m 6= n. Note that despite the time independence among { Xk , m ( t ) } t≥1 for a certain arm-player pair ( k , m ) , correlations can exist among the random utility variables of different arm-player pairs , i.e. , among Xk , m ( t ) for different ( k , m ) pairs . To ease the exposition , we define S = { S = [ s1 , ... , sM ] |sm ∈ [ K ] , ∀m ∈ [ M ] } as the set of all possible matchings S and abbreviate the arm k of player m as arm ( k , m ) . We further denote µ = [ µk , m ] ( k , m ) ∈ [ K ] × [ M ] and µS = [ µsm , m ] m∈ [ M ] for S = [ s1 , ... , sM ] . System Rewards . Besides players ’ individual outcomes , with matching S ( t ) chosen at time t , a random system reward , denoted as V ( S ( t ) , t ) , is collected for the entire system . The most commonly- 1The term “ outcome ” distinguishes players ’ individual rewards from the later introduced system rewards . studied reward function ( Bistritz and Leshem , 2020 ; Boursier et al. , 2020 ) is the sum of outcomes from different players ( referrd to as the linear reward function ) , i.e. , V ( S ( t ) , t ) : = ∑ m∈ [ M ] Osm ( t ) , m ( t ) . With this linear reward function , for matching S , the expected system reward is denoted as Vµ , S : = E [ V ( S , t ) ] = ∑ m∈ [ M ] µsm , mηsm ( S ) under matrix µ . As almost all of the existing MP-MAB literature focus on the linear reward function , we also focus on this case first , but note that the problem formulation presented in this section can be extended to general ( nonlinear ) reward functions in Section 5 . Feedback Model . Different feedback models exist in the MP-MAB literature , and this work focuses on the collision-sensing model ( Bistritz and Leshem , 2018 , 2020 ; Boursier and Perchet , 2019 ; Boursier et al. , 2020 ) . Specifically , player m can access her own outcome Osm ( t ) , m ( t ) and the corresponding no-collision indicator ηsm ( t ) ( S ( t ) ) , but neither the overall reward V ( S ( t ) , t ) nor outcomes of other players . In other words , at time t , player m chooses arm sm ( t ) based on her own history Hm ( t ) = { sm ( τ ) , Osm ( τ ) , m ( τ ) , ηsm ( τ ) ( S ( τ ) ) } 1≤τ≤t−1 . Regret Definition . If µ is known a priori , the optimal choice is the matching that gives the highest expected reward Vµ , ∗ : = maxS∈S Vµ , S . We formally define the regret after T rounds of playing as R ( T ) = TVµ , ∗ − E [ T∑ t=1 V ( S ( t ) , t ) ] , ( 2 ) where the expectation is w.r.t . the randomness of the policy and the environment . One technical novelty worth noting is that this work considers the general case with possibly multiple optimal matchings , instead of the commonly assumed unique one ( Bistritz and Leshem , 2018 , 2020 ) . Multiple optimal matchings might be uncommon for the linear reward function , but often occur under more sophisticated reward functions that will be discussed later , e.g. , the minimal function , and brings substantial difficulties into player coordination . In addition , the proposed BEACON design is also applicable to the homogeneous setting , i.e. , ∀m ∈ [ M ] , µk , m = µk , with some trivial adjustments . 3 The BEACON Algorithm . 3.1 Algorithm Structure and Key Ideas . After the orthogonalization procedure ( Wang et al. , 2020 ) at the beginning of the game , during which each player individually estimates the number of players M and assigns herself of a unique index m ∈ [ M ] , BEACON proceeds in epochs and each epoch consists of two phases : ( implicit ) communication and exploration.2 While similar two-phase structures have been adopted by other heterogeneous MP-MAB algorithms ( Tibrewal et al. , 2019 ; Boursier et al. , 2020 ) , those designs fail to have regrets approaching the centralized lower bound . The challenge in approaching the centralized lower bound is not only designing more efficient implicit communications and explorations , but also connecting them in a way that neither phase dominates the overall regret and both approach the centralized lower bound simultaneously . BEACON precisely achieves these goals , with several key ideas that not only are crucial to closing the regret gap but also hold individual values in MP-MAB research . First , a novel adaptive differential communication ( ADC ) method is proposed , which is fundamental in improving the effectiveness and efficiency of implicit communications . Specifically , ADC drastically reduces the communication cost from up to O ( log ( T ) ) per epoch in state-of-the-art designs ( Boursier et al. , 2020 ) to O ( 1 ) per epoch , which ensures a low communication cost . Second , CUCB principles ( Chen et al. , 2013 ) are incorporated with a batched exploration structure to ensure a low exploration loss ( see Section 8 for more discussions on the relationship between CMAB and MP-MAB ) . CUCB principles address a critical challenge of large amount of matchings in heterogeneous MP-MAB ( i.e. , |S| = KM ) , which hampered prior designs . The batched structure , on the other hand , is carefully embedded and optimized such that the need of communication and exploration is balanced , leading to neither dominating the overall regret . 2Details of the orthogonalization procedure are given in the supplementary material . In addition , by “ exploration phase ” , we mean the time steps in one epoch that are not used for ( implicit ) communications , which actually contain both exploration and exploitation .

Summary: This paper proposes the BEACON algorithm under Multi-player MAB settings. The authors make use of the ideas in combinatorial-MAB area and implement batched exploration, in which they choose an update rule for the arm counters to sufficiently but not excessively explore the arms. Then, they  extends the adaptive communication to adaptive differential communication, also, the communication only happens when the sample mean is sufficiently precise. With the combination of these strategies, the BEACON achieves the regret lower bound while requiring less assumptions (unknown horizon, non-linear reward functions, etc.) than prior works in the area.