A Gang of Adversarial Bandits

√ ln ( NK/Ψ ) ΨKT ) and per-trial time complexity of O ( K ln ( N ) ) , whilst GABA-II has a weaker O ( √ ln ( N/Ψ ) ln ( NK/Ψ ) ΨKT ) regret , but a better O ( ln ( K ) ln ( N ) ) per-trial time complexity . We highlight improvements of both algorithms over running independent standard MABs across users . 1 Introduction . During the last decade multi-armed bandits ( MAB ) have received a great deal of attention in machine learning and related fields , due to their wide practical and theoretical importance . The central problem is to design a decision strategy whereby a learner explores sequentially the environment in order to find the best item ( arm ) within a prescribed set . At each step in the exploration the learner chooses an arm , after which feedback ( typically a loss or reward corresponding to the selected arm ) is observed from the environment . Then the next decision is made by the learner based on past interactions , and the process repeats . The goal is to design efficient exploration strategies which incur a small cumulative loss in comparison to the cumulative loss that would have been obtained by always * Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . selecting the best arm in hindsight . Applications of MAB are numerous , including recommender systems [ 1 ] , clinical trials [ 2 ] , and adaptive routing [ 3 ] , among others . In this paper we study the problem in which the learner is facing several MAB problems that are related according to a prescribed interaction graph . A main motivation behind this problem are online recommendation systems , whereby each of several users is associated with a MAB problem ( task ) , where the arms correspond to a finite set of products , and the graph represents a social network among users . The goal is to exploit users ’ similarity in order to improve the efficiency of learning users ’ preferences via online exploration of products . In the standard full information setting , a lot of work has been done showing that techniques from multitask learning are effective in reducing the amount of data needed to learn each of the individual tasks , both in the statistical and adversarial settings , see [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] and references therein . Graphs have been used to model task relationships , with different tasks ’ parameters encouraged to be close according to the graph topology . In contrast , multitask learning in the bandit setting is much less explored . The algorithms that we present exploit the network homophily principle [ 12 ] which formulates that users that are connected in the network have similar preferences , that is , they tend to share preferred recommendations . We will show that our algorithms exploit graph structure and enjoy potentially much smaller regret bounds than the cumulative regret of standard MAB run independently on each user . Since the original graph may be dense , we exploit a randomized sparsification technique to develop fast prediction algorithms . Our approach builds upon previous work on online learning over graphs [ 13 , 14 ] to generate a perfect full oriented binary tree , whose leaves are in one-to-one correspondence with the nodes of the original graph . This construction approximately preserves the relevant graph properties in expectation , and provides the starting point for designing our efficient algorithms . A further ingredient in our algorithm is provided by the method of specialists [ 15 , 16 ] . Our learning strategies combine the above ingredients to devise efficient online algorithms under partial feedback . Contributions . We introduce two Gang of Adversarial BAndit algorithms , GABA-I and GABAII that learn jointly MAB models for N users over K possible actions . Both algorithms are designed to exploit network structure while being ( extremely ) computationally efficient . We derive expected ( over the algorithms ’ randomizations ) regret bounds . The bounds scale with the dispersion measure Ψ ∈ [ 1 , N ] of the best actions over the graph . For GABA-I the bound1 is of order of O ( √ ln ( NK/Ψ ) ΨKT ) , where T is the number of trials , and has a per-trial time complexity of O ( K ln ( N ) ) . On the other hand GABA-II has a weaker expected regret bound of O ( √ ln ( N/Ψ ) ln ( NK/Ψ ) ΨKT ) but is faster , having a per-trial time complexity of O ( ln ( K ) ln ( N ) ) . Thus the GABA-I algorithm improves on algorithms that treat each user independently , as in the best case the regret improves from O ( √ N ) to O ( √ lnN ) and in the worst case the regret degrades by at most a constant factor . GABA-II has slightly weaker regret bounds ; however , it is more computationally efficient . Outline of Main Results . The social network graph G is determined by a set of undirected links between users { ωu , v } Nu < v where ωu , v ∈ [ 0 , ∞ ) indicates the magnitude of the link between user u and v. For all t ∈ [ T ] we have a user ut ∈ [ N ] and a loss vector ` t ∈ [ 0 , 1 ] K which are selected by Nature before learning begins and are unknown to Learner ; i.e. , Nature is a deterministic oblivious adversary ( see e.g. , [ 17 , Section 5.1 ] ) . Learning then proceeds in trials t = 1 , 2 , . . . , T . On trial t : 1 . Nature reveals user ut ∈ [ N ] to Learner , 2 . Learner selects action at ∈ [ K ] , 3 . Nature reveals loss ` t , at ∈ [ 0 , 1 ] to Learner . Before reflecting on the N -user case we review the well-known results for the single user ( N = 1 ) . The seminal EXP3 algorithm [ 18 ] obtains the following ( uniform ) regret bound2 , E [ ∑ t∈ [ T ] ` t , at ] − min a∈ [ K ] ∑ t∈ [ T ] ` t , a ∈ O ( √ K ln ( K ) T ) , ( 1 ) 1The bounds of GABA-I and GABA-II however depend on oracular knowledge of optimal tuning parameters . We discuss this as well as a means of lessening this dependency following Corollary 5 . 2An algorithm was given in [ 19 ] that removed the lnK term from the regret . where the expectation is with respect to the internal randomization of the EXP3 algorithm . In the N -user setting , by running a copy of EXP3 independently for each user , we may obtain a uniform regret bound of ( see e.g. , [ 20 ] ) E [ ∑ t∈ [ T ] ` t , at ] − min y : [ N ] → [ K ] ∑ t∈ [ T ] ` t , y ( ut ) ∈ O ( √ K ln ( K ) NT ) , ( 2 ) i.e. , for each user u the best action is y ( u ) and we now pay an additional constant factor of √ N in our regret . In this work we exploit the social network structure to prove a non-uniform regret bound for the GABA-I algorithm ( see Corollary 4 ) of R ( y ) : = E [ ∑ t∈ [ T ] ` t , at ] − ∑ t∈ [ T ] ` t , y ( ut ) ∈ O ( √ K ln ( KN Ψ ( y ) ) Ψ ( y ) T ) , ( 3 ) for any mapping of users to actions y : [ N ] → [ K ] . The non-uniform regret now depends on Ψ ( y ) ∈ [ 1 , N ] ( see ( 5 ) ) which measures dispersion of users ’ ‘ best ’ actions across the network . Thus , by taking network structure into account , we may upper bound the scaling in the regret with respect to the number of users by O ( √ ln ( eNΨ ( y ) ) Ψ ( y ) ) . When the best action across the network is nearly uniform then the dispersion Ψ ( y ) ∈ O ( 1 ) , in contrast when the dispersion is maximal then Ψ ( y ) = N thus in the best case the regret with respect to the number of users improves from O ( √ N ) to O ( √ lnN ) and in the worst case the regret only increases by a constant factor . The first algorithm GABA-I obtains the regret ( 3 ) while requiring O ( K lnN ) time to predict and update . The second algorithm GABA-II ’ s regret ( see Corollary 5 ) is larger by a O ( √ lnN/Ψ ( y ) ) factor but now prediction is at an even faster O ( ln ( K ) ln ( N ) ) time per trial , that is , prediction time improves exponentially with respect to the cardinality of the action set [ K ] . Thus both algorithms support very large user networks and the second algorithm allows efficient prediction with very large action sets . Related Work . We mention here some of the key papers that are more closely related to ours and refer the reader to the technical appendices for an expanded literature review . There has been much work in the heterogenous multi-user setting for linear-stochastic bandits . Out of these works , those closest to us are when the users are in a known social network and it is assumed that neighbouring users respond to context vectors in a similar way [ 21 , 22 , 23 , 24 , 25 , 26 ] but as far as we are aware no works on this model have so far been done in the adversarial setting . Other works on this topic include those in which it is assumed that there is an unknown clustering of the users , where users in the same cluster are homogenous [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ] ; as well as other models [ 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ] . There are also works on networked , homogenous multi-user bandit models with limited communication between users [ 44 , 45 , 46 , 47 , 48 , 49 , 50 ] . Related to the multiuser setting are works on transfer learning and meta-learning with linear-stochastic bandits [ 51 , 52 ] . Whilst our work assumes a known network structure over the users , there is a wide literature on bandit problems in which the actions are structured in a network and it is assumed that neighbouring actions give similar losses [ 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ] , as well as other networked-action models [ 61 , 62 , 63 ] . In addition to the seminal paper on adversarial bandits [ 18 ] , our work utilises ideas from several different papers [ 13 , 14 , 15 , 16 , 64 , 65 ] . Notation . Given a set X we define 2X to be its power-set , that is : the set of all subsets of X . For any positive integer m , we define [ m ] : = { 1 , 2 , . . . , m } . For any predicate PRED , JPREDK : = 1 if PRED is true and equals 0 otherwise . Given vectors x , x′ ∈ RK we define x · x′ to be their inner product ( i.e. , equal to ∑ i∈ [ K ] xixi′ ) and we define x x′ to be their component-wise product ( i.e. , ( x x′ ) i : = xix′i for all i ∈ [ K ] ) . We define ‘ 1 ’ to be the K-dimensional vector in which each component is equal to 1 . Given a full oriented binary tree B we denote the set of its vertices also by B . Given a non-leaf vertex n ∈ B let / ( n ) and . ( n ) be its left child and right child respectively . Given a non-root vertex n ∈ B let ↑ ( n ) be its parent . Given a vertex n ∈ B let ⇑ ( n ) and ⇓ ( n ) be its set of ancestors and leaf-descendants ( i.e . descendants which are leaves ) respectively . Given a vertex n ∈ B we define J ( n ) and I ( n ) as the left-most and right-most descendants ( which are leaves ) of n respectively . Finally , we denote the user graph by G , which is an undirected connected graph with edge weights { ωu , v : 1 ≤ u < v ≤ N } . For convenience we assume N is a power of two.3 3This assumption does not limit our results because to run our algorithms one can always add dummy vertices without altering input weights , so as to force N to be a power of two .

Summary: This paper studies a new multi-armed bandit setting termed "Gang of Adversarial Bandits", in which a learner solves an instance of the adversarial MAB problem for each of $N$ different users. Without further assumptions on the $N$ users, essentially one can only solve the instances independently (e.g., running $N$ copies of the EXP3 algorithm), and incur a loss that scales as $\sqrt{KNT}$, where $K$ is the number of actions and $T$ is the time horizon.  In contrast, the authors assumes that the connections between the users are specified by a known weighted graph $G$, and assumes that nearby vertices in the graph (measured in terms of the effective resistance) are more likely to have the same action in the "benchmark" mapping $y: [N] \to [K]$. Both algorithms proposed in the paper are instantiations of the "specialists" framework, using graph $G$ to define certain initial/prior distributions over a carefully-chosen family of "specialists".  This approach involves sampling from the (exponentially large) space of mappings from users to actions, so a naive implementation would be computationally costly. The authors show that by sampling a random spanning tree and then embedding the users to a binary tree, the running time per step can be significantly reduced to either $O(K\log N)$ or $O(\log K\log N)$.