A Gang of Adversarial Bandits

√ ln ( NK/Ψ ) ΨKT ) and per-trial time complexity of O ( K ln ( N ) ) , whilst GABA-II has a weaker O ( √ ln ( N/Ψ ) ln ( NK/Ψ ) ΨKT ) regret , but a better O ( ln ( K ) ln ( N ) ) per-trial time complexity . We highlight improvements of both algorithms over running independent standard MABs across users . 1 Introduction . During the last decade multi-armed bandits ( MAB ) have received a great deal of attention in machine learning and related fields , due to their wide practical and theoretical importance . The central problem is to design a decision strategy whereby a learner explores sequentially the environment in order to find the best item ( arm ) within a prescribed set . At each step in the exploration the learner chooses an arm , after which feedback ( typically a loss or reward corresponding to the selected arm ) is observed from the environment . Then the next decision is made by the learner based on past interactions , and the process repeats . The goal is to design efficient exploration strategies which incur a small cumulative loss in comparison to the cumulative loss that would have been obtained by always * Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . selecting the best arm in hindsight . Applications of MAB are numerous , including recommender systems [ 1 ] , clinical trials [ 2 ] , and adaptive routing [ 3 ] , among others . In this paper we study the problem in which the learner is facing several MAB problems that are related according to a prescribed interaction graph . A main motivation behind this problem are online recommendation systems , whereby each of several users is associated with a MAB problem ( task ) , where the arms correspond to a finite set of products , and the graph represents a social network among users . The goal is to exploit users ’ similarity in order to improve the efficiency of learning users ’ preferences via online exploration of products . In the standard full information setting , a lot of work has been done showing that techniques from multitask learning are effective in reducing the amount of data needed to learn each of the individual tasks , both in the statistical and adversarial settings , see [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] and references therein . Graphs have been used to model task relationships , with different tasks ’ parameters encouraged to be close according to the graph topology . In contrast , multitask learning in the bandit setting is much less explored . The algorithms that we present exploit the network homophily principle [ 12 ] which formulates that users that are connected in the network have similar preferences , that is , they tend to share preferred recommendations . We will show that our algorithms exploit graph structure and enjoy potentially much smaller regret bounds than the cumulative regret of standard MAB run independently on each user . Since the original graph may be dense , we exploit a randomized sparsification technique to develop fast prediction algorithms . Our approach builds upon previous work on online learning over graphs [ 13 , 14 ] to generate a perfect full oriented binary tree , whose leaves are in one-to-one correspondence with the nodes of the original graph . This construction approximately preserves the relevant graph properties in expectation , and provides the starting point for designing our efficient algorithms . A further ingredient in our algorithm is provided by the method of specialists [ 15 , 16 ] . Our learning strategies combine the above ingredients to devise efficient online algorithms under partial feedback . Contributions . We introduce two Gang of Adversarial BAndit algorithms , GABA-I and GABAII that learn jointly MAB models for N users over K possible actions . Both algorithms are designed to exploit network structure while being ( extremely ) computationally efficient . We derive expected ( over the algorithms ’ randomizations ) regret bounds . The bounds scale with the dispersion measure Ψ ∈ [ 1 , N ] of the best actions over the graph . For GABA-I the bound1 is of order of O ( √ ln ( NK/Ψ ) ΨKT ) , where T is the number of trials , and has a per-trial time complexity of O ( K ln ( N ) ) . On the other hand GABA-II has a weaker expected regret bound of O ( √ ln ( N/Ψ ) ln ( NK/Ψ ) ΨKT ) but is faster , having a per-trial time complexity of O ( ln ( K ) ln ( N ) ) . Thus the GABA-I algorithm improves on algorithms that treat each user independently , as in the best case the regret improves from O ( √ N ) to O ( √ lnN ) and in the worst case the regret degrades by at most a constant factor . GABA-II has slightly weaker regret bounds ; however , it is more computationally efficient . Outline of Main Results . The social network graph G is determined by a set of undirected links between users { ωu , v } Nu < v where ωu , v ∈ [ 0 , ∞ ) indicates the magnitude of the link between user u and v. For all t ∈ [ T ] we have a user ut ∈ [ N ] and a loss vector ` t ∈ [ 0 , 1 ] K which are selected by Nature before learning begins and are unknown to Learner ; i.e. , Nature is a deterministic oblivious adversary ( see e.g. , [ 17 , Section 5.1 ] ) . Learning then proceeds in trials t = 1 , 2 , . . . , T . On trial t : 1 . Nature reveals user ut ∈ [ N ] to Learner , 2 . Learner selects action at ∈ [ K ] , 3 . Nature reveals loss ` t , at ∈ [ 0 , 1 ] to Learner . Before reflecting on the N -user case we review the well-known results for the single user ( N = 1 ) . The seminal EXP3 algorithm [ 18 ] obtains the following ( uniform ) regret bound2 , E [ ∑ t∈ [ T ] ` t , at ] − min a∈ [ K ] ∑ t∈ [ T ] ` t , a ∈ O ( √ K ln ( K ) T ) , ( 1 ) 1The bounds of GABA-I and GABA-II however depend on oracular knowledge of optimal tuning parameters . We discuss this as well as a means of lessening this dependency following Corollary 5 . 2An algorithm was given in [ 19 ] that removed the lnK term from the regret . where the expectation is with respect to the internal randomization of the EXP3 algorithm . In the N -user setting , by running a copy of EXP3 independently for each user , we may obtain a uniform regret bound of ( see e.g. , [ 20 ] ) E [ ∑ t∈ [ T ] ` t , at ] − min y : [ N ] → [ K ] ∑ t∈ [ T ] ` t , y ( ut ) ∈ O ( √ K ln ( K ) NT ) , ( 2 ) i.e. , for each user u the best action is y ( u ) and we now pay an additional constant factor of √ N in our regret . In this work we exploit the social network structure to prove a non-uniform regret bound for the GABA-I algorithm ( see Corollary 4 ) of R ( y ) : = E [ ∑ t∈ [ T ] ` t , at ] − ∑ t∈ [ T ] ` t , y ( ut ) ∈ O ( √ K ln ( KN Ψ ( y ) ) Ψ ( y ) T ) , ( 3 ) for any mapping of users to actions y : [ N ] → [ K ] . The non-uniform regret now depends on Ψ ( y ) ∈ [ 1 , N ] ( see ( 5 ) ) which measures dispersion of users ’ ‘ best ’ actions across the network . Thus , by taking network structure into account , we may upper bound the scaling in the regret with respect to the number of users by O ( √ ln ( eNΨ ( y ) ) Ψ ( y ) ) . When the best action across the network is nearly uniform then the dispersion Ψ ( y ) ∈ O ( 1 ) , in contrast when the dispersion is maximal then Ψ ( y ) = N thus in the best case the regret with respect to the number of users improves from O ( √ N ) to O ( √ lnN ) and in the worst case the regret only increases by a constant factor . The first algorithm GABA-I obtains the regret ( 3 ) while requiring O ( K lnN ) time to predict and update . The second algorithm GABA-II ’ s regret ( see Corollary 5 ) is larger by a O ( √ lnN/Ψ ( y ) ) factor but now prediction is at an even faster O ( ln ( K ) ln ( N ) ) time per trial , that is , prediction time improves exponentially with respect to the cardinality of the action set [ K ] . Thus both algorithms support very large user networks and the second algorithm allows efficient prediction with very large action sets . Related Work . We mention here some of the key papers that are more closely related to ours and refer the reader to the technical appendices for an expanded literature review . There has been much work in the heterogenous multi-user setting for linear-stochastic bandits . Out of these works , those closest to us are when the users are in a known social network and it is assumed that neighbouring users respond to context vectors in a similar way [ 21 , 22 , 23 , 24 , 25 , 26 ] but as far as we are aware no works on this model have so far been done in the adversarial setting . Other works on this topic include those in which it is assumed that there is an unknown clustering of the users , where users in the same cluster are homogenous [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ] ; as well as other models [ 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ] . There are also works on networked , homogenous multi-user bandit models with limited communication between users [ 44 , 45 , 46 , 47 , 48 , 49 , 50 ] . Related to the multiuser setting are works on transfer learning and meta-learning with linear-stochastic bandits [ 51 , 52 ] . Whilst our work assumes a known network structure over the users , there is a wide literature on bandit problems in which the actions are structured in a network and it is assumed that neighbouring actions give similar losses [ 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ] , as well as other networked-action models [ 61 , 62 , 63 ] . In addition to the seminal paper on adversarial bandits [ 18 ] , our work utilises ideas from several different papers [ 13 , 14 , 15 , 16 , 64 , 65 ] . Notation . Given a set X we define 2X to be its power-set , that is : the set of all subsets of X . For any positive integer m , we define [ m ] : = { 1 , 2 , . . . , m } . For any predicate PRED , JPREDK : = 1 if PRED is true and equals 0 otherwise . Given vectors x , x′ ∈ RK we define x · x′ to be their inner product ( i.e. , equal to ∑ i∈ [ K ] xixi′ ) and we define x x′ to be their component-wise product ( i.e. , ( x x′ ) i : = xix′i for all i ∈ [ K ] ) . We define ‘ 1 ’ to be the K-dimensional vector in which each component is equal to 1 . Given a full oriented binary tree B we denote the set of its vertices also by B . Given a non-leaf vertex n ∈ B let / ( n ) and . ( n ) be its left child and right child respectively . Given a non-root vertex n ∈ B let ↑ ( n ) be its parent . Given a vertex n ∈ B let ⇑ ( n ) and ⇓ ( n ) be its set of ancestors and leaf-descendants ( i.e . descendants which are leaves ) respectively . Given a vertex n ∈ B we define J ( n ) and I ( n ) as the left-most and right-most descendants ( which are leaves ) of n respectively . Finally , we denote the user graph by G , which is an undirected connected graph with edge weights { ωu , v : 1 ≤ u < v ≤ N } . For convenience we assume N is a power of two.3 3This assumption does not limit our results because to run our algorithms one can always add dummy vertices without altering input weights , so as to force N to be a power of two .

Summary: This paper considers the problem of contextual bandit problem with adversarial loss and finite number of contexts. Specifically, the authors consider the case where the contexts are related in some sense such that the action generated by a policy seeing one context may be related to the one seeing another similar context. The similarity is measured by a weighted undirected graph G. The authors then design two efficient algorithms which achieves $\widetilde{O}(\sqrt{\Psi(y) K T})$ regret bound in this setting where $y$ is the benchmark policy and $\Psi(y)$ is a dispersion measure of $y$ over the context. This result is better than $\widetilde{O}(\sqrt{NKT})$, where $N$ is the number of contexts.  The algorithm is mainly based on classic EXP3/EXP4 algorithm. As generally EXP4 is inefficient, the authors provide two efficient implementation by using online belief propagation over a complete binary tree for contexts, which is generated from the graph G. The first algorithm GABA-I has $O(K\ln(N))$ computation complexity per round while the second algorithm GABA-II has $O(\ln(K)\ln(N))$ computation complexity but with a factor of $\log(N)$ worse regret bound.