A Gang of Adversarial Bandits

√ ln ( NK/Ψ ) ΨKT ) and per-trial time complexity of O ( K ln ( N ) ) , whilst GABA-II has a weaker O ( √ ln ( N/Ψ ) ln ( NK/Ψ ) ΨKT ) regret , but a better O ( ln ( K ) ln ( N ) ) per-trial time complexity . We highlight improvements of both algorithms over running independent standard MABs across users . 1 Introduction . During the last decade multi-armed bandits ( MAB ) have received a great deal of attention in machine learning and related fields , due to their wide practical and theoretical importance . The central problem is to design a decision strategy whereby a learner explores sequentially the environment in order to find the best item ( arm ) within a prescribed set . At each step in the exploration the learner chooses an arm , after which feedback ( typically a loss or reward corresponding to the selected arm ) is observed from the environment . Then the next decision is made by the learner based on past interactions , and the process repeats . The goal is to design efficient exploration strategies which incur a small cumulative loss in comparison to the cumulative loss that would have been obtained by always * Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . selecting the best arm in hindsight . Applications of MAB are numerous , including recommender systems [ 1 ] , clinical trials [ 2 ] , and adaptive routing [ 3 ] , among others . In this paper we study the problem in which the learner is facing several MAB problems that are related according to a prescribed interaction graph . A main motivation behind this problem are online recommendation systems , whereby each of several users is associated with a MAB problem ( task ) , where the arms correspond to a finite set of products , and the graph represents a social network among users . The goal is to exploit users ’ similarity in order to improve the efficiency of learning users ’ preferences via online exploration of products . In the standard full information setting , a lot of work has been done showing that techniques from multitask learning are effective in reducing the amount of data needed to learn each of the individual tasks , both in the statistical and adversarial settings , see [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] and references therein . Graphs have been used to model task relationships , with different tasks ’ parameters encouraged to be close according to the graph topology . In contrast , multitask learning in the bandit setting is much less explored . The algorithms that we present exploit the network homophily principle [ 12 ] which formulates that users that are connected in the network have similar preferences , that is , they tend to share preferred recommendations . We will show that our algorithms exploit graph structure and enjoy potentially much smaller regret bounds than the cumulative regret of standard MAB run independently on each user . Since the original graph may be dense , we exploit a randomized sparsification technique to develop fast prediction algorithms . Our approach builds upon previous work on online learning over graphs [ 13 , 14 ] to generate a perfect full oriented binary tree , whose leaves are in one-to-one correspondence with the nodes of the original graph . This construction approximately preserves the relevant graph properties in expectation , and provides the starting point for designing our efficient algorithms . A further ingredient in our algorithm is provided by the method of specialists [ 15 , 16 ] . Our learning strategies combine the above ingredients to devise efficient online algorithms under partial feedback . Contributions . We introduce two Gang of Adversarial BAndit algorithms , GABA-I and GABAII that learn jointly MAB models for N users over K possible actions . Both algorithms are designed to exploit network structure while being ( extremely ) computationally efficient . We derive expected ( over the algorithms ’ randomizations ) regret bounds . The bounds scale with the dispersion measure Ψ ∈ [ 1 , N ] of the best actions over the graph . For GABA-I the bound1 is of order of O ( √ ln ( NK/Ψ ) ΨKT ) , where T is the number of trials , and has a per-trial time complexity of O ( K ln ( N ) ) . On the other hand GABA-II has a weaker expected regret bound of O ( √ ln ( N/Ψ ) ln ( NK/Ψ ) ΨKT ) but is faster , having a per-trial time complexity of O ( ln ( K ) ln ( N ) ) . Thus the GABA-I algorithm improves on algorithms that treat each user independently , as in the best case the regret improves from O ( √ N ) to O ( √ lnN ) and in the worst case the regret degrades by at most a constant factor . GABA-II has slightly weaker regret bounds ; however , it is more computationally efficient . Outline of Main Results . The social network graph G is determined by a set of undirected links between users { ωu , v } Nu < v where ωu , v ∈ [ 0 , ∞ ) indicates the magnitude of the link between user u and v. For all t ∈ [ T ] we have a user ut ∈ [ N ] and a loss vector ` t ∈ [ 0 , 1 ] K which are selected by Nature before learning begins and are unknown to Learner ; i.e. , Nature is a deterministic oblivious adversary ( see e.g. , [ 17 , Section 5.1 ] ) . Learning then proceeds in trials t = 1 , 2 , . . . , T . On trial t : 1 . Nature reveals user ut ∈ [ N ] to Learner , 2 . Learner selects action at ∈ [ K ] , 3 . Nature reveals loss ` t , at ∈ [ 0 , 1 ] to Learner . Before reflecting on the N -user case we review the well-known results for the single user ( N = 1 ) . The seminal EXP3 algorithm [ 18 ] obtains the following ( uniform ) regret bound2 , E [ ∑ t∈ [ T ] ` t , at ] − min a∈ [ K ] ∑ t∈ [ T ] ` t , a ∈ O ( √ K ln ( K ) T ) , ( 1 ) 1The bounds of GABA-I and GABA-II however depend on oracular knowledge of optimal tuning parameters . We discuss this as well as a means of lessening this dependency following Corollary 5 . 2An algorithm was given in [ 19 ] that removed the lnK term from the regret . where the expectation is with respect to the internal randomization of the EXP3 algorithm . In the N -user setting , by running a copy of EXP3 independently for each user , we may obtain a uniform regret bound of ( see e.g. , [ 20 ] ) E [ ∑ t∈ [ T ] ` t , at ] − min y : [ N ] → [ K ] ∑ t∈ [ T ] ` t , y ( ut ) ∈ O ( √ K ln ( K ) NT ) , ( 2 ) i.e. , for each user u the best action is y ( u ) and we now pay an additional constant factor of √ N in our regret . In this work we exploit the social network structure to prove a non-uniform regret bound for the GABA-I algorithm ( see Corollary 4 ) of R ( y ) : = E [ ∑ t∈ [ T ] ` t , at ] − ∑ t∈ [ T ] ` t , y ( ut ) ∈ O ( √ K ln ( KN Ψ ( y ) ) Ψ ( y ) T ) , ( 3 ) for any mapping of users to actions y : [ N ] → [ K ] . The non-uniform regret now depends on Ψ ( y ) ∈ [ 1 , N ] ( see ( 5 ) ) which measures dispersion of users ’ ‘ best ’ actions across the network . Thus , by taking network structure into account , we may upper bound the scaling in the regret with respect to the number of users by O ( √ ln ( eNΨ ( y ) ) Ψ ( y ) ) . When the best action across the network is nearly uniform then the dispersion Ψ ( y ) ∈ O ( 1 ) , in contrast when the dispersion is maximal then Ψ ( y ) = N thus in the best case the regret with respect to the number of users improves from O ( √ N ) to O ( √ lnN ) and in the worst case the regret only increases by a constant factor . The first algorithm GABA-I obtains the regret ( 3 ) while requiring O ( K lnN ) time to predict and update . The second algorithm GABA-II ’ s regret ( see Corollary 5 ) is larger by a O ( √ lnN/Ψ ( y ) ) factor but now prediction is at an even faster O ( ln ( K ) ln ( N ) ) time per trial , that is , prediction time improves exponentially with respect to the cardinality of the action set [ K ] . Thus both algorithms support very large user networks and the second algorithm allows efficient prediction with very large action sets . Related Work . We mention here some of the key papers that are more closely related to ours and refer the reader to the technical appendices for an expanded literature review . There has been much work in the heterogenous multi-user setting for linear-stochastic bandits . Out of these works , those closest to us are when the users are in a known social network and it is assumed that neighbouring users respond to context vectors in a similar way [ 21 , 22 , 23 , 24 , 25 , 26 ] but as far as we are aware no works on this model have so far been done in the adversarial setting . Other works on this topic include those in which it is assumed that there is an unknown clustering of the users , where users in the same cluster are homogenous [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 ] ; as well as other models [ 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ] . There are also works on networked , homogenous multi-user bandit models with limited communication between users [ 44 , 45 , 46 , 47 , 48 , 49 , 50 ] . Related to the multiuser setting are works on transfer learning and meta-learning with linear-stochastic bandits [ 51 , 52 ] . Whilst our work assumes a known network structure over the users , there is a wide literature on bandit problems in which the actions are structured in a network and it is assumed that neighbouring actions give similar losses [ 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ] , as well as other networked-action models [ 61 , 62 , 63 ] . In addition to the seminal paper on adversarial bandits [ 18 ] , our work utilises ideas from several different papers [ 13 , 14 , 15 , 16 , 64 , 65 ] . Notation . Given a set X we define 2X to be its power-set , that is : the set of all subsets of X . For any positive integer m , we define [ m ] : = { 1 , 2 , . . . , m } . For any predicate PRED , JPREDK : = 1 if PRED is true and equals 0 otherwise . Given vectors x , x′ ∈ RK we define x · x′ to be their inner product ( i.e. , equal to ∑ i∈ [ K ] xixi′ ) and we define x x′ to be their component-wise product ( i.e. , ( x x′ ) i : = xix′i for all i ∈ [ K ] ) . We define ‘ 1 ’ to be the K-dimensional vector in which each component is equal to 1 . Given a full oriented binary tree B we denote the set of its vertices also by B . Given a non-leaf vertex n ∈ B let / ( n ) and . ( n ) be its left child and right child respectively . Given a non-root vertex n ∈ B let ↑ ( n ) be its parent . Given a vertex n ∈ B let ⇑ ( n ) and ⇓ ( n ) be its set of ancestors and leaf-descendants ( i.e . descendants which are leaves ) respectively . Given a vertex n ∈ B we define J ( n ) and I ( n ) as the left-most and right-most descendants ( which are leaves ) of n respectively . Finally , we denote the user graph by G , which is an undirected connected graph with edge weights { ωu , v : 1 ≤ u < v ≤ N } . For convenience we assume N is a power of two.3 3This assumption does not limit our results because to run our algorithms one can always add dummy vertices without altering input weights , so as to force N to be a power of two .

Summary: This paper studies the following contextual variant of the adversarial multi-armed bandit problem. Each round t (for T rounds), the adversary begins by choosing a context (“user”) u_t in [N] and a loss function l_t from [K] -> R, and reveals the context to the learning algorithm. Based on this context, the learner must select an action a_t in [K]. The adversary then reveals the value of l_t(a_t) to the learner, which the learner suffers as loss. By running a separate instance of EXP3 for each context, it is possible for the learner to achieve a total loss of O(sqrt(NKT)). This paper asks the question of whether we can do better if we believe that “similar” users should be subject to “similar” losses.  The authors answer this in the following way. They assume there is some given graph structure on the N users (“social network graph”), where users i and j are connected with a link of weight w_{i, j} (and higher weights mean users are more likely to have similar losses). They then give the following non-uniform regret bound: against the fixed policy y : [N] -> [K], their algorithms get regret of the form ~ O(sqrt(KT f(y))), where f(y) is a function that measures how “consistent” y is with the social network graph. For example, if y(s) is a constant function (doesn’t depend on user), then this is very consistent with any graph structure, and f(y) ~= log(N). On the other hand, for arbitrary y, y(s) can be as large as ~O(N), recovering the original ~O(sqrt(KTN)) bound. Formally, f(y) is something like the average effective resistance between users with different values of y(s).   The author’s algorithms are based on a technique they call “predicting with specialists” (extending the work of Freund et al.), where one maintains a distribution over functions mapping contexts to either actions or the decision to abstain. In particular, it’s possible to get a non-uniform regret bound for such an algorithm that only depends on the starting distribution and the policy to which the algorithm is being compared (similar in flavor to the regret bounds above). The authors then define two different starting distributions which lead to two algorithms (GABA-I and GABA-II) with good regret bounds for their setting. The first starting distribution has a higher support size (and larger eventual runtime), but lower overall regret than the second. The second algorithm is based off a clever embedding of the social network graph into a full binary tree that partially captures the connectedness of the original network graph.  Finally, the authors demonstrate how to implement their algorithms efficiently. In particular, even though the first distribution has an exponentially large support size (and thus naively would require exponential time / space to run), they show how to implement it in only O(K log N) time per round via a clever online belief propagation algorithm. Similarly, they show how to implement GABA-II in only O(log K log N) time per round (exponentially faster than GABA-1).