Fair Algorithms for Multi-Agent Multi-Armed Bandits

1 Introduction . In the classical ( stochastic ) multi-armed bandit ( MAB ) problem , a principal has access to K arms and pulling arm j generates a stochastic reward for the principal from an unknown distribution Dj with an unknown mean µ∗j . If the mean rewards were known a priori , the principal could just repeatedly pull the best arm given by arg maxj µ ∗ j . However , the principal has no apriori knowledge of the quality of the arms . Hence , she uses a learning algorithm which operates in rounds , pulls arm jt in round t , observes the stochastic reward generated , and uses that information to learn the best arm over time . The performance of such an algorithm is measured in terms of its cumulative regret up to a horizon T , defined as ∑T t=1 ( maxj µ ∗ j − µ∗jt ) . Note that this is the difference between the total mean reward that would have been achieved if the best arm was pulled repeatedly and the total mean reward of the arms pulled by the learning algorithm up to round T . This problem can model situations where the principal is deliberating a policy decision and the arms correspond to the different alternatives she can implement . However , in many real-life scenarios , making a policy decision affects not one , but several agents . For example , imagine a company making a decision that affects all its employees , or a conference deciding the structure of its review process , which affects various research communities . This can be modeled by a multi-agent variant of the multi-armed bandit ( MA-MAB ) problem , in which there are N agents and pulling arm j generates a ( possibly different ) stochastic reward for each agent i from an unknown distribution Di , j with an unknown mean µ∗i , j . Before pondering about learning the “ best arm ” over time , we must ask what the best arm even means in this context . Indeed , the “ best arm ” for one agent may not be the best for another . A first attempt may be to associate some “ aggregate quality ” to each arm ; for example , the quality of arm 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . j may be defined as the total mean reward it gives to all agents , i.e. , ∑ i µ ∗ i , j . This would nicely reduce our problem to the classic multi-armed bandit problem , for which we have an armory of available solutions [ 1 ] . However , this approach suffers from the tyranny of the majority [ 2 ] . For example , imagine a scenario with ten agents , two arms , and deterministic rewards . Suppose four agents derive a reward of 1 from the first arm but 0 from the second , while the remaining six derive a reward of 1 from the second arm but 0 from the first . The aforementioned approach will deem the second arm as the best and a classical MAB algorithm will converge to repeatedly pulling the second arm , thus unfairly treating the first four agents ( a minority ) . A solution which treats each group in a “ proportionally fair ” [ 2 ] manner should ideally converge to pulling the first arm 40 % of the time and the second 60 % of the time . Alternatively , we can allow the learning algorithm to “ pull ” a probability distribution over the arms and seek an algorithm that converges to placing probability 0.4 on the first arm and 0.6 on the second . This problem of making a fair collective decision when the available alternatives — in this case , probability distributions over the arms — affect multiple agents is well-studied in computational social choice [ 3 ] . The literature offers a compelling fairness notion called the Nash social welfare , named after John Nash . According to this criterion , the fairest distribution maximizes the product of the expected utilities ( rewards ) to the agents . A distribution p that places probability pj on each arm j gives expected utility ∑ j pj · µ∗i , j to agent i . Hence , the goal is to maximize NSW ( p , µ∗ ) =∏N i=1 ( ∑K j=1 pj · µ∗i , j ) over p. One can verify that this approach on the aforementioned example indeed yields probability 0.4 on the first arm and 0.6 on the second , as desired . It is also interesting to point out that with a single agent ( N = 1 ) , the distribution maximizing the Nash social welfare puts probability 1 on the best arm , thus effectively reducing the problem to the classical multi-armed bandit problem ( albeit with subtle differences which we highlight in Appendix A ) . Maximizing the Nash social welfare is often seen as a middle ground between maximizing the utilitarian social welfare ( sum of utilities to the agents ) , which is unfair to minorities ( as we observed ) , and maximizing the egalitarian social welfare ( minimum utility to any agent ) , which is considered too extreme [ 2 ] . The solution maximizing the Nash social welfare is also known to satisfy other qualitative fairness desiderata across a wide variety of settings [ 4–11 ] . For example , a folklore result shows that in our setting such a solution will always lie in the core ; we refer the reader to the work of Fain et al . [ 4 ] for a formal definition of the core as well as a short derivation of this fact using the first-order optimality condition . For further discussion on this , see Sections 1.2 and 6 . When exactly maximizing the Nash social welfare is not possible ( either due to a lack of complete information , as in our case , or due to computational difficulty ) , researchers have sought to achieve approximate fairness by approximately maximizing this objective [ 12–17 ] . Following this approach in our problem , we define the ( cumulative ) regret of an algorithm at horizon T as∑T t=1 ( maxp NSW ( p , µ ∗ ) − NSW ( pt , µ∗ ) ) , where pt is the distribution selected in round t. Our goal in this paper is to design algorithms whose regret is sublinear in T . 1.1 Our Results . We consider three classic algorithms for the multi-armed bandit problem : Explore-First , EpsilonGreedy , and UCB [ 1 ] . All three algorithms attempt to balance exploration ( pulling arms only to learn their rewards ) and exploitation ( using the information learned so far to pull “ good ” arms ) . Explore-First performs exploration for a number of rounds optimized as a function of T followed by exploitation in the remaining rounds to achieve regret Õ ( K1/3T 2/3 ) . Epsilon-Greedy flips a coin in each round to decide whether to perform exploration or exploitation and achieves the same regret bound . Its key advantage over Explore-First is that it does not need to know the horizon T upfront . UCB merges exploration and exploitation to achieve a regret bound of Õ ( K1/2T 1/2 ) . Here , Õ hides log factors . Traditionally , the focus is on optimizing the exponent of T rather than that of K as the horizon T is often much larger than the number of arms K. It is known that the dependence of UCB ’ s regret on T is optimal : no algorithm can achieve instance-independent o ( T 1/2 ) regret [ 18 ] .1 1In instance-independent bounds , the constant inside the big-Oh notation is not allowed to depend on the ( unknown ) distributions in the given instance . UCB also achieves an O ( log T ) instance-dependent regret bound , which is also known to be asymptotically optimal [ 19 ] . For further discussion , see Section 6 . We propose natural multi-agent variants of these three algorithms . Our variants take the Nash social welfare objective into account and select a distribution over the arms in each round instead of a single arm . For Explore-First , we derive Õ ( N2/3K1/3T 2/3 ) regret bound , which recovers the aforementioned single-agent bound with an additional factor of N2/3 . We also show that changing a parameter of the algorithm yields a regret bound of Õ ( N1/3K2/3T 2/3 ) , which offers a different tradeoff between the dependence on N and K. For Epsilon-Greedy , we recover the same two regret bounds , although the analysis becomes much more intricate . This is because , as mentioned above , Epsilon-Greedy is a horizon-independent algorithm ( i.e . it does not require apriori knowledge of T ) , unlike Explore-First . For UCB , we derive Õ ( NKT 1/2 ) and Õ ( N1/2K 3 2T 1/2 ) regret bounds ; our dependence on K worsens compared to the classical single-agent case , but importantly , we recover the same √ T dependence , which is provably optimal ( see Appendix A ) . Deriving these regret bounds for the multi-agent case requires overcoming two key difficulties that do not appear in the single-agent case . First , our goal is to optimize a complicated function , the Nash social welfare , rather than simply selecting the best arm . This requires a Lipschitz-continuity analysis of the Nash social welfare function and the use of new tools such as the McDiarmid ’ s inequality which are not needed in the standard analysis . Second , the optimization is over an infinite space ( the set of distributions over arms ) rather than over a finite space ( the set of arms ) . Thus , certain tricks such as a simple union bound no longer work ; we use the concept of δ-covering , used heavily in the Lipschitz bandit framework [ 20 ] , in order to address this . Our contributions are twofold . Conceptually , we promote a multi-agent viewpoint that requires striking a tradeoff between multiple reward functions ; this can be applied to other classical singleagent problems . More technically , as explained below , the existing MAB literature does not provide a√ T bound for the Nash social welfare objective . Our regret bounds may be generalizable to a broader family of objectives for which a √ T bound was previously unknown . 1.2 Related Work . Since the multi-armed bandit problem was introduced by Thompson [ 21 ] , many variants of it have been proposed , such as sleeping bandit [ 22 ] , contextual bandit [ 23 ] , dueling bandit [ 24 ] , Lipschitz bandit [ 20 ] , etc . However , all these variants involve optimizing the cumulative regret or identifying the Pareto frontier of multiple objectives [ 25 ] from the perspective of a single agent . We note that other multi-agent variants of the multi-armed bandit problem have been explored recently [ 26 , 27 ] , including in distributed environments [ 28–30 ] . However , they still involve a common reward like in the classical multi-armed bandit problem . Their focus is on getting the agents to cooperate to maximize this common reward . It is possible to view our work from a single-agent prespective by treating theK-simplex of probability distributions over arms as a “ continuum of arms ” and the Nash welfare as a non-linear objective to be optimized . Kleinberg et al . [ 31 ] explore continuum-armed bandits , but establish a Õ ( T γ+1 γ+2 ) regret bound , where γ is the zooming dimension , which would be Θ ( K ) in our case . Bubeck et al . [ 32 ] also study continuum-armed bandits and establish a similar regret bound , but with respect to the near-optimality dimension , which is again Θ ( K ) in our case . Hence , the guarantees of Kleinberg et al . [ 31 ] and Bubeck et al . [ 32 ] , when applied to our setting , are worse than our Õ ( √ T ) regret bound . The literature on bandit convex optimization [ 33–35 ] offers Õ ( √ T ) regret bound , but requires the objective to be concave , which the Nash social welfare ( NSW ) is not.2 Another key aspect of our framework is the focus on fairness . Recently , several papers have focused on fairness in the multi-armed bandit problem . For instance , Joseph et al . [ 37 ] design a UCB variant which guarantees what they refer to as meritocratic fairness to the arms , i.e. , that a worse arm is never preferred to a better arm regardless of the algorithm ’ s confidence intervals for them . Liu et al . [ 38 ] require that similar arms be treated similarly , i.e. , two arms with similar mean rewards be selected with similar probabilities . Gillen et al . [ 39 ] focus on satisfying fairness with respect to an unknown fairness metric . Finally , Patil et al . [ 40 ] assume that there are external constraints requiring that 2Since NSW is log-concave , one might argue that we can apply these results to log-NSW . However , a regret bound in log-NSW does not imply the desired regret bound in NSW . each arm be pulled in at least a certain fraction of the rounds and design algorithms that achieve low regret subject to this constraint . All these papers seek to achieve fairness with respect to the arms . In contrast , in our work , the arms are “ inanimate ” ( e.g . policy decisions ) and we seek fairness with respect to the agents , who are separate from the arms .

Summary: This paper studies a multi-armed bandits problem with multiple agents. Arms have agent specific reward distributions associated with them. An arm is pulled at each step and each agent receives a reward sampled from their corresponding agent specific reward distribution for this arm. The goal of the learner is to find a distribution over arms that maximizes an appropriate aggregate notion of rewards earned by all agents – the Nash welfare.  The paper develops variants of three standard algorithms – one that explores for a fixed number of rounds and then exploits (Explore-First), an epsilon-greedy style algorithm, and a UCB style algorithm, to minimize a notion of regret which measures the difference between the Nash welfare of the solution at time t and the optimal Nash welfare. The paper derives upper bounds on the expected regret of the algorithm in each case. For UCB, the upper bound matches the lower bound (in the appendix) in terms of its dependence on parameter T (time horizon).