Title: No-Regret Learning for Fair Multi-Agent Social Welfare Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Online Learning

Nash Social Welfare
Abstract: We consider the problem of online multi-agent Nash social welfare (NSW) maximization. While previous works of Hossain et al. [2021], Jones et al. [2023] study similar problems in stochastic multi-agent multi-armed bandits and show that $\sqrt{T}$-regret is possible after $T$ rounds, their fairness measure is the product of all agents' rewards, instead of their NSW (that is, their geometric mean). Given the fundamental role of NSW in the fairness literature, it is more than natural to ask whether no-regret fair learning with NSW as the objective is possible. In this work, we provide a complete answer to this question in various settings. Specifically, in stochastic $N$-agent $K$-armed bandits, we develop an algorithm with $\widetilde{\mathcal{O}}(K^{\frac{2}{N}}T^{\frac{N-1}{N}})$ regret and prove that the dependence on $T$ is tight, making it a sharp contrast to the $\sqrt{T}$-regret bounds of Hossain et al. [2021], Jones et al. [2023]. We then consider a more challenging version of the problem with adversarial rewards. Somewhat surprisingly, despite NSW being a concave function, we prove that no algorithm can achieve sublinear regret. To circumvent such negative results, we further consider a setting with full-information feedback and design two algorithms with $\sqrt{T}$-regret: the first one has no dependence on $N$ at all and is applicable to not just NSW but a broad class of welfare functions, while the second one has better dependence on $K$ and is preferable when $N$ is small.
Finally, we also show that logarithmic regret is possible whenever there exists one agent who is indifferent about different arms.

No-Regret Learning for Fair Multi-Agent Social Welfare Optimization

Abstract

We consider the problem of online multi-agent Nash social welfare (NSW) maximization. While previous works of Hossain et al. [2021], Jones et al. [2023] study similar problems in stochastic multi-agent multi-armed bandits and show that √ T -regret is possible after T rounds, their fairness measure is the product of all agents' rewards, instead of their NSW (that is, their geometric mean). Given the fundamental role of NSW in the fairness literature, it is more than natural to ask whether no-regret fair learning with NSW as the objective is possible. In this work, we provide a complete answer to this question in various settings. Specifically, in stochastic N -agent K -armed bandits, we develop an algorithm with ˜ O ( K 2 N T N -1 N ) regret and prove that the dependence on T is tight, making it a sharp contrast to the √ T -regret bounds of Hossain et al. [2021], Jones et al. [2023]. We then consider a more challenging version of the problem with adversarial rewards. Somewhat surprisingly, despite NSW being a concave function, we prove that no algorithm can achieve sublinear regret. To circumvent such negative results, we further consider a setting with full-information feedback and design two algorithms with √ T -regret: the first one has no dependence on N at all and is applicable to not just NSW but a broad class of welfare functions, while the second one has better dependence on K and is preferable when N is small. Finally, we also show that logarithmic regret is possible whenever there exists one agent who is indifferent about different arms.

1 Introduction

In this paper, we study online multi-agent Nash social welfare (NSW) maximization, which is a generalization of the classic multi-armed bandit (MAB) problem [Thompson, 1933, Lai and Robbins, 1985]. Different from MAB, in which the learner makes her decisions sequentially in order to maximize her own reward, in online multi-agent NSW maximization, the learner's decision affects multiple agents and the goal is to maximize the NSW over all the agents. Specifically, NSW is defined as the geometric mean of the expected utilities over all agents [Moulin, 2004], which can be viewed as a measure of fairness among the agents. This problem includes many important real-life applications such as resource allocation [Jones et al., 2023], where the learner needs to guarantee fair allocations among multiple agents. We refer the readers to [Hossain et al., 2021, Jones et al., 2023] for more applications of NSW maximization.

Recent work by Hossain et al. [2021], Jones et al. [2023] studies a similar problem but with NSWprod as the objective, a variant of NSW that is defined as the product of the utilities over agents instead of

their geometric mean. While the optimal strategy is the same if the utility for each agent is stationary, this is not the case with a non-stationary environment. Moreover, NSWprod is homogeneous of degree N instead of degree 1 , where N is the number of agents, meaning that NSWprod is more sensitive to the scale of the utility. Specifically, if the utilities of each agent are scaled by 2 , then NSW is scaled by 2 as well, but NSWprod is scaled by 2 N . Therefore, it is arguably more reasonable to consider regret with respect to NSW, which has not been studied before (to our knowledge) and is the main objective of our work.

From a technical perspective, however, due to the lack of Lipschitzness, NSW poses much more challenges in regret minimization compared to NSWprod. For example, one cannot directly apply the algorithm for Lipschitz bandits [Kleinberg et al., 2019] to our problem, while it is directly applicable to NSWprod as mentioned in [Hossain et al., 2021, Jones et al., 2023]. Despite such challenges, we manage to provide complete answers to this problem in various setting. Specifically, our contributions are listed below (where T, N , and K denote the number of rounds, agents, and arms/actions respectively):

(Section 3) We first study the stochastic bandit setting, where the utility matrix at each round is i.i.d. drawn from an unknown distribution, and the learner can only observe the utilities (for different agents) of the action she picked. In this case, we develop an algorithm with ˜ O ( K 2 N T N -1 N ) regret. 1 While our algorithm is also naturally based on the Upper Confidence Bound (UCB) algorithm as in Hossain et al. [2021], Jones et al. [2023], we show that a novel analysis with Bernstein-type confidence intervals is important for handling the lack of Lipschitzness of NSW. Moreover, we prove a lower bound of order ˜ Ω( 1 N 3 · K 1 N T N -1 N ) , showing that the dependence on T is tight. This is in sharp contrast to the √ T -regret bound of Hossain et al. [2021], Jones et al. [2023] and demonstrates the difficulty of learning with NSW compared to NSWprod.

(Section 4.1) We then consider a more challenging setting where the utility matrix at each round can be adversarially chosen by the environment. Somewhat surprisingly, we show that no algorithm can achieve sublinear regret in this case, despite NSW being concave and the vast literature on bandit online maximization with concave utility functions (the subtlety lies in the slightly different feedback model). In fact, the same impossibility result also holds for NSWprod as we show.

(Section 4.2) To bypass such impossibility, we further consider this adversarial setting under richer feedback, where the learner observes the full utility matrix after her decision (the so-called fullinformation feedback). Contrary to the bandit feedback setting, learning is not only possible now but can also be much faster despite having adversarial utilities. Specifically, we design two different algorithms with √ T -regret. The first algorithm is based on Follow-the-Regularized-Leader (FTRL) with the log-barrier regularizer, which achieves O ( √ KT log T ) regret (Section 4.2.1). Notably, this algorithm does not have any dependence on the number of agents N and can also be generalized to a broader class of social welfare functions. The second algorithm is based on FTRL with a Tsallis entropy regularizer, which achieves ˜ O ( K 1 2 -1 N √ NT ) regret and is thus more favorable when K is much larger than N (Section 4.2.2). Finally, we also show that improved logarithmic regret is possible as long as at each round there exists at least one agent who is indifferent about the learner's choice of arm (Section 4.2.3).

1.1 Related Work

Hossain et al. [2021], Jones et al. [2023] are most related to our work. Hossain et al. [2021] is the first to consider designing no-regret algorithms under NSWprod for the stochastic multi-agent multi-armed bandit problem. Specifically, they propose two algorithms. The first one is based on ε -greedy and achieves O ( T 2 3 ) regret efficiently, and the second one is based on UCB and achieves ˜ O ( √ T ) regret inefficiently. Jones et al. [2023] improves these results by providing a better UCB-based algorithm that is efficient and achieves the same ˜ O ( √ T ) regret. To the best of our knowledge, there are no previous results for regret minimization over NSW under this particular setup.

However, several other models of fairness have been introduced in (single-agent or multi-agent) multi-armed bandits, some using NSW as well. These models differ in whether they aim to be fair among different objectives, different arms, different agents, different rounds, or others. Most

related to this paper is multi-objective bandits, in which the learner tries to increase different and possibly competing objectives in a fair manner. For example, Drugan and Nowe [2013] introduces the multi-objective stochastic bandit problem and offers a regret measure to explore Pareto Optimal solutions, and Busa-Fekete et al. [2017] investigates the same setting using the Generalized Gini Index in their regret measure to promote fairness over objectives. Their regret measure closely resembles the one we use, except they apply some social welfare function (SWF) to the cumulative expected utility of agents over all rounds as opposed to the expected utility of agents each round. On the other hand, some other works study fairness among different rounds which incentivizes the learner to perform well consistently over all rounds [Barman et al., 2023, Sawarni et al., 2024]. Besides, there are other models that measure fairness in different ways, including how often each arm is pulled [Joseph et al., 2016, Liu et al., 2017, Gillen et al., 2018, Chen et al., 2020] and how the regret is allocated across different groups [Baek and Farias, 2021].

Kaneko and Nakamura [1979] axiomatically derives the NSW function. It is a fundamental and widely-adopted fairness measure and is especially popular for the task of fairly allocating goods. Caragiannis et al. [2019] justifies the fairness of NSW by showing that its maximum solution ensures some desirable envy-free property. This result prompted the design of approximation algorithms for the problem of allocating indivisible goods by maximizing NSW, which is known to be NP-hard even for simple valuation functions [Barman et al., 2018, Cole and Gkatzelis, 2015, Garg et al., 2023, Li and Vondrák, 2021].

There is a vast literature on the multi-armed bandit problem; see the book by Lattimore and Szepesvári [2020] for extensive discussions. The standard algorithm for the stochastic setting is UCB [Lai and Robbins, 1985, Auer et al., 2002a], while the standard algorithm for the adversarial setting is FTRL or the closely related Online Mirror Descent (OMD) algorithm [Auer et al., 2002b, Audibert and Bubeck, 2010, Abernethy et al., 2015]. For FTRL/OMD, the log-barrier or Tsallis entropy regularizers have been extensively studied in recent years due to some of their surprising properties (e.g., [Foster et al., 2016, Wei and Luo, 2018, Zimmert and Seldin, 2019, Lee et al., 2020]). They are rarely used in the full-information setting as far as we know, but our analysis reveals that they are useful even in such settings, especially for dealing with the lack of Lipschitzness of NSW.

2 Preliminaries

General Notation. Throughout this paper, we denote the set { 1 , 2 , . . . , n } by [ n ] for any positive integer n . For a matrix M ∈ R m × n , we denote the i -th row vector of M by M i, : ∈ R n , the j -th column vector of M by M : ,j ∈ R m , and the ( i, j ) -th entry of M by M i,j . We say M ⪰ 0 if M is a positive semi-definite matrix. The ( K -1) -dimensional simplex is denoted as ∆ K , and its clipped version with a parameter δ > 0 is denoted as ∆ K,δ = { p ∈ ∆ K | p i ≥ δ, ∀ i ∈ [ K ] } . We use 0 and 1 to denote the all-zero and all-one vector in an appropriate dimension. For two random variables X and Y , we use X d = Y to say X is equivalent to Y in distribution.

For a twice differentiable function f , we use ∇ f ( · ) and ∇ 2 f ( · ) to denote its gradient and Hessian. For concave functions that are not differentiable, ∇ f ( · ) denotes a super-gradient. Throughout the paper, we study functions of the form f ( u ⊤ p ) for u ∈ [0 , 1] m × n and p ∈ ∆ m . In such cases, the gradient, super-gradient, or hessian are all with respect to p unless denoted otherwise (for example, we write ∇ u f ( u ⊤ p ) , with an explicit subscript u , to denote the gradient with respect to u ).

Social Welfare Functions A social welfare function (SWF) f : [0 , 1] N → [0 , 1] measures the desirability of the agents' expected utilities. Specifically, for two different vectors of expected utilities µ, µ ′ ∈ [0 , 1] N , f ( µ ) > f ( µ ′ ) means that µ is a fairer alternative than µ ′ . In each setting we explore, each action by the learner yields some expected utility for each of the N agents, and the learner's goal is maximize some SWF applied to these N expected utilities.

Nash Social Welfare (NSW) For the majority of this paper, we focus on a specific type of SWF, namely the Nash Social Welfare (NSW) function [Nash, 1950, Kaneko and Nakamura, 1979]. Specifically, for µ ∈ [0 , 1] N , NSW is defined as the geometric mean of the N coordinates:

As mentioned, Hossain et al. [2021], Jones et al. [2023] considered a closely related variant that is simply the product of the coordinates: NSWprod ( µ ) = ∏ n ∈ [ N ] µ n . It is clear that NSW has a better scaling property since it is homogeneous: scaling each µ n by a constant c also scales NSW ( µ ) by c , but it scales NSWprod ( µ ) by c N . This makes NSWprod an unnatural learning objective, which motivates us to use NSW as our choice of SWF. Learning with NSW, however, brings extra challenges since it is not Lipschitz in the small-utility regime (while NSWprod is Lipschitz over the entire [0 , 1] N ). We shall see in subsequent sections how we address such challenges.

We remark that while our main focus is regret minimization with respect to NSW, some of our results also apply to NSWprod or more general classes of SWFs (as will become clear later).

Problem Setup. The N -agent K -armed social welfare optimization problem we consider is defined as follows (with N ≥ 2 and K ≥ 2 throughout). Ahead of time, with the knowledge of the learner's algorithm, the environment decides T utility matrices u 1 , . . . , u T ∈ [0 , 1] K × N , where u t,i,n is the utility of agent n if arm/action i is selected at round t . Then, the learner interacts with the environment for T rounds: at each round t , the learner decides a distribution p t ∈ ∆ K and then samples an action i t ∼ p t . In the full-information feedback setting, the learner observes the full utility matrix u t after her decision, and in the bandit feedback setting, the learner only observes u t,i t ,n for each agent n ∈ [ N ] , that is, the utilities of the selected action.

We consider two different types of environments, the stochastic one and the adversarial one, with a slight difference in their regret definition. In the stochastic environment, there exists a mean utility matrix u ∈ [0 , 1] K × N such that at each round t , u t is an i.i.d. random variable with mean u . Fix an SWF f . The social welfare of a strategy p ∈ ∆ K is defined as f ( u ⊤ p ) , which is with respect to the agents' expected utilities over the randomness of both the learner's and the environment's. The regret is then defined as follows:

which is the difference between the total social welfare of the optimal strategy and that of the learner. When f is chosen to be NSWprod, Eq. (2) reduces to the regret notion considered in Hossain et al. [2021], Jones et al. [2023].

On the other hand, in the adversarial environment, we do not make any distributional assumption on the utility matrices and allow them to be selected arbitrarily. The social welfare of a strategy p ∈ ∆ K for time t is defined as f ( u ⊤ t p ) , and the overall regret of the learner is correspondingly defined as:

In both Eq. (2) and Eq. (3), the expectation is taken with respect to the randomness of the algorithm.

Social welfare of expected utilities versus expected social welfare of realized utilities. One might wonder why we measure fairness using the social welfare of expected utilities (e.g., f ( u ⊤ p ) ), instead of the expected social welfare of realized utilities (e.g., E i ∼ p [ f ( u ⊤ e i )] ). This is because the former is arguably more meaningful as a fairness measure. To see this, consider f = NSW or f = NSWprod and a setting with 2 agents, 2 arms, and u being the identity matrix. Then, in terms of f ( u ⊤ p ) , the uniform distribution is the best policy (which makes sense from a fairness viewpoint), while in terms of E i ∼ p [ f ( u ⊤ e i )] , all distributions achieve the same value of 0, implying that all polices are as fair, which is clearly undesired.

Connection to Bandit Convex optimization. When taking f = NSW (our main focus) and considering the bandit feedback setting, our problem is seemingly an instance of the heavily-studied Bandit Convex optimization (BCO) problem, since -NSW is convex. However, there is a slight but critical difference in the feedback model: a BCO algorithm would require observing f ( u ⊤ t p t ) , or equivalently u ⊤ t p t , at the end of each round t , while in our problem the learner only observes u t,i t , : , a much more realistic scenario. Even though they have the same expectation, due to the non-linearity of NSW, this slight difference in the feedback turns out to cause a huge difference in terms of learning √

-the minimax regret for BCO is known to be Θ( T ) , while in our problem (with bandit feedback),

Algorithm 1 UCB for N -agent K -armed NSW maximization

Input: warm-up phase length

Initialization:

1

,i,n

u

= 1

for

t

= 1

,

2

, . . . , T

̂

if

t

do

KN

0

then else

calculate

t

select

i

=

p

t

= argmax

Observe

t

t,i

,n for all

u

Update counters

N

for all

0

n

>

∈

⌈

p

∈

n

[

∈

0

[

t

N

∆

N

.

N

]

,

i

∈

⌉

0

K

]

.

N

t

+1

,i

t

=

N

t,i

t

;

NSW

+1

[

K

⊤

t

̂

(

u

and

Update upper confidence utility matrix:

for all n ∈ [ N ] and i ∈ [ K ] where ¯ u t,i,n = 1 N t +1 ,i ∑ τ ≤ t u τ,i,n ✶ { i τ = i } . end as we will soon show, the regret is either Θ( T N -1 N ) in the stochastic setting or even Ω( T ) in the adversarial setting. Therefore, in a sense our problem is much more difficult than BCO. For more details on BCO, we refer the reader to a recent survey by Lattimore [2024].

3 Stochastic Environments with Bandit Feedback

In this section, we consider regret minimization over f = NSWwith bandit feedback in the stochastic setting, where the utility matrix u t at each round t ∈ [ T ] is i.i.d. drawn from a distribution with mean u . Again, this is the same setup as Hossain et al. [2021], Jones et al. [2023] except that NSWprod is replaced with NSW.

3.1 Upper Bound: a Refined Analysis of UCB with a Bernstein-Type Confidence Set

We start by describing our algorithm and its regret guarantee, followed by discussion on what the key ideas are and how the algorithm/analysis is different from previous work. Specifically, our algorithm, shown in Algorithm 1, is based on the classic UCB algorithm. It starts by picking each action for N 0 = ˜ O (1) rounds. After this warm-up phase, at each time t , the algorithm picks the optimal strategy p t that maximizes the NSW with respect to some upper confidence utility matrix ̂ u t . After sampling an action i t ∼ p t , the algorithm observes the utility of each agent for action i t and then updates the upper confidence utility matrix ̂ u t +1 as the empirical average utility plus a certain Bernstein-type confidence width (Eq. (4)).

The following theorem shows that Algorithm 1 guarantees ˜ O ( K 2 N T K -1 K ) expected regret (with f , in the definition of Reg sto , set to NSW; the same below unless stated otherwise). The full proof can be found in Appendix A.

Other than replacing NSWprod with NSW, our algorithm differs from that of Jones et al. [2023] in the form of the confidence width, and the analysis sketch below explains why we need this change. Specifically, for either f = NSWprod or f = NSW, standard analysis of UCB states that the regret is bounded by ∑ T t =1 ∣ ∣ f ( ̂ u ⊤ t p t ) -f ( u ⊤ p t ) ∣ ∣ . When f is NSWprod, a Lipschitz function, Hossain et al. [2021, Lemma 3] shows

and the rest of the analysis follows by direct calculations. However, when f is NSW, a non-Lipschitz function, we cannot expect something similar to Eq. (5) anymore. Indeed, direct calculation shows that the Lipschitz constant of NSW ( u ⊤ p ) with respect to u : ,n equals to Θ ( ∑ N n =1 ⟨ p, u : ,n ⟩ -N -1 N ) , which can be arbitrarily large when ⟨ p, u : ,n ⟩ is close to 0 for some n ∈ [ N ] and N ≥ 2 .

]

p

.

)

N

t

1

,i for all

N

= 0

and select

+1

i

i

t

p

∼

,i

=

N

t,i for

i

t

̸

=

∈

;

i

t

[

K

.

]

.

To handle this issue, we require a more careful analysis. Specifically, using Freedman's inequality, we know that with a high probability,

With the help of Eq. (6), we consider two different cases at each round t . The first case is that there exists certain n ∈ [ N ] such that ⟨ p t , u : ,n ⟩ ≤ σ for some σ > 0 to be chosen later. In this case, we use Eq. (6) to show

where the first inequality uses Eq. (6) and the second inequality is because NSW ( u ⊤ p t ) ≤ ⟨ p t , u n ⟩ 1 N for any n ∈ [ N ] . For the second term in Eq. (7), a standard analysis shows that it is upper bounded by ˜ O ( K 1 N T N -1 N ) .

Now we consider the case where ⟨ p t , u : ,n ⟩ ≥ σ for all n ∈ [ N ] . In this case, via a decomposition lemma (Lemma C.1), we show that

To bound Eq. (8), we use Eq. (6) again:

where the last inequality is due to the condition ⟨ p t , u : ,n ⟩ ≥ σ for all n ∈ [ N ] . Finally, combining Eq. (7), Eq. (8), Eq. (9), followed by direct calculations, we show that

Picking the optimal choice of σ finishes the proof.

Wenowhighlight the importance of using a Bernstein-type confidence width in Eq. (4): if the standard Hoeffding-type confidence width is used instead, then one can only obtain ̂ u t,i,n -u i,n ≤ O ( √ 1 N t,i ) , and consequently, Eq. (8) can only be bounded by O ( σ -N -1 N √ KT ) after taking summation over t ∈ [ T ] . This eventually leads to a worse regret bound of ˜ O ( K 1 2 N T 2 N -1 2 N ) .

3.2 Lower Bound

Next, we prove an ˜ Ω( T N -1 N ) lower bound for this setting. This not only shows that the regret bound we achieve via Algorithm 1 is tight in T , but also highlights the difference and difficulty of learning with NSW compared to learning with NSWprod, since in the latter case, Θ( √ T ) regret is minimax optimal [Hossain et al., 2021, Jones et al., 2023].

Theorem 3.2. In the bandit feedback setting, for any algorithm, there exists a stochastic environment in which the expected regret (with respect to NSW ) of this algorithm is Ω ( (log K ) 3 N 3 · K 1 N T N -1 N ) for N ≥ log K and sufficiently large T .

We defer the full proof to Appendix A.2 and discuss the hard instance used in the proof below. First, the mean utility vector u : ,n for each agent n ≥ 2 is a constant vector 1 . This makes the problem

equivalent to a one-agent problem, but with ⟨ p, u : , 1 ⟩ 1 /N as the reward, instead of ⟨ p, u : , 1 ⟩ as in standard stochastic K -armed bandits.

Then, for the first agent, different from the standard K -armed bandits, where the hardest instance is to hide one arm with a slightly better expected reward of 1 2 + √ K/T among other K -1 arms with expected reward of exactly 1 2 , 2 we hide one arm with expected reward K/T among other K -1 arms with exactly 0 reward (so overall the rewards are shifted towards 0 but with a smaller gap between the best arm and the others). By standard information theory arguments, within T rounds the learner cannot distinguish the best arm from the others. Therefore, the best strategy she can apply is to pick a uniform distribution over actions, suffering Ω((1 -K -1 N ) · ( K/T ) 1 N ) = ˜ Ω( K 1 N T -1 N ) regret per round and leading to ˜ Ω( K 1 N T N -1 N ) regret overall.

4 Adversarial Environments

Now that we have a complete answer for the stochastic setting, we move on to consider the adversarial case where each u t is chosen arbitrarily, a multi-agent generalization of the expert problem (fullinformation feedback) [Freund and Schapire, 1997] and the adversarial multi-armed bandit problem (bandit feedback) [Auer et al., 2002b]. There are no prior studies on this problem, be it with f = NSW or f = NSWprod, as far as we know.

4.1 Impossibility Results with Bandit Feedback

We start by considering the bandit feedback setting. As mentioned in Section 2, even though NSW is a concave function, our problem is not an instance of Bandit Convex Optimization, since we can only observe u t,i t , : instead of NSW ( u ⊤ t p t ) at the end of round t . Somewhat surprisingly, this slight difference in the feedback in fact makes a sharp separation in learnability - while O ( √ T ) regret is achievable in BCO, we prove that o ( T ) regret is impossible in our problem.

Before showing the theorem and its proof, we first give high level ideas on the construction of the hard environments. Specifically, we consider the environment with 2 agents, 2 arms, and binary utility matrix u t ∈ { 0 , 1 } 2 × 2 . Similar to the hard instance in the stochastic environment, we set u t, : , 2 = 1 , reducing the problem to a single-agent one. For the first agent, we let u t, : , 1 at each round t be i.i.d. drawn from a stationary distribution over the 4 binary utility vectors { (0 , 0) , (0 , 1) , (1 , 0) , (1 , 1) } . Then, we construct two different distributions, E and E ′ , over these 4 binary utility vectors satisfying that: 1) the distribution of the learner's observation is identical for E and E ′ ; 2) the optimal strategy for E and E ′ are significantly different. The first property guarantees that no algorithm can distinguish these two environments, while the second property ensures that there is no one single strategy that can perform well in both environments. Formally, we prove the following theorem.

Theorem 4.1. In the bandit feedback setting, for any algorithm, there exists an adversarial environment such that E [ Reg adv ] = Ω( T ) for f = NSW .

Proof. As sketched earlier, we consider two different environments with 2 agents, 2 arms, and binary utility matrices u t ∈ { 0 , 1 } 2 × 2 , t ∈ [ T ] . In both environments, we have u t, : , 2 = 1 . Next, we construct two different distributions from which u t, : , 1 is potentially drawn from, E and E ′ , over { (0 , 0) , (0 , 1) , (1 , 0) , (1 , 1) } . Specifically, E is characterized by ( q 00 , q 01 , q 10 , q 11 ) = ( 4 10 , 2 10 , 1 10 , 3 10 ) , where q xy is the probability of the vector ( x, y ) in E ; E ′ is characterized by ( q ′ 00 , q ′ 01 , q ′ 10 , q ′ 11 ) = ( 3 10 , 3 10 , 2 10 , 2 10 ) , where q ′ xy is the probability of vector ( x, y ) in E ′ . With a slight abuse of notation, we write u ∼ E for a matrix u ∈ { 0 , 1 } 2 × 2 if u : , 1 is drawn from E and u : , 2 = 1 ; the same for E ′ .

We argue that the learner's observations are equivalent in distribution in E and E ′ , since the marginal distributions of the utility of each action are the same. Specifically,

When action 1 is chosen, the distributions of the learner's observation in both E and E ′ are a Bernoulli random variable with mean q 10 + q 11 = q ′ 10 + q ′ 11 = 4 10 ;

√

When action 2 is chosen, the distributions of the learner's observation in both E and E ′ are a Bernoulli random variable with mean q 01 + q 11 = q ′ 01 + q ′ 11 = 5 10 .

Direct calculation shows p ⋆ = argmax p ∈ ∆ K E u ∼E [ NSW ( u ⊤ p ) ] = ( q 2 10 q 2 01 + q 2 10 , q 2 01 q 2 01 + q 2 10 ) = (0 . 2 , 0 . 8) and p ′ ⋆ = argmax p ∈ ∆ K E u ∼E ′ [ NSW ( u ⊤ p ) ] = ( q ′ 2 10 q ′ 2 10 + q ′ 2 01 , q ′ 2 01 q ′ 2 10 + q ′ 2 01 ) = ( 4 13 , 9 13 ) , which are constant apart from each other. Pick a threshold value θ = 33 130 ∈ (0 . 2 , 4 13 ) . Direct calculation shows that for a strategy p with p 1 ≥ θ , we have E u ∼E [ NSW ( u ⊤ p ⋆ ) -NSW ( u ⊤ p )] ≥ ∆ where ∆ = 1 500 ; similarly, for a strategy p with p 1 < θ , we have E u ∼E ′ [ NSW ( u ⊤ p ⋆ ) -NSW ( u ⊤ p )] ≥ ∆ as well. Now, given an algorithm, let α E be the probability that the number of rounds p t, 1 ≥ θ is larger than T 2 under environment E , and ¯ α E ′ be the probability of the complement of this event under environment E ′ . We have,

Finally, since the feedback for the algorithm is the same in distribution in these two environments, we know α E + ¯ α E ′ = 1 , and thus

which finishes the proof.

In fact, by a similar but more involved construction (that actually requires using two agents in a non-trivial way), the same impossibility result also holds for f = NSWprod; see Appendix B.1. We remark that non-linearity of f in these results plays an important role in the hard instance construction, since otherwise, the optimal strategy for E and E ′ will be the same as they both induce the same marginal distributions.

4.2 Full-Information Feedback

To sidestep the impossibility result due to the bandit feedback, we shift our focus to the fullinformation feedback model, where the learner observes the entirety of the utility matrix u t at the end of round t . As mentioned, this corresponds to a multi-agent generalization of the well-known expert problem [Freund and Schapire, 1997]. We propose several algorithms for this setting, showing that the richer feedback not only makes learning possible but also leads to much lower regret.

4.2.1 FTRL with Log-Barrier Regularizer

When f is concave, our problem is in fact also an instance of the well-known Online Convex Optimization (OCO) [Zinkevich, 2003]. However, standard OCO algorithms such as Online Gradient Descent, an instance of the more general Follow-the-Regularized-Leader algorithm with a ℓ 2 regularizer, require the utility function to also be Lipschitz and thus cannot be directly applied to learning NSW. Nevertheless, we will show that using a different regularizer that induces more stability than the ℓ 2 regularizer can resolve this issue.

More specifically, the FTRL algorithm is shown in Algorithm 2, which predicts at time t the distribution p t = argmin p ∈ ∆ K ⟨ p, -∑ t -1 s =1 ∇ f ( u ⊤ s p s ) ⟩ + 1 η ψ ( p ) for some learning rate η and some strongly convex regularizer ψ . Standard analysis shows that the regret of FTRL contains two terms: the regularization penalty term that is of order 1 /η and the stability term that is of order η ∑ t ∥∇ f ( u ⊤ t p t ) ∥ 2 ∇ -2 ψ ( p t ) where we use the notation ∥ a ∥ M = √ a ⊤ Ma . To deal with the lack the Lipschitzness, that is, the potentially large ∇ f ( u ⊤ t p t ) , we need to find a regularizer ψ so that the induced local norm ∥∇ f ( u ⊤ t p t ) ∥ ∇ -2 ψ ( p t ) is always reasonably small despite ∇ f ( u ⊤ t p t ) being large (in ℓ 2 norm for example).

It turns out that the log-barrier regularizer, ψ ( p ) = -∑ K i =1 log p i , ensures such a property. In fact, it induces a small local norm not just for NSW, but also for a broad family of SWFs as long as they are concave and Pareto optimal -anSWF f : [0 , 1] N → [0 , 1] is Pareto optimal if for two utility vectors x and y such that x n ≥ y n for all i ∈ [ N ] , we have f ( x ) ≥ f ( y ) . NSW is clearly in this family, and there are many other standard fairness measures that fall into this class; see Appendix B.2.1. For any SWF in this family, we prove the following regret bound, which remarkably has no dependence on the number of agents N at all.

Theorem 4.2. For any f : [0 , 1] N → [0 , 1] that is concave and Pareto optimal, Algorithm 2 with the log-barrier regularizer ψ ( p ) = -∑ K i =1 log p i and η = √ K log T T guarantees Reg adv = O ( √ KT log T ) .

Proof Sketch. Using the concrete form of ψ , it is clear that the local norm ∥∇ f ( u ⊤ t p t ) ∥ 2 ∇ -2 ψ ( p t ) simplifies to ∑ K i =1 p 2 t,i [ ∇ f ( u ⊤ t p t )] 2 i ≤ 〈 p t , ∇ f ( u ⊤ t p t ) 〉 2 , where the inequality is due to [ ∇ f ( u ⊤ t p t )] i ≥ 0 implied by Pareto optimality. Furthermore, by concavity, we have 〈 p t , ∇ f ( u ⊤ t p t ) 〉 ≤ f ( u ⊤ t p t ) -f (0) ≤ 1 , and thus the local norm at most 1 . The rest of the proof is by direct calculation.

4.2.2 FTRL with Tsallis Entropy Regularizer

In fact, when f = NSW, using the special structure of the welfare function, we find yet another regularizer that ensures a small O ( N ) local norm, with the benefit of having smaller dependence on K for the penalty term.

Theorem 4.3. For f = NSW , Algorithm 2 with the Tsallis entropy regularizer ψ ( p ) = 1 -∑ K i =1 p β i 1 -β , β = 2 N , and the optimal choice of η guarantees Reg adv = ˜ O ( K 1 2 -1 N √ NT ) .

The proof is more involved and is deferred to Appendix B.2.3. While the regret in Theorem 4.3 suffers polynomial dependence on N , it has better dependence on K compared to Theorem 4.2, and is thus more preferable when K is much larger than N .

4.2.3 Logarithmic Regret for a Special Case

Finally, we discuss a special case with f = NSW where logarithmic regret is possible. This is based on a simple observation that when there is one agent who is indifferent about the learner's choice (that is, the agent's utility is the same for all arms for this round), then -NSWis not only convex, but also exp-concave, a stronger curvature property. Therefore, by applying known results, specifically the EWOO algorithm [Hazan et al., 2007], we achieve the following result.

Theorem 4.4. Fix f = NSW . Suppose that for each time t , there is a set of agents A t ⊆ [ N ] such that | A t | ≥ M and u t, : ,n = c t,n 1 with c t,n ≥ 0 for each agent n ∈ A t . Then the EWOO algorithm guarantees Reg adv = O ( N -M M · K log T ) .

The proof, which verifies the exp-concavity of -NSW in this special case, can be found in Appendix B.2.4. We note that the reason that we apply EWOO instead of Online Newton Step, another algorithm discussed in [Hazan et al., 2007] for exp-concave losses, is that the latter requires Lipschizness (which, again, is not satisfied by NSW).

5 Conclusion

In this work, motivated by recent research on social welfare maximization for the problem of multiagent multi-armed bandits, we consider a variant with the arguably more natural version of Nash social welfare as the objective function, and develop multiple algorithms and regret upper/lower bounds in different settings (stochastic versus adversarial and full-information versus bandit feedback). Our results show a sharp separation between our problem and previous settings, including the heavily studied Bandit Convex Optimization problem.

There are many interesting future directions. First, in the stochastic bandit setting, we have only shown the tight dependence on T , so what about K and N ? Second, is there a more general strategy/analysis that works for different social welfare functions (similar to our result in Theorem 4.2)? Taking one step further, similar to the recent research on 'omniprediction' [Gopalan et al., 2022], is there one single algorithm that works for a class of social welfare functions simultaneously?

Acknowledgments and Disclosure of Funding

HL and MZ are supported by NSF Award IIS-1943607.

References

Jacob D Abernethy, Chansoo Lee, and Ambuj Tewari. Fighting bandits with a new kind of smoothness. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling a band of bandit algorithms. In Conference on Learning Theory , pages 12-38. PMLR, 2017.

Jean-Yves Audibert and Sébastien Bubeck. Regret bounds and minimax policies under partial monitoring. The Journal of Machine Learning Research , 11:2785-2836, 2010.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002a.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002b.

Jackie Baek and Vivek Farias. Fair exploration via axiomatic bargaining. Advances in Neural Information Processing Systems , 34:22034-22045, 2021.

Siddharth Barman, Sanath Kumar Krishnamurthy, and Rohit Vaish. Finding fair and efficient allocations. In Proceedings of the 2018 ACM Conference on Economics and Computation , pages 557-574, 2018.

Siddharth Barman, Arindam Khan, Arnab Maiti, and Ayush Sawarni. Fairness and welfare quantification for regret in multi-armed bandits. AAAI'23. AAAI Press, 2023. ISBN 978-1-57735-880-0. doi: 10.1609/aaai.v37i6.25829.

Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 19-26. JMLR Workshop and Conference Proceedings, 2011.

Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, and Shie Mannor. Multi-objective bandits: Optimizing the generalized gini index. In International Conference on Machine Learning , pages 625-634. PMLR, 2017.

Ioannis Caragiannis, David Kurokawa, Hervé Moulin, Ariel D. Procaccia, Nisarg Shah, and Junxing Wang. The unreasonable fairness of maximum nash welfare. ACM Trans. Econ. Comput. , 7(3), sep 2019. ISSN 2167-8375. doi: 10.1145/3355902.

Violet Xinying Chen and J.N. Hooker. A guide to formulating fairness in an optimization model. Annals of Operation Research , 326:581-619, 2023.

Yifang Chen, Alex Cuellar, Haipeng Luo, Jignesh Modi, Heramb Nemlekar, and Stefanos Nikolaidis. Fair contextual multi-armed bandits: Theory and experiments. In Conference on Uncertainty in Artificial Intelligence , pages 181-190. PMLR, 2020.

Richard Cole and Vasilis Gkatzelis. Approximating the nash social welfare with indivisible items. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 371-380, 2015.

Madalina M Drugan and Ann Nowe. Designing multi-objective multi-armed bandits algorithms: A study. In The 2013 international joint conference on neural networks (IJCNN) , pages 1-8. IEEE, 2013.

Dylan J Foster, Zhiyuan Li, Thodoris Lykouris, Karthik Sridharan, and Eva Tardos. Learning in games: Robustness of fast convergence. Advances in Neural Information Processing Systems , 29, 2016.

Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences , 55(1):119-139, 1997.

Jugal Garg, Pooja Kulkarni, and Rucha Kulkarni. Approximating nash social welfare under submodular valuations through (un) matchings. ACM Transactions on Algorithms , 19(4):1-25, 2023.

Stephen Gillen, Christopher Jung, Michael Kearns, and Aaron Roth. Online learning with an unknown fairness metric. Advances in neural information processing systems , 31, 2018.

Parikshit Gopalan, Adam Tauman Kalai, Omer Reingold, Vatsal Sharan, and Udi Wieder. Omnipredictors. In 13th Innovations in Theoretical Computer Science Conference (ITCS 2022) , volume 215, pages 79:1-79:21, 2022.

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2):169-192, 2007.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Safwan Hossain, Evi Micha, and Nisarg Shah. Fair algorithms for multi-agent multi-armed bandits. Advances in Neural Information Processing Systems , 34:24005-24017, 2021.

Matthew Jones, Huy Nguyen, and Thy Nguyen. An efficient algorithm for fair multi-agent multiarmed bandit with low regret. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 8159-8167, 2023.

Matthew Joseph, Michael Kearns, Jamie H Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

Mamoru Kaneko and Kenjiro Nakamura. The nash social welfare function. Econometrica , 47: 423-435, 1979.

Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Bandits and experts in metric spaces. Journal of the ACM (JACM) , 66(4):1-77, 2019.

Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Tor Lattimore. Bandit convex optimisation. arXiv preprint arXiv:2402.06535 , 2024.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Chung-Wei Lee, Haipeng Luo, Chen-Yu Wei, and Mengxiao Zhang. Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps. Advances in neural information processing systems , 33:15522-15533, 2020.

Wenzheng Li and Jan Vondrák. Estimating the nash social welfare for coverage and other submodular valuations. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 1119-1130. SIAM, 2021.

Yang Liu, Goran Radanovic, Christos Dimitrakakis, Debmalya Mandal, and David C Parkes. Calibrated fairness in bandits. arXiv preprint arXiv:1707.01875 , 2017.

Haipeng Luo. Lecture note 13, Introduction to Online Learning. 2017. URL https:// haipeng-luo.net/courses/CSCI699/lecture13.pdf .

Hervé Moulin. Fair division and collective welfare . MIT press, 2004.

John F Nash. The bargaining problem. Econometrica , 18(2):155-162, 1950.

Ayush Sawarni, Soumyabrata Pal, and Siddharth Barman. Nash regret guarantees for linear bandits. Advances in Neural Information Processing Systems , 36, 2024.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In Conference On Learning Theory , pages 1263-1291. PMLR, 2018.

Julian Zimmert and Yevgeny Seldin. An optimal algorithm for stochastic and adversarial bandits. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 467-475. PMLR, 2019.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03) , pages 928-936, 2003.

A Omitted Details in Section 3

In this section, we provide the omitted proofs for the results in Section 3. In Appendix A.1, we provide the proof for Theorem 3.1 and in Appendix A.2, we provide the proof for Theorem 3.2.

A.1 Proof of Theorem 3.1

To prove Theorem 3.1, we first consider the following two events.

Event 1. For all t ∈ { KN 0 +1 , . . . , T } and i ∈ [ K ] ,

where N t,i 's and p t,i 's are defined in Algorithm 1.

Event 2. For all t ∈ { KN 0 +1 , . . . , T } , i ∈ [ K ] , and n ∈ [ N ] ,

where N t,i 's are defined in Algorithm 1.

As we prove in Lemma A.1 and Lemma A.2, Event 1 and Event 2 hold with probability at least 1 -1 T . Now we prove Theorem 3.1. For convenience, we restate the theorem as follows.

Theorem 3.1. With N 0 = 1+18log KT , Algorithm 1 guarantees E [ Reg sto ] = ˜ O ( K 2 N T N -1 N + K ) .

Proof. Let p ⋆ = argmax p ∈ ∆ K NSW ( u ⊤ p ) . According to a standard regret decomposition for UCB-type algorithms, we know that Reg sto can be upper bounded as follows:

(based on the definition of p t )

In the following, we bound the first term

As discussed in Section 3.1, we consider two cases. First, consider the set of rounds T σ such that for all t ∈ T σ there exists at least one n ∈ [ N ] such that ⟨ p t , u : ,n ⟩ ≤ σ for some σ that we will specify later. Denote such n to be n t (if there are multiple such n 's, we pick an arbitrary one). According to Event 2, we know that for all i ∈ [ K ] ,

where the last inequality is because of AM-GM inequality. Therefore, we know that

Now consider ∑ t ∈T σ ( NSW ( ̂ u ⊤ t p t ) -NSW ( u ⊤ p t ) ) . Direct calculation shows that

Now consider the regret within t ∈ { NK 0 +1 , . . . , T }\T σ , in which we have ⟨ p t , u : ,n ⟩ ≥ σ for all n ∈ [ N ] . In this case, we bound ∑ t/ ∈T σ ( NSW ( ̂ u ⊤ t p t ) -NSW ( u ⊤ p t ) ) as follows:

where the last inequality is because Lemma A.3. Combining the regret for both parts, we know that

Picking the optimal σ leads to the expected regret bounded by E [ Reg sto ] ≤ ˜ O ( K 2 N T N -1 N + K ) .

Lemma A.1. Event 1 happens with probability at least 1 -1 T .

Proof. According to Algorithm 1, we know that N ( KN 0 +1) ,i = N 0 for each i ∈ [ K ] . Consider the case when t ≥ KN 0 +1 . According to Freedman's inequality (Lemma C.3), we have with probability at least 1 -δ , for a fixed t ≥ KN 0 +1 ,

Therefore, we know that with probability at least 1 -δ , for a fixed t ≥ KN 0 +1 ,

Picking δ = 1 KT 2 and taking a union bound over all i ∈ [ K ] and KN 0 + 1 ≤ t ≤ T reach the result.

Lemma A.2. Event 2 happens with probability at least 1 -1 T .

Proof. According to Freedman's inequality Lemma C.3, applying a union bound over t ∈ [ T ] , i ∈ [ K ] , and n ∈ [ N ] , we know that with probability at least 1 -δ , for all t ∈ [ T ] , i ∈ [ K ] , and n ∈ [ N ] ,

Solving the inequality with respect to u i,n , we know that

Using the above inequality and picking δ = 1 T , we can lower bound ̂ u t,i,n as follows:

where the last inequality uses Eq. (10). To upper bound ̂ u t,i,n , we have

Combining the lower and the upper bound finishes the proof.

Lemma A.3. Under Event 1, Algorithm 1 guarantees that

for all i ∈ [ K ] and KN 0 +1 ≤ t ≤ T .

Proof. Since Event 1 holds, we know that N t,i ≥ 1 2 ∑ t τ = KN 0 +1 p τ,i +1 holds for all i ∈ [ K ] and τ ≥ KN 0 + 1 based on the choice of N 0 = 18log KT + 1 . Therefore, we know that for each t ≥ KN 0 +1 ,

As for the term ∑ t τ = KN 0 +1 √ p τ,i N τ,i , we have

A.2 Omitted Details in Section 3.2

Theorem 3.2. In the bandit feedback setting, for any algorithm, there exists a stochastic environment in which the expected regret (with respect to NSW ) of this algorithm is Ω ( (log K ) 3 N 3 · K 1 N T N -1 N ) for N ≥ log K and sufficiently large T .

Proof. Consider the environment E that picks u uniformly from { u (1) , . . . , u ( K ) } , where u ( i ) : , 1 = ε · e i ∈ R K and u ( i ) : ,j = 1 for all j ∈ { 2 , 3 , . . . , N } . Here, ε ∈ (0 , 1 9 ] is some constant to be specified later. Denote u (0) to be the environment where u : ,n = 0 for all n ∈ [ N ] . At each round t , u t,i,n is an i.i.d. Bernoulli random variable with mean u i,n . For notational convenience, we use E i [ · ] when we take expectation over the environment u ( i ) for i ∈ { 0 } ∪ [ K ] . Let n i be the number of rounds that action i is selected over the total horizon T for all i ∈ [ K ] . Therefore, the expected regret with respect to environment E (a uniform distribution over u ( i ) , i ∈ [ K ] ) is lower bounded as follows:

where the last inequality is again due to Hölder's inequality. Let Ber ( α ) be the Bernoulli distribution with mean α . Combining Exercise 14.4 of [Lattimore and Szepesvári, 2020], Pinsker's inequality, and Lemma 15.1 of [Lattimore and Szepesvári, 2020], we have

where the last inequality is because ε ≤ 1 9 .

Taking summation over all i ∈ [ K ] , we obtain that

where the second inequality is due to Cauchy-Schwarz inequality.

Applying Eq. (13) to Eq. (11), we obtain that

E

E

[

Reg

]

where the third inequality is according to Lemma C.2 with x = 1 N and α = 1 K , meaning that N ( 1 -1 K 1 N ) ≥ log K 2 .

Picking ε = (log K ) 2 N · K (4 N ) 2 N T , we know that

Combining the above all together, we know that E E [ Reg ] ≥ Ω ( (log K ) 3 N 3 · K 1 N T N -1 N ) . Therefore, there exists one environment among u ( i ) , i ∈ [ K ] such that E i [ Reg ] ≥ Ω ( (log K ) 3 N 3 · K 1 N T N -1 N ) , which finishes the proof.

B Omitted Details in Section 4

B.1 Omitted Details in Section 4.1

In this section, we prove that that in the adversarial environment, it is also impossible to achieve sublinear regret when f = NSWprod. The hard instance construction shares a similar spirit to the one for f = NSW shown in Theorem 4.1.

Theorem B.1. In the bandit feedback setting, for any algorithm, there exists an adversarial environment such that E [ Reg adv ] = Ω( T ) for f = NSWprod .

Proof. We consider the learning environment with two agents and two arms. The agents utilities are binary, meaning that u ∈ { 0 , 1 } 2 × 2 . We construct two distributions E and E ′ with support { 0 , 1 } 2 × 2 . To define environment E , we use q wxyz for any w,x,y, z ∈ { 0 , 1 } to denote the probability that u 1 , : = ( w,x ) and u 2 , : = ( y, z ) when u ∼ E . For simplicity of notation, the binary number wxyz will be written in decimal form (i.e. q 8 = Pr u ∼E [ u 1 , : = (1 , 0) , u 2 , : = (0 , 0)] ). For environment E , we assign the probabilities

Similarly, for environment E ′ , we use q ′ wxyz for any w,x,z, y ∈ { 0 , 1 } to denote the probability that u ′ 1 , : = ( w,x ) and u ′ 2 , : = ( y, z ) when u ′ ∼ E ′ . Again, we will write the binary number wxyz in decimal form for ease of notation. To environment E ′ we assign probabilities

Next, we argue that the learner's observations are equivalent in distribution in E and E ′ , since the marginal distribution of every possible observation of each action is the same. Specifically,

When action 1 is played, the learner's observation u 1 , : in both E and E ′ are given by the following marginal distribution.

-The probability of observation (0 , 0) is q 0 + q 1 + q 2 + q 3 = q ′ 0 + q ′ 1 + q ′ 2 + q ′ 3 = 1 4

-The probability of observation (0 , 1) is q 4 + q 5 + q 6 + q 7 = q ′ 4 + q ′ 5 + q ′ 6 + q ′ 7 = 1 4

-The probability of observation (1 , 0) is q 8 + q 9 + q 10 + q 11 = q ′ 8 + q ′ 9 + q ′ 10 + q ′ 11 = 1 4

-The probability of observation (1 , 1) is q 12 + q 13 + q 14 + q 15 = q ′ 12 + q ′ 13 + q ′ 14 + q ′ 15 = 1 4

When action 2 is played, the learner's observation u 2 , : in both E and E ′ are given by the following marginal distribution.

-The probability of observation (0 , 0) is q 0 + q 4 + q 8 + q 12 = q ′ 0 + q ′ 4 + q ′ 8 + q ′ 12 = 1 4

-The probability of observation (0 , 1) is q 1 + q 5 + q 9 + q 13 = q ′ 1 + q ′ 5 + q ′ 9 + q ′ 13 = 1 4

-The probability of observation (1 , 0) is q 2 + q 6 + q 10 + q 14 = q ′ 2 + q ′ 6 + q ′ 10 + q ′ 14 = 1 4

-The probability of observation (1 , 1) is q 3 + q 7 + q 11 + q 15 = q ′ 3 + q ′ 7 + q ′ 11 + q ′ 15 = 1 4

Direct calculation shows that

Therefore, we compute the learner's best strategy for environments E and E ′ by direct calculation:

Next, consider a distribution p ∈ ∆ 2 such that p 1 ∈ [ 0 , 1 4 ] ∪ [ 3 4 , 1 ] . For such p , direct calculation shows that E u ∼E [ NSWprod ( u ⊤ p ⋆ ) -NSWprod ( u ⊤ p ) ] ≥ ∆ , where ∆ = 1 256 . On the other hand, for a strategy p ∈ ∆ 2 with p 1 ∈ ( 1 4 , 3 4 ) , we have E u ∼E ′ [ NSWprod ( u ⊤ p ′ ⋆ ) -NSWprod ( u ⊤ p ) ] > ∆ as well. Given any algorithm, let α E be the probability that the number of rounds p t, 1 ∈ [ 0 , 1 4 ] ∪ [ 3 4 , 1 ] is larger than T 2 under environment E . Let ¯ α E ′ be the probability of the complement of this event under environment E ′ . By definition

Since the feedback for the algorithm is the same in distribution, we have α E + ¯ α E ′ = 1 . Thus, we have

B.2 Omitted Details in Section 4.2

B.2.1 Concave and Pareto Optimal SWFs

Formally, for a function f : [0 , 1] N ↦→ [0 , 1] , concavity and Pareto Optimality are defined as:

Concavity: f ( αx +(1 -α ) y ) ≤ αf ( x ) + (1 -α ) f ( y ) for any α ∈ [0 , 1] and x, y ∈ [0 , 1] N .

Pareto optimality: for any x, y ∈ [0 , 1] N , x n ≥ y n for all n ∈ [ N ] implies f ( x ) ≥ f ( y ) .

Pareto optimality is a 'fundamental property' in social choice theory because it ensures that a SWF prefers alternatives that strictly more efficient: everyone is no worse off [Kaneko and Nakamura, 1979]. Concavity appears less in social choice literature. However, it promotes equity by modeling diminishing levels of desirability with the increase of a single agent's utility.

In the following, we provide examples of SWFs that satisfy concavity and Pareto optimality. Each of the following SWFs are parameterized by fixed weights w ∈ ∆ K .

Utilitarian SWF: f ( u ) = ⟨ w,u ⟩ ;

Generalized Gini Index (GGI): f ( u ) = min π ∈ S N ⟨ w π , u ⟩ , where S N is the set of permutations over N items and w π is weights w permuted according to π ∈ S N ;

Weighted NSW: f ( u ) = ∏ n ∈ N u w n n .

The last notable fact about the class of concave and Pareto Optimal SWFs is that it is closed under convex combinations. Specifically, for two concave and Pareto Optimal functions f, g : [0 , 1] N → [0 , 1] , the function h ( · ) = λf ( · ) + (1 -λ ) g ( · ) for any λ ∈ [0 , 1] is concave and Pareto Optimal. [Chen and Hooker, 2023] discusses how such convex combinations can be used to combine a SWF prioritizing efficiency and another prioritizing equity to derive a different SWF that prioritizes a balance between efficiency and equity.

B.2.2 Omitted Details in Section 4.2.1

In this section, we prove Theorem 4.2, which shows that O ( √ KT log T ) regret is achievable for all concave and Pareto optimal SWFs.

Theorem 4.2. For any f : [0 , 1] N → [0 , 1] that is concave and Pareto optimal, Algorithm 2 with the log-barrier regularizer ψ ( p ) = -∑ K i =1 log p i and η = √ K log T T guarantees Reg adv = O ( √ KT log T ) .

Proof. Using the concavity of f , we can upper bound Reg adv as follows:

To bound Term ( 1 ) , according to a standard analysis of FTRL/OMD with log-barrier regularizer (e.g. Lemma 12 of [Agarwal et al., 2017]), we know that:

where D ψ ( p, q ) ≜ ψ ( p ) -ψ ( q ) -⟨∇ ψ ( q ) , p -q ⟩ is the Bregman divergence between p and q with respect to ψ . To further bound the right-hand side, note that p 1 = 1 K · 1 . Direct calculation shows that for any p ∈ ∆ K, 1 KT ,

Using the Pareto optimality property of f and the positivity of the utility matrix u t , we know that [ ∇ f ( u ⊤ t p )] i ≥ 0 , meaning that ∑ K i =1 p 2 t,i [ ∇ f ( u ⊤ t p t ) ] 2 i ≤ 〈 p t , ∇ f ( u ⊤ t p t ) 〉 2 .

Moreover, using the concavity property of f , we know that 〈 p t , ∇ f ( u ⊤ t p t ) 〉 ≤ f ( u ⊤ t p t ) -f ( u ⊤ t 0 ) = f ( u ⊤ t p t ) ≤ 1 . Combining the above two inequalities means that

Combining Eq. (15) and Eq. (16), we can upper bound Term ( 1 ) as follows:

Denote the optimal distribution p ⋆ = argmax p ∈ ∆ K ∑ T t =1 f ( u ⊤ t p ) . Recall that p 1 = 1 K · 1 . Now we upper bound Term ( 2 ) as follows:

Combining Eq. (17) and Eq. (18), and choosing η = √ K log T T finishes the proof.

B.2.3 Omitted Details in Section 4.2.2

In this section, we present the omitted proof for Theorem 4.3, which shows a better dependency on K compared with Theorem 4.2.

Theorem 4.3. For f = NSW , Algorithm 2 with the Tsallis entropy regularizer ψ ( p ) = 1 -∑ K i =1 p β i 1 -β , β = 2 N , and the optimal choice of η guarantees Reg adv = ˜ O ( K 1 2 -1 N √ NT ) .

Proof. Consider the case when N ≥ 3 . Direct calculation shows that

Using the concavity of f and a standard analysis of FTRL with Tsallis entropy (e.g., [Luo, 2017, Theorem 1]), we know that

Picking β = 2 N , the first term can be upper bounded by 3 K 1 -2 N η and the second term can be upper bounded by ηT β = ηNT 2 . Further picking the optimal choice of η finishes the proof.

When N = 2 and β = 2 N = 1 , the regularizer ψ ( p ) = 1 -∑ K i =1 p β i 1 -β becomes the negative Shannon entropy ψ ( p ) = ∑ K i =1 p i log p i . Using the concavity of f and following a standard analysis of FTRL with Shannon entropy regularizer (e.g., [Hazan et al., 2016, Theorem 5.2]), we obtain that

B.2.4 Omitted Details in Section 4.2.3

In this section, we show the proof for Theorem 4.4, which shows that logarithmic regret is achievable when there is at least one agent who is indifferent about the learner's choice.

Theorem 4.4. Fix f = NSW . Suppose that for each time t , there is a set of agents A t ⊆ [ N ] such that | A t | ≥ M and u t, : ,n = c t,n 1 with c t,n ≥ 0 for each agent n ∈ A t . Then the EWOO algorithm guarantees Reg adv = O ( N -M M · K log T ) .

Proof. To show that EWOO algorithm achieves logarithmic regret, we need to show that f t ( p ) ≜ -NSW ( u ⊤ t p ) is α -exp-concave for some α > 0 for all t ∈ [ T ] , meaning that

Let A t ⊆ [ N ] be the set of agents with u t, : ,n = c t,n · 1 for all n ∈ A on round t ∈ [ T ] . It is guaranteed that | A t | ≥ M for all t ∈ [ T ] . Denote B t = [ N ] \ A t . Direct calculation shows that

For notational convenience, let λ t = Π m ∈ A t c 1 N t,m ≤ 1 . Picking α = N M -M , we know that

where the first inequality is because λ t ≤ 1 and f t ( p ) ≤ 0 ; the second inequality is because f t ( p ) ≥ -1 ; the third inequality is because | B t | ≤ 1 N -M . This shows that the f t ( p ) is M ( N -M ) -expconcave. Therefore, according to Theorem 4.4 of [Hazan et al., 2016], we know that the EWOO algorithm guarantees that

C Auxiliary Lemmas

In this section, we include several auxiliary lemmas that are useful in the analysis.

Lemma C.1. Let a 1 , . . . , a n , b 1 , . . . b n ∈ [0 , 1] , where a i ≥ b i for all i ∈ [ n ] . Then, ∏ n i =1 a i -∏ n i =1 b i ≤ ∑ n i =1 ( a i -b i ) .

Proof. Direct calculation shows that

Lemma C.2. For all x ∈ (0 , 1) and α ∈ (0 , 1) satisfying 1 + x log α ≥ 0 , we have 1 -α x x ≥ -log α 2 .

Proof. Let y = log α < 0 . We know that

which is true since e u ≤ 1 + u 2 for all u ∈ [ -1 , 0] and xy = x log α ≥ -1 .

Lemma C.3 (Theorem 1 in [Beygelzimer et al., 2011]) . Let X 1 , . . . , X T ∈ [ -B,B ] for some B > 0 be a martingale difference sequence and with ∑ T t =1 E t [ X 2 t ] ≤ V for some fixed quantity V > 0 . We have for all δ ∈ (0 , 1) , with probability at least 1 -δ ,

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See abstract and Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 3, Section 4, and the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics. The research conducted in this paper conforms with it in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is mostly theoretical, and we do not foresee any negative ethical or societal outcomes.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a new fairness measure, say NSW, lacking Lipschitzness. Unlike previous measures, the multi-armed bandit problem cannot obtain common $O(\\sqrt T)$ regret. For stochastic MAB, the authors find an algorithm achieving $\\tilde{O}(T^{\\frac{N-1}{N}})$ regret and show its tightness. For adversary scenarios, the authors show that no algorithm can achieve sublinear regret. Then, they consider an easier information structure, say full-information feedback and $\\sqrt T$ regret is possible. Finally, they consider the situation when logarithmic regret is achievable.\n\nSTRENGTHS:\nThe model is clear and the algorithms proposed are easy to understand. Besides, the authors consider different information structures and models to depict the problem completely. And the construction of lower bounds is very nice.\n\nWEAKNESSES:\n1. The paper shows why considering $NSW$ is better than considering $NSW_{prod}$. However, why not consider simple average which is more intuitive? I recommend that the authors provide some realistic applications for NSW.\n\n2. If I understand correctly, you consider an addictive utility, for example, you consider expected regret when showing lower bounds. However, for (2), why do you use $NSW(u^Tp_t)$ rather than $\\sum_i p_{t,i}NSW(u_{t,i})$? The definition of regret seems strange.\n\n3. The statement in Line 165 is not correct. We don't need to observe $u^T_t p_t$ when using BCO though we may need the dimension of $u$ to be 1, i.e. $N=1$. When you prove the lower bounds, I think the negative result, i.e., Thm 3.2, holds because NSW is concave but https://arxiv.org/pdf/2402.06535 needs convexity. However, in Appendix B. 1., you have convexity, so you can only construct a hard-to-learn example with at least two agents. I'm not sure whether my intuition is correct. If it is, some statement in the paper doesn't hold. If not, please explain more about the construction of your lower bounds to make it easy to understand.\n\n4. You mention that NSW is not Lipschitz, but why not make a truncation? If an agent has a very small utility, like $\\sigma$ in the paper, the regret will be small. Otherwise, we can use Lipschitzness. Is there any difference between your methods? If not, show the reason why NSW doesn't make the problem harder. Also, if you have time, it's meaningful to use experiments to compare these two ideas.\n\nQUESTIONS:\n1. What will happen if the learning can only observe NSW rather than everyone's utility? Will it change the dependence of $K$ and $N$?\n\n2. In Line 324, what is $f(p_t)$? I believe the input of $f$ is an $R^N$ vector. However, $p_t$ belongs to $R^K$.\n\n3. Typo: In Line 338, \"is not only convex\" should be \"concave\".\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of maximizing Nash Social Welfare (NSW) in a centralized multi-agent multi-armed bandit setting. In each round, a central entity selects a probability distribution $p_t$ over the $K$ actions, and draw an action $i_t  \\sim p_t$.  Upon selecting an action, the central entity observes the $i$-th row of the $K\\times N$ utility matrix $u_t$ which indicates the utility received by each of the $N$ agent when choosing one of the $K$ actions.The goal of the central entity is to maximize the sum of the Nash Social Welfares over the rounds, where the NSW for a given round is defined as the geometric mean of the expected utilities received by the agents: $$NSW(p_t, u_t) = \\Pi_{j = 1}^{N} \\langle p_t \\vert u_t\\rangle^{1/N}.$$\n\nThe authors note that, unlike the product of expected utilities, the Nash Social Welfare (NSW) is not Lipschitz continuous. Consequently, the analytical techniques used in previous works that study the product of expected utilities are not applicable here. In the scenario where the utility matrix $u_t$is drawn i.i.d. from some unknown distribution, the authors develop an algorithm for this problem and establish both upper and lower bounds that match in terms of scaling with the horizon $T$. Next, they consider the adversarial framework. They demonstrate that with bandit feedback, the worst-case regret must scale linearly. They propose two different algorithms for the adversarial case with full information and show that both achieve a regret of $\\tilde{O(\\sqrt{T}}$, with different dependence in $N$ and $K$.  Finally, they show that in some special cases, the regret can be logarithmic in $T$.\n\nSTRENGTHS:\nI am not very familiar with the literature on Nash Social Welfare; however this paper seem to address a well-motivated problem in multi-agent decision-making.\n\nThe authors propose a UCB algorithm for maximizing the sum of Nash Social Welfares (NSW), utilizing new confidence bounds to achieve optimal regret rates. They establish a matching lower bound on the regret, providing theoretical robustness to their approach. Additionally, they explore the adversarial case, a scenario that had not been previously addressed. Their results reveal that bandit feedback alone is insufficient to achieve sub-linear regret rates. To complete their investigation, they propose a Follow-the-Regularized-Leader (FTRL) algorithm. While FTRL is not a novel algorithm, its application here concludes a comprehensive analysis of the problem at hand.\n\nThe paper is very clear, with the key ideas of the proofs well outlined and the problem effectively contrasted with existing works.\n\nWEAKNESSES:\nI see no major weaknesses. As a minor remark, the use of the Nash Social Welfare is not motivated. While it is a standard welfare measure in economics, it may be unfamiliar to those in the bandit literature. Therefore, it is worth introducing and discussing its significance and relevance.\n\nQUESTIONS:\n- Could you please quickly comment on why the Nash Social Welfare is defined as the geometric average of the expected utility, instead of the expectation of the geometric averages of the utilities? Admittedly, the latter might be less interesting from a mathematical standpoint. Is this a choice you are making, or is it a commonly accepted definition? \n- Could you please detail a bit more how you obtain the thrid line from the second line of the computations at the end of page 14?\n- There is a typo at the last line of page 4, where a \"O\" is capitalized in the middle of a sentence.\n\nLIMITATIONS:\nThe limitations are well addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of online social welfare maximization. More precisely, the authors consider the online learning setting where the learner, at each round $t \\in [T]$, picks an action $i \\in [K]$ that then determines the utility of each of the $n$ agents. The utility of agent $j$ is given by the $(i,j)$ entry of a utility matrix $u_t \\in \\mathbb{R}^{K \\times n}$, which can either change arbitrarily or be sampled from a predetermined probability distribution. The cost of the learner is given by the Nash Welfare, defined as the geometric mean of the agents' utilities: $NW(\\mu) = (\\Pi_{n \\in [n] \\mu_n})^{1/n}$. The goal of the learner is to pick a sequence of mixed actions to achieve social welfare comparable to that achieved by the best fixed mixed actions.\n\nThe authors consider both the stochastic and the deterministic versions of the problem, as well as both the bandit and full-information feedback models. For the stochastic case with bandit feedback (where the learner learns only the utilities of the agents for the randomly sampled action), the authors provide an $O(T^{N-1/N})$-regret online learning algorithm and show that this regret bound is tight for the stochastic setting. They then study the bandit feedback model with adversarial changes and establish an $\\Omega(T)$ regret lower bound. In view of this negative result, the authors shift their attention to the full-information feedback and adversarial changes case, for which they provide $O(\\sqrt{T})$-regret online learning algorithms using Follow the Regularized Leader with log-barrier regularization.\n\nSTRENGTHS:\nThe paper studies an interesting and technically challenging setting. The writing is relatively good, and the authors clearly explain their contributions and the crucial differences between their setting and classical online concave optimization with bandit feedback. I found it particularly interesting that there is a significant discrepancy in the possible regret bounds between online concave optimization and the bandit feedback case of the considered setting. Additionally, I appreciate that the authors present both upper and lower bounds for all the considered settings.\n\nWEAKNESSES:\nDespite the paper's solid technical contribution, my only concern lies with the motivation for the setting. The authors briefly mention that the setting has applications in resource allocation but do not provide any concrete examples or a convincing discussion on why this setting is particularly interesting. While I do not doubt that the setting is indeed interesting, I believe a detailed discussion on the potential applications of the model would significantly enhance the paper.\n\nQUESTIONS:\nCould you elaborate more on the potential applications of the considered setting?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}