Title: Nash Regret Guarantees for Linear Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: Sub-Poisson Distribution

Nash Social Welfare

Fairness Quantification

John Ellipsoid

Kiefer-Wolfowitz Optimal Design

Algorithmic Game Theory

Online Learning
Abstract: We obtain essentially tight upper bounds for a strengthened notion of regret in the stochastic linear bandits framework. The strengthening---referred to as Nash regret---is defined as the difference between the (a priori unknown) optimum and the geometric mean of expected rewards accumulated by the linear bandit algorithm. Since the geometric mean corresponds to the well-studied Nash social welfare (NSW) function, this formulation quantifies the performance of a bandit algorithm as the collective welfare it generates across rounds. NSW is known to satisfy fairness axioms and, hence, an upper bound on Nash regret provides a principled fairness guarantee.    

We consider the stochastic linear bandits problem over a horizon of $\mathsf{T}$ rounds and with a set of arms ${\cal X}$ in ambient dimension $d$. Furthermore, we focus on settings in which the stochastic reward---associated with each arm in ${\cal X}$---is a non-negative, sub-Poisson random variable. For this setting, we develop an algorithm that achieves a Nash regret of $O\left( \sqrt{\frac{d}{\mathsf{T}}} \log(\mathsf{T} |{\cal X}|)\right)$. In addition, addressing linear bandit instances in which the set of arms ${\cal X}$ is not necessarily finite, we obtain a Nash regret upper bound of $O\left( \frac{d^\frac{5}{4}}{\sqrt{\mathsf{T}}}  \log(\mathsf{T})\right)$. Since bounded random variables are sub-Poisson, these results hold for bounded, non-negative rewards. Our linear bandit algorithm is built upon the successive elimination method with novel technical insights, including tailored concentration bounds and the use of sampling via John ellipsoid in conjunction with the Kiefer–Wolfowitz optimal design.

Nash Regret Guarantees for Linear Bandits

Abstract

We obtain essentially tight upper bounds for a strengthened notion of regret in the stochastic linear bandits framework. The strengthening-referred to as Nash regret-is defined as the difference between the (a priori unknown) optimum and the geometric mean of expected rewards accumulated by the linear bandit algorithm. Since the geometric mean corresponds to the well-studied Nash social welfare (NSW) function, this formulation quantifies the performance of a bandit algorithm as the collective welfare it generates across rounds. NSW is known to satisfy fairness axioms and, hence, an upper bound on Nash regret provides a principled fairness guarantee.

We consider the stochastic linear bandits problem over a horizon of T rounds and with set of arms X in ambient dimension d . Furthermore, we focus on settings in which the stochastic reward-associated with each arm in X -is a non-negative, ν -sub-Poisson random variable. For this setting, we develop an algorithm that achieves a Nash regret of O (√ dν T log( T |X| ) ) . In addition, addressing linear bandit instances in which the set of arms X is not necessarily finite, we obtain a Nash regret upper bound of O ( d 5 4 ν 1 2 √ T log( T ) ) . Since bounded random variables are sub-Poisson, these results hold for bounded, positive rewards. Our linear bandit algorithm is built upon the successive elimination method with novel technical insights, including tailored concentration bounds and the use of sampling via John ellipsoid in conjunction with the Kiefer-Wolfowitz optimal design.

1 Introduction

Bandit optimization is a prominent framework for sequential decision making and has several applications across multiple domains, such as healthcare [26, 23, 27] and advertising [22]. In this framework, we have a set of arms (possible actions) with unknown means and a time horizon. The goal is to sequentially pull the arms such that the regret-which is a notion of loss defined over the bandit instance-is minimized.

We consider settings wherein the stochastic rewards generated by a sequential algorithm induces welfare across a population of agents. Specifically, there are T agents, arriving one per round; in particular, the reward accrued at each round t ∈ [ T ] corresponds to the value accrued by the t th agent. Indeed, such a welfarist connection exists in various applications of the bandit framework. Consider, for instance, the classic context of drug trials [24]: Suppose there are T patients and several available drugs. In each round t ∈ [ T ] , one of the available drugs is administered to the t th patient. Subsequently, the reward accrued at the t th round corresponds to the efficacy of the administered drug to the t th patient. In such a setting, fairness is a fundamental consideration. That is, in addition to cumulative efficacy, individual effectiveness of the drugs is quite important.

A central notion in the bandit literature is that of average regret, defined as the difference between the (a priori unknown) optimum and the arithmetic mean of the expected rewards (accumulated by the algorithm) [4]. However, average regret fails to capture the fairness criterion that the rewards should be balanced (across the agents) and not just cumulatively high. From a welfarist viewpoint, the standard notion of (average) regret equates the algorithm's performance to the social welfare it induces. Social welfare is defined as the sum of agent's rewards [19] and can be high among a set of agents even if a fraction of them receive indiscriminately low rewards. For instance, in the drug-trials example provided above, high average efficacy (i.e., high social welfare) does not rule out a severely ineffective outcome for a subset of agents.

Given that average regret is defined using the sum of expected rewards, this notion inherits this utilitarian limitation of social welfare. In summary, in welfare-inducing contexts, a bandit algorithm with low average regret is not guaranteed to induce fair outcomes across rounds.

Addressing this issue and with the overarching aim of achieving fairness across rounds (i.e., across agents that receive rewards from a bandit algorithm), the current work considers a strengthened notion of regret. The strengthening-referred to as Nash regret-is defined as the difference between the (a priori unknown) optimum and the geometric mean of expected rewards induced by the bandit algorithm. It is relevant to note that the geometric mean (of rewards) corresponds to the Nash social welfare (NSW) function [19]. This welfare function has been extensively studied in mathematical economics (see, e.g., [19]) and is known to satisfy fundamental fairness axioms, including the PigouDalton transfer principle, scale invariance, and independence of unconcerned agents. Hence, by definition, Nash regret quantifies the performance of a bandit algorithm as the NSW it generates.

Quantitatively speaking, in order for the geometric mean (i.e., the NSW) to be large, the expected reward at every round should be large enough. The AM-GM inequality also highlights that Nash regret is a more demanding objective that average regret.

We obtain novel results for Nash regret in the stochastic linear bandits framework. In this well-studied bandit setup each arm corresponds to a d -dimensional vector x (an arm-specific context) and the unknown arm means are modelled to be a linear function of x . With a focus on average regret, stochastic linear bandits have been extensively studied in the past decade [1, 9, 21]. The current paper extends the line of work on linear bandits with fairness and welfare considerations.

Note that an ostensible approach for minimizing Nash regret is to take the logarithm of the observed rewards and, then, solve the average regret problem. However, this approach has the following shortcomings: (i) Taking log implies the modified rewards can have a very large range possibly making the regret vacuous, and (ii) This approach leads to a multiplicative guarantee and not an additive one. In a recent work of [5], the authors study Nash regret in the context of stochastic multi-armed bandits (with bounded rewards) and provide optimal guarantees. The current work notably generalizes this prior work to linear bandits.

1.1 Our Contributions and Techniques

We consider the stochastic linear bandits setting with a set of arms X over a finite horizon of T rounds. Since we consider the welfarist viewpoint, we assume that the rewards across all the rounds are positive and, in particular, model the distribution of the arm rewards to be ν -sub-Poisson, for parameter ν ∈ R + . Our goal is to minimize the Nash regret NR T .

We develop a novel algorithm LINNASH that obtains essentially optimal Nash regret guarantees for this setting. Specifically, for a finite set of arms X ⊂ R d , our algorithm LINNASH achieves Nash

(√ )

regret NR T = O dν T log( T |X| ) . For infinite sets of arms, a modified version of LINNASH

Recall that Nash regret is a strengthening of the average regret; the AM-GM inequality implies that, for any bandit algorithm, the Nash regret is at least as much as its average regret. Hence, in the linear bandits context, the known Ω ( d/ √ T ) lower bound on average regret (see [17], Chapter 24) holds

for Nash regret as well. 1 This observation implies that, up to a logarithmic factor, our upper bound on Nash regret is tight with respect to the number of rounds T . We also note that for instances in which the number of arms |X| = ω (2 d ) , the Nash-regret dependence on d has a slight gap. Tightening this gap is an interesting direction of future work.

We note that bounded, positive random variables are sub-Poisson (Lemma 1). Hence, our results hold for linear bandit instances wherein the stochastic rewards are bounded and positive. This observation also highlights the fact that the current work is a generalization of the result obtained in [5]. In addition, notice that, by definition, Poisson distributions are 1 -sub-Poisson. Hence, our guarantees further hold of rewards that are not necessarily sub-Gaussian. Given the recent interest in obtaining regret guarantees beyond sub-Gaussian rewards [18, 3], our study of sub-Poisson rewards is interesting in its own right. 2

Our linear bandit algorithm, LINNASH, has two parts. In the first part, we develop a novel approach of sampling arms such that in expectation the reward obtained is a linear function of the center of John Ellipsoid [13]. Such a strategy ensures that the expected reward in any round of the first part is sufficiently large. The second part of LINNASH runs in phases of exponentially increasing length. In each phase, we sample arms according to a distribution that is obtained as a solution of a concave optimization problem, known as D-optimal design. We construct confidence intervals at each phase and eliminate sub-optimal arms. A key novelty in our algorithm and analysis is the use of confidence widths that are estimate dependent. We define these widths considering multiplicative forms of concentration bounds and crucially utilize the sub-Poisson property of the rewards. The tail bounds we develop might be of independent interest.

1.2 Other Related Work

There has been a recent surge in interest to achieve fairness guarantees in the context of multi-armed bandits; see, e.g., [14, 7, 20, 6, 12]. However, these works mostly consider fairness across arms and, in particular, impose fairness constraints that require each arm to be pulled a pre-specified fraction of times. By contrast, our work considers fairness across rounds.

Alternative Regret Formulations. In the current work, for the welfare computation, each agent t 's value is considered as the expected reward in round t . One can formulate stronger notions of regret by, say, considering the expectation of the geometric mean of the rewards, rather than the geometric mean of the expectations. However, as discussed in [5], it is not possible to obtain non-trivial guarantees for such reformulations in general: every arm must be pulled at least once. Hence, if one considers the realized rewards (and not their expectations), even a single pull of a zero-reward arm will render the geometric mean zero.

2 Problem Formulation and Preliminaries

We will write [ m ] to denote the set { 1 , 2 , . . . , m } . For a matrix X , let Det ( X ) to denote the determinant of X . For any discrete probability distribution λ with sample space Ω , write Supp ( λ ) ≜ { x ∈ Ω : Pr X ∼ λ { X = x } > 0 } to denote the points for which the probability mass assigned by λ is positive. For a vector a ∈ R d and a positive definite matrix V ∈ R d × d , we will denote || a || V := √ a T V a . Finally, let B := { x ∈ R d | || x || 2 = 1 } be the d -dimensional unit ball.

We address the problem of stochastic linear bandits with a time horizon of T ∈ Z + rounds. Here, an online algorithm (decision maker) is given a set of arms X ⊂ R d . Each arm corresponds to a d -dimensional vector. Furthermore, associated with each arm x ∈ X , we have a stochastic reward r x ∈ R + . In the linear bandits framework, the expected value of the reward r x is modeled to be a linear function of x ∈ R d . In particular, there exists an unknown parameter vector θ ∗ ∈ R d such that, for each x ∈ X , the associated reward's expected value E [ r x ] = ⟨ x, θ ∗ ⟩ . Given the focus on welfare contexts, we will, throughout, assume that the rewards are positive, r x > 0 , for all x ∈ X .

The online algorithm (possibly randomized) must sequentially select an arm X t in each round t ∈ [ T ] and, then, it observes the corresponding (stochastic) reward r X t > 0 . 3 For notational convenience,

we will write r t to denote r X t . In particular, if in round t the selected arm X t = x , then the expected reward is ⟨ x, θ ∗ ⟩ , i.e., E [ r t | X t = x ] = ⟨ x, θ ∗ ⟩ . We will, throughout, use x ∗ to denote the optimal arm, x ∗ = arg max x ∈X ⟨ x, θ ∗ ⟩ and ̂ θ to denote estimator of θ ∗ .

In the stochastic linear bandits framework, our overarching objective is to minimize the Nash regret, defined as follows:

Note that the definition of Nash regret is obtained by applying the Nash social welfare (geometric mean) onto ex ante rewards, E [ ⟨ X t , θ ∗ ⟩ ] , 4 accrued across the T rounds.

2.1 Sub-Poisson Rewards

In order to model the environment with positive rewards ( r x > 0 ), we assume that the rewards r x associated with the arms x ∈ X are ν -sub Poisson , for some parameter ν > 0 . Formally, their moment-generating function satisfies the following bound

Note that a Poisson random variable is 1 -sub Poisson. To highlight the generality of ν -sub-Poisson distributions, we note that bounded, non-negative random variables are sub-Poisson (Lemma 1). Further, in Lemma 2, we establish a connection between non-negative sub-Gaussian and sub-Poisson random variables.

Lemma 1. Any non-negative random variable X ∈ [0 , B ] is B -sub-Poisson, i.e., if mean E [ X ] = µ , then for all λ ∈ R , we have E [ e λX ] ≤ exp ( B -1 µ ( e Bλ -1 )) .

Lemma 2. Let X be a non-negative sub-Gaussian random variable X with mean µ = E [ X ] and sub-Gaussian norm σ . Then, X is also ( σ 2 µ ) -sub-Poisson.

The proofs of Lemmas 1 and 2 appear in Appendix A. Lemma 2 has useful instantiations. In particular, the lemma implies that the half-normal random variable, with variance of σ , is also a ( Cσ ) -sub-Poisson, where C is a constant (independent of distribution parameters). Similarly, for other well-studied, positive sub-Gaussian random variables (including truncated and folded normal distributions), the sub-Poisson parameter is small.

Next, we discuss the necessary preliminaries for our algorithm and analysis.

2.2 Optimal Design.

Write ∆( X ) to denote the probability simplex associated with the set of arms X . Let λ ∈ ∆( X ) be such a probability distribution over the arms, with λ x denoting the probability of selecting arm x . The following optimization problem, defined over the set of arms X , is well-known and is referred to as the G-optimal design problem.

The solution to (3) provides the optimal sequence of arm pulls (for a given budget of rounds) to minimize the confidence width of the estimated rewards for all arms x ∈ X . The G-optimal design problem connects to the following optimization problem (known as D-optimal design problem):

The lemma below provides an important result of Kiefer and Wolfowitz [15].

Lemma 3 (Kiefer-Wolfowitz) . If the set X is compact and X spans R d , then there exists λ ∗ ∈ ∆( X ) supported over at most d ( d +1) / 2 arms such that λ ∗ minimizes the objective in equation (3) with g ( λ ∗ ) = d . Furthermore, λ ∗ is also a maximizer of the D-optimal design objective, i.e., λ ∗ maximizes the function f ( λ ) = log Det ( U ( λ )) subject to λ ∈ ∆( X ) .

At several places in our algorithm, our goal is to find a probability distribution that minimizes the non-convex optimization problem (3). However, instead we will maximize the concave function f ( λ ) = log Det ( U ( λ )) over λ ∈ ∆( X ) . The Frank-Wolfe algorithm, for instance, can be used to solve the D-optimal design problem (4) and compute λ ∗ efficiently ([17], Chapter 21). Lemma 3 ensures that this approach works, since the G-optimal and the D-optimal design problems have the same optimal solution λ ∗ ∈ ∆( X ) , which satisfies Supp ( λ ∗ ) ≤ d ( d +1) / 2 . 5

2.3 John Ellipsoid.

For any convex body K ⊂ R d , a John ellipsoid is an ellipsoid with maximal volume that can be inscribed within K . It is known that K itself is contained within the John Ellipsoid dilated by a factor of d . Formally, 6

Lemma 4 ([11]) . Let K ⊂ R d be a convex body (i.e., a compact, convex set with a nonempty interior). Then, there exists an ellipsoid E (called the John ellipsoid) that satisfies E ⊆ K ⊆ c + d ( E -c ) . Here, c ∈ R d denotes the center of E and c + d ( E -c ) refers to the (dialated) set { c + d ( x -c ) : x ∈ E } .

3 Our Algorithm LINNASH and Main Results

In this section, we detail our algorithm LINNASH (Algorithm 2), and establish an upper bound on the Nash regret achieved by this algorithm. Subsection 3.1 details Part I of LINNASH and related analysis. Then, Subsection 3.2 presents and analyzes Part II of the algorithm. Using the lemmas from these two subsections, the regret bound for the algorithm is established in Subsection 3.3.

3.1 Part I : Sampling via John Ellipsoid and Kiefer-Wolfowitz Optimal Design

As mentioned previously, Nash regret is a more challenging objective than average regret: if in any round t ∈ [ T ] , the expected 7 reward E [ r t ] is zero (or very close to zero), then geometric mean ( ∏ T t =1 E [ r X t ]) 1 / T goes to zero, even if the expected rewards in the remaining rounds are large. Hence, we need to ensure that in every round t ∈ [ T ] , specifically the rounds in the beginning of the algorithm, the expected rewards are bounded from below. In [5], this problem was tackled for stochastic multi-armed bandits (MAB) by directly sampling each arm uniformly at random in the initial rounds. Such a sampling ensured that, in each of those initial rounds, the expected reward is bounded from below by the average of the expected rewards. While such a uniform sampling

Algorithm 2 LINNASH (Nash Regret Algorithm for Finite Set of Arms)

Input: Arm set X and horizon of play T .

1: Initialize matrix V ← [0] d,d and number of rounds ˜ T = 3 √ T dν log( T |X| ) . Part I

2: Generate arm sequence S for the first ˜ T rounds using Algorithm 1.

3: for t = 1 to ˜ T do

6: Set estimate ̂ θ := V -1 ( ∑ ˜ T t =1 r t X t )

4: Pull the next arm X t from the sequence S , observe corresponding reward r t , and update V ← V + X t X T t 5: end for

7: Compute confidence bounds LNCB( x, ̂ θ, ˜ T / 3) and UNCB( x, ̂ θ, ˜ T / 3) , for all x ∈ X (see equation (7))

8: Set ˜ X = { x ∈ X : UNCB( x, ̂ θ, ˜ T / 3) ≥ max z ∈X LNCB( z, ̂ θ, ˜ T / 3) } and initialize T ′ = 2 3 ˜ T Part II

9: while end of time horizon T is reached do

10: Initialize V = [0] d,d to be an all zeros d × d matrix and s = [0] d to be an all-zeros vector. // Beginning of new phase.

11: Find the probability distribution λ ∈ ∆( ˜ X ) by maximizing the following objective

12: for each arm a in Supp ( λ ) do

13: Pull arm a for the next ⌈ λ a T ′ ⌉ rounds. Update V ← V + ⌈ λ a T ′ ⌉ · aa T .

14: Observe ⌈ λ a T ′ ⌉ corresponding rewards z 1 , z 2 , . . . and update s ← s +( ∑ j z j ) a .

15: end for

16: Set estimate ̂ θ = V -1 s and compute LNCB( x, ̂ θ, T ′ ) and UNCB( x, ̂ θ, T ′ ) , for all x ∈ X (see equation (7))

{ }

18: Update T ← 2 T .

17: Set ˜ X = x ∈ ˜ X : UNCB( x, ̂ θ, T ′ ) ≥ max z ∈X LNCB( z, ̂ θ, T ′ ) . // End of phase. ′ ′

19: end while

strategy is reasonable for the MAB setting, it can be quite unsatisfactory in the current context of linear bandits. To see this, consider a linear bandit instance in which, all-except for one-arms in X are orthogonal to θ ∗ . Here, a uniform sampling strategy will lead to an expected reward of ⟨ x ∗ , θ ∗ ⟩ / |X| , which can be arbitrarily small for large cardinality X .

To resolve this issue we propose a novel approach in the initial ˜ T := 3 √ T dν log( T |X| ) rounds. In particular, we consider the convex hull of the set of arms X -denoted as cvh( X ) -and find the center c ∈ R d of the John ellipsoid E for the convex hull cvh( X ) . Since E ⊆ cvh( X ) , the center c of the John ellipsoid is contained within cvh( X ) as well. Furthermore, via Carathéodory's theorem [10], we can conclude that the center c can be expressed as a convex combination of at most ( d +1) points in X . Specifically, there exists a size-( d +1) subset Y := { y 1 , . . . , y d +1 } ⊆ X and convex coefficients α 1 , . . . , α d +1 ∈ [0 , 1] such that c = ∑ d +1 i =1 α i y i with ∑ d +1 i =1 α i = 1 . Therefore, the convex coefficients induce a distribution U ∈ ∆( X ) of support size d +1 and with E x ∼ U [ x ] = c .

Lemma 5 below asserts that sampling according to the distribution U leads to an expected reward that is sufficiently large. Hence, U is used in the subroutine GenerateArmSequence (Algorithm 1).

In particular, the purpose of the subroutine is to carefully construct a sequence (multiset) of arms S , with size |S| = ˜ T and to be pulled in the initial ˜ T rounds. The sequence S is constructed such that (i) upon pulling arms from S , we have a sufficiently large expected reward in each pull, and (ii) we obtain an initial estimate of the inner product of the unknown parameter vector θ ∗ with all arms in X . Here, objective (i) is achieved by considering the above-mentioned distribution U . Now, towards the objective (ii), we compute distribution λ ∈ ∆( X ) by solving the optimization problem (also known as the D-optimal design problem) stated in equation (5).

We initialize sequence S = ∅ and run the subroutine GenerateArmSequence for ˜ T iterations. In each iteration (of the for-loop in Line 4), with probability 1 / 2 , we sample an arm according to the distribution U (Line 7) and include it in S . Also, in each iteration, with remaining probability 1 / 2 , we consider the computed distribution λ and, in particular, pick arms z from the support of λ in a round-robin manner. We include such arms z in S while ensuring that, at the end of the subroutine,

each such arm z ∈ Supp ( λ ) is included at least ⌈ λ z ˜ T / 3 ⌉ times. We return the curated sequence of arms S at the end of the subroutine.

Our main algorithm LINNASH (Algorithm 2) first calls subroutine GenerateArmSequence to generated the sequence S . Then, the algorithm LINNASH sequentially pulls the arms X t from S , for 1 ≤ t ≤ ˜ T rounds. For these initial ˜ T = |S| rounds, let r t denote the noisy, observed rewards. Using these ˜ T observed rewards, the algorithm computes the ordinary least squares (OLS) estimate ̂ θ (see Line 6 in Algorithm 2); in particular, ̂ θ := ( ∑ ˜ T t =1 X t X T t ) -1 ( ∑ ˜ T t =1 r t X t ) . The algorithm uses the OLS estimate ̂ θ to eliminate several low rewarding arms (in Lines 7 and 8 in Algorithm 2). This concludes Part I of the algorithm LINNASH.

Before detailing Part II (in Subsection 3.2), we provide a lemma to be used in the analysis of Part I of LINNASH.

Lemma 5. Let c ∈ R d denote the center of a John ellipsoid for the convex hull cvh( X ) and let U ∈ ∆( X ) be a distribution that satisfies E x ∼ U x = c . Then, it holds that

Proof. Lemma 4 ensures that there exists a positive definite matrix H with the property that

Now, write y := c -x ∗ -c d and note that

Therefore, y ∈ cvh( X ) . Recall that, for all arms x ∈ X , the associated reward ( r x ) is nonnegative and, hence, the rewards' expected value satisfies ⟨ x, θ ∗ ⟩ ≥ 0 . This inequality and the containment y ∈ cvh( X ) give us ⟨ y, θ ∗ ⟩ ≥ 0 . Substituting y = c -x ∗ -c d in the last inequality leads to ⟨ c, θ ∗ ⟩ ≥ ⟨ x ∗ , θ ∗ ⟩ / ( d + 1) . Given that E x ∼ U [ x ] = c , we obtain the desired inequality E x ∼ U ⟨ x, θ ∗ ⟩ = ⟨ c, θ ∗ ⟩ ≥ ⟨ x ∗ ,θ ∗ ⟩ ( d +1) .

Note that at each iteration of the subroutine GenerateArmSequence , with probability 1 / 2 , we insert an arm into S that is sampled according to U . Using this observation and Lemma 5, we obtain that, for any round t ∈ [ ˜ T ] and for the random arm X t pulled from the sequence S according to our procedure, the observed reward r X t must satisfy E [ r X t ] ≥ ⟨ x ∗ ,θ ∗ ⟩ 2( d +1) . 8

Further, recall that in the subroutine GenerateArmSequence , we insert arms x ∈ Supp ( λ ) at least ⌈ λ x ˜ T / 3 ⌉ times, where λ corresponds to the solution of D-optimal design problem defined in equation (5). Therefore, we can characterize the confidence widths of the estimated rewards for each arm in X computed using the least squares estimate ̂ θ computed in Line 7 in Algorithm 2.

Broadly speaking, we can show that all arms with low expected reward (less than a threshold) also have an estimated reward at most twice the true reward. On the other hand, high rewarding arms must have an estimated reward to be within a factor of 2 of the true reward. Thus, based on certain high probability confidence bounds (equation (7)), we can eliminate arms in X with true expected reward less than some threshold, with high probability.

3.2 Part II : Phased Elimination via Estimate Dependent Confidence Widths

Note that while analyzing average regret via confidence bound algorithms, it is quite common to use, for each arm x , a confidence width (interval) that does not depend on x 's estimated reward. This is a reasonable design choice for bounding average regret, since the regret incurred at each round is the sum of confidence intervals that grow smaller with the round index and, hence, this choice

leads to a small average regret. However, for the analysis of the Nash regret, a confidence width that is independent of the estimated reward can be highly unsatisfactory: the confidence width might be larger than the optimal ⟨ x ∗ , θ ∗ ⟩ . This can in turn allow an arm with extremely low reward to be pulled leading to the geometric mean going to zero. In order to alleviate this issue, it is vital that our confidence intervals are reward dependent. This in turn, requires one to instantiate concentration bounds similar to the multiplicative version of the standard Chernoff bound. In general, multiplicative forms of concentration bounds are much stronger than the additive analogues [16]. In prior work [5] on Nash regret for the stochastic multi-armed bandits setting, such concentration bounds were readily available through the multiplicative version of the Chernoff bound. However, in our context of linear bandits, the derivation of analogous concentration bounds (and the associated confidence widths) is quite novel and requires a careful use of the sub-Poisson property.

In particular, we use the following confidence bounds (with estimate dependent confidence widths) in our algorithm. We define the lower and upper confidence bounds considering any arm x , any least squares estimator ϕ (of θ ∗ ), and t the number of observations used to compute the estimator ϕ . That is, for any triple ( x, ϕ, t ) ∈ X × R d × [ T ] , we define Lower Nash Confidence Bound ( LNCB ) and Upper Nash Confidence Bound ( UNCB ) as follows:

As mentioned previously, the confidence widths in equation (7) are estimate dependent.

Next, we provide a high level overview of Part II in Algorithm 2. This part is inspired from the phased elimination algorithm for the average regret ([17], Chapter 21); a key distinction here is to use the Nash confidence bounds defined in (7). Part II in Algorithm 2 begins with the set of arms ˜ X ⊆ X obtained after an initial elimination of low rewarding arms in Part I . Subsequently, Part II runs in phases of exponentially increasing length and eliminates sub-optimal arms in every phase.

Suppose at the beginning of the ℓ th phase, ˜ X is the updated set of arms. We solve the D-optimal design problem (see (6)) corresponding to ˜ X to obtain a distribution λ ∈ ∆( ˜ X ) . For the next O ( d 2 +2 ℓ ˜ T ) rounds, we pull arms a in the support of λ (Line 13): each arm a ∈ Supp ( λ ) is pulled ⌈ λ a T ′ ⌉ times where T ′ = O (2 ℓ ˜ T ) . Using the data covariance matrix and the observed noisy rewards, we recompute: (1) an improved estimate ̂ θ (of θ ∗ ) and (2) improved confidence bounds for every surviving arm. Then, we eliminate arms based on the confidence bounds and update the set of surviving arms (Lines 16 and 17).

The following lemma provides the key concentration bound for the least squares estimate.

Lemma 6. Let x 1 , x 2 , . . . , x s ∈ R d be a fixed set of vectors and let r 1 , r 2 , . . . , r s be independent ν -sub-Poisson random variables satisfying E r s = ⟨ x s , θ ∗ ⟩ for some unknown θ ∗ . Further, let matrix V = ∑ s j =1 x j x T j and ̂ θ = V -1 ( ∑ j r j x j ) be the least squares estimator of θ ∗ . Consider any z ∈ R d with the property that z T V -1 x j ≤ γ for all j ∈ [ s ] . Then, for any δ ∈ [0 , 1] we have

Lemma 6 is established in Appendix B. Using this lemma, we can show that the optimal arm x ∗ is never eliminated with high probability.

Lemma 7. Consider any bandit instance in which for the optimal arm x ∗ ∈ X we have ⟨ x ∗ , θ ∗ ⟩ ≥ 192 √ d ν T log( T |X| ) . Then, with probability at least ( 1 -4 log T T ) , the optimal arm x ∗ always exists in the surviving set ˜ X in Part I and in every phase in Part II of Algorithm 2.

Finally, using Lemmas 6 and 7, we show that, with high probability, in every phase of Part II all the surviving arms x ∈ ˜ X have sufficiently high reward means.

Lemma 8. Consider any phase ℓ in Part II of Algorithm 2 and let ˜ X be the surviving set of arms at the beginning of that phase. Then, with ˜ T = √ dν T log( T |X| ) , we have

Here, ν is the sub-Poisson parameter of the stochastic rewards.

The proofs of the Lemmas 7 and 8 are deferred to Appendix C.

3.3 Main Result

This section states and provides a proof sketch of the main theorem. 9

Theorem 1. For any given stochastic linear bandits problem with (finite) set of arms X ⊂ R d , time horizon T ∈ Z + , and ν -sub-Poisson rewards, Algorithm 2 achieves Nash regret

Here, β = max { 1 , ⟨ x ∗ , θ ∗ ⟩ log d } , with x ∗ ∈ X denoting the optimal arm and θ ∗ the (unknown) parameter vector.

Note that, in Theorem 1, lower the value of the optimal expected reward, ⟨ x ∗ , θ ∗ ⟩ , stronger is the Nash regret guarantee. In particular, with a standard normalization assumption that ⟨ x ∗ , θ ∗ ⟩ ≤ 1 and

(√ )

for 1 -sub Poisson rewards, we obtain a Nash regret of O d T log( T |X| ) . Also, observe that the regret guarantee provided in Theorem 1 depends logarithmically on the size of X . Hence, the Nash regret is small even when |X| is polynomially large in d .

Proof Sketch of Theorem 1: We condition on the event E defined as the intersection of the (high probability) events defined in Lemmas 7 and 8, respectively. Union bound implies that event E c holds with probability at most O ( T -1 ) . For Part I, in the first ˜ T = 3 √ T dν log( T |X| ) rounds, we bound from below the product of the expected rewards, using Lemma 5, as follows

For remaining rounds, we invoke Lemma 8 (specifically, equation (10)) to obtain

The above equations reduce to the following bound on the geometric mean of the expected rewards (across the T rounds):

Therefore, the Nash regret of LINNASH satisfies

This completes the proof sketch.

Computational Efficiency of LINNASH. We note that Algorithm 2 (LINNASH) executes in polynomial time. In particular, the algorithm calls the subroutine GENERATEARMSEQUENCE in Part I for computing the John Ellipsoid. Given a set of arm vectors as input, this ellipsoid computation can be performed efficiently (see Chapter 3 in [25]). In fact, for our purposes an approximate version of the John Ellipsoid suffices, and such an approximation can be found much faster [8]; specifically, in time O ( |X| 2 d ) . Furthermore, the algorithm solves the D-optimal design problem, once in Part I and at most O (log T ) times in Part II. The D-optimal design is a concave maximization problem, which can be efficiently solved using, say, the Frank-Wolfe algorithm with rank1 updates. Each iteration takes O ( |X| 2 ) time, and the total number of iterations is at most O ( d ) (see, e.g., Chapter 21 of [17] and Chapter 3 in [25]). Overall, we get that LINNASH is a polynomial-time algorithm.

4 Extension of Algorithm LINNASH for Infinite Arms

The regret guarantee in Theorem 1 depends logarithmically on |X| . Such a dependence makes the guarantee vacuous when the set of arms X is infinitely large (or even |X| = Ω(2 √ T d -1 ) ). To resolve this limitation, we extend LINNASH with a modified confidence width that depends only on the largest estimated reward γ := max x ∈X ⟨ x, ̂ θ ⟩ . Specifically, we consider the confidence width 16 √ γ d 5 2 ν log ( T ) T ′ , for all the arms, and select the set of surviving arms in each phase (of Part II of the algorithm for infinite arms) as follows:

See Algorithm 3 in Appendix D for details. The theorem below is the main result of this section.

Theorem 2. For any given stochastic linear bandits problem with set of arms X ⊂ R d , time horizon T ∈ Z + , and ν -sub-Poisson rewards, Algorithm 2 achieves Nash regret

Here, β = max { 1 , ⟨ x ∗ , θ ∗ ⟩ log d } , with x ∗ ∈ X denoting the optimal arm and θ ∗ the (unknown) parameter vector.

Proof of Theorem 2 and a detailed regret analysis of Algorithm 3 can be found in Appendix D.

5 Conclusion and Future Work

Fairness and welfare considerations have emerged as a central design objectives in online decisionmaking contexts. Motivated, broadly, by such considerations, the current work addresses the notion of Nash regret in the linear bandits framework. We develop essentially tight Nash regret bounds for linear bandit instances with a finite number of arms.

In addition, we extend this guarantee to settings wherein the number of arms is infinite. Here, our regret bound scales as d 5 / 4 , where d is the ambient dimension. Note that, for linear bandits with infinite arms, [1] obtains a bound of d/ √ T for average regret. We conjecture that a similar dependence should be possible for Nash regret as well and pose this strengthening as a relevant direction of future work. Another important direction would be to study Nash regret for more other bandit frameworks (such as contextual bandits and combinatorial bandits) and Markov Decision Processes (MDPs).

Acknowledgments and Disclosure of Funding

Siddharth Barman's reserach is supported by a SERB Core research grant (CRG/2021/006165).

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013.

Shubhada Agrawal, Sandeep K Juneja, and Wouter M Koolen. Regret minimization in heavytailed bandits. In Conference on Learning Theory , pages 26-62. PMLR, 2021.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Siddharth Barman, Arindam Khan, Arnab Maiti, and Ayush Sawarni. Fairness and welfare quantification for regret in multi-armed bandits. arXiv preprint arXiv:2205.13930 , 2022.

Ilai Bistritz, Tavor Baharav, Amir Leshem, and Nicholas Bambos. My fair bandit: Distributed learning of max-min fairness with multi-player bandits. In International Conference on Machine Learning , pages 930-940. PMLR, 2020.

Ioannis Caragiannis, David Kurokawa, Hervé Moulin, Ariel D Procaccia, Nisarg Shah, and Junxing Wang. The unreasonable fairness of maximum nash welfare. ACM Transactions on Economics and Computation (TEAC) , 7(3):1-32, 2019.

Michael B Cohen, Ben Cousins, Yin Tat Lee, and Xin Yang. A near-optimal algorithm for approximating the john ellipsoid. In Conference on Learning Theory , pages 849-873. PMLR, 2019.

Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. 2008.

Jürgen Eckhoff. Helly, radon, and carathéodory type theorems. In Handbook of convex geometry , pages 389-448. Elsevier, 1993.

Martin Grötschel, László Lovász, and Alexander Schrijver. Geometric algorithms and combinatorial optimization , volume 2. Springer Science & Business Media, 2012.

Safwan Hossain, Evi Micha, and Nisarg Shah. Fair algorithms for multi-agent multi-armed bandits. Advances in Neural Information Processing Systems , 34:24005-24017, 2021.

Ralph Howard. The john ellipsoid theorem. University of South Carolina , 1997.

Matthew Joseph, Michael Kearns, Jamie H Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. Advances in neural information processing systems , 29, 2016.

Jack Kiefer and Jacob Wolfowitz. The equivalence of two extremum problems. Canadian Journal of Mathematics , 12:363-366, 1960.

William Kuszmaul and Qi Qi. The multiplicative version of azuma's inequality, with an application to contention analysis. arXiv preprint arXiv:2102.05077 , 2021.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Andres Munoz Medina and Scott Yang. No-regret algorithms for heavy-tailed linear bandits. In International Conference on Machine Learning , pages 1642-1650. PMLR, 2016.

Hervé Moulin. Fair division and collective welfare . MIT press, 2004.

Vishakha Patil, Ganesh Ghalme, Vineet Nair, and Yadati Narahari. Achieving fairness in the stochastic multi-armed bandit problem. The Journal of Machine Learning Research , 22(1):78857915, 2021.

Paat Rusmevichientong and John N Tsitsiklis. Linearly parameterized bandits. Mathematics of Operations Research , 35(2):395-411, 2010.

Eric M Schwartz, Eric T Bradlow, and Peter S Fader. Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science , 36(4):500-522, 2017.

Ambuj Tewari and Susan A Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile Health: Sensors, Analytic Methods, and Applications , pages 495-517, 2017.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Michael J Todd. Minimum-volume ellipsoids: Theory and algorithms . SIAM, 2016.

Yogatheesan Varatharajah and Brent Berry. A contextual-bandit-based approach for informed decision-making in clinical trials. Life , 12(8):1277, 2022.

Sofia Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the optimal design of clinical trials: Benefits and challenges. Statistical Science , 30:199-215, 05 2015.

Limitations: The main contributions of our works are theoretical. From a theoretical point of view, the limitations of our paper are discussed in Section 5. In particular, we believe that tightening the gap between the upper and lower bounds in Nash regret for an infinite set of arms will require novel and non-trivial algorithmic ideas - we leave this as an important direction of future work.

Broader Impact: Due to the theoretical nature of this work, we do not foresee any adverse societal impact of this work.

A Proof of Lemmas 1 and 2

Lemma 1. Any non-negative random variable X ∈ [0 , B ] is B -sub-Poisson, i.e., if mean E [ X ] = µ , then for all λ ∈ R , we have E [ e λX ] ≤ exp ( B -1 µ ( e Bλ -1 )) .

Proof. For random variable X we have

Lemma 2. Let X be a non-negative sub-Gaussian random variable X with mean µ = E [ X ] and sub-Gaussian norm σ . Then, X is also ( σ 2 µ ) -sub-Poisson.

Proof. Since X is a σ -sub-Gaussian random variable, for any non-negative scalar s ≥ 0 , we have

The fact that X is a positive random variable implies that the mean µ > 0 . Also, the considered scalar s ≥ 0 and, hence, the term sσ 2 µ > 0 . Also, recall that e x ≥ 1 + x + x 2 2 , for any non-negative x . Using these observations and equation (12), we obtain

For random variable X , inequality (13) ensures that the required mgf bound (equation (2)) holds for all non-negative s and with sub-Poisson parameter equal to σ 2 µ .

We next complete the proof by showing that the mgf bound holds for negative s as well. Towards this, write B := σ 2 µ and define random variable Y := 1 { X ≤ B } X + 1 { X>B } B . Note that Y is a positive, bounded random variable. Furthermore, for any negative s , we have exp( sY ) ≥ exp( sX ) . Therefore, for a negative s , it holds that E [exp( sX )] ≤ E [exp ( sY )] . Since positive random variable Y ∈ [0 , B ] , the mgf bound obtained in Lemma 1 gives us

Since B := σ 2 µ , the mgf bound (equation (2)) on X holds for negative s as well. This, overall, shows that X is a ( σ 2 µ ) -sub-Poission random variable. The lemma stands proved.

B Proof of Concentration Bounds

Lemma 6. Let x 1 , x 2 , . . . , x s ∈ R d be a fixed set of vectors and let r 1 , r 2 , . . . , r s be independent ν -sub-Poisson random variables satisfying E r s = ⟨ x s , θ ∗ ⟩ for some unknown θ ∗ . Further, let matrix V = ∑ s j =1 x j x T j and ̂ θ = V -1 ( ∑ j r j x j ) be the least squares estimator of θ ∗ . Consider any z ∈ R d with the property that z T V -1 x j ≤ γ for all j ∈ [ s ] . Then, for any δ ∈ [0 , 1] we have

Proof. We use the Chernoff method to get an upper bound on the desired probabilities, as shown below

Substituting c = log(1+ δ ) νγ , we get

Since 1 γ z T V -1 x t ≤ 1 we have (1 + δ ) 1 γ z T V -1 x t ≤ 1 + δ · 1 γ z T V -1 x t . Substituting in (14) we get

Using the logarithmic inequality log(1 + δ ) ≥ 2 δ 2+ δ , we further simplify as

Following similar steps and substituting c = log(1 -δ ) νγ , we obtain a bound on the lower tail (inequality 9):

Now, using the logarithmic inequality (1 -δ ) log(1 -δ ) ≥ -δ + δ 2 2 , we get

Combining (9) and (8) we get the following Corollary.

Corollary 9. Using the notations as in Lemma 6, we have

The next two lemmas are variants of Lemma 6 where we bound the error in terms of an upper bound on ⟨ z, θ ∗ ⟩ .

Lemma 10. Let x 1 , x 2 , . . . , x s ∈ R d be a fixed set of vectors and let r 1 , r 2 , . . . , r s be independent ν -sub Poisson random variables satisfying E r s = ⟨ x s , θ ∗ ⟩ for some unknown θ ∗ . In that case, let matrix V = ∑ s j =1 x j x T j and ̂ θ = V -1 ( ∑ j r j x j ) be the least squares estimator of θ ∗ . Consider any z ∈ R d that satisfies z T V -1 x j ≤ γ for all j ∈ [ s ] and ⟨ z, θ ∗ ⟩ ≤ α . Then for any δ ∈ [0 , 1] we have

Proof. Following the same approach as in the proof of Lemma 6, we have

Now, substituting c = 1 νγ log (1 + δ )) and using (1 + δ ) 1 γ z T V -1 x t ≤ 1 + δ · 1 γ z T V -1 x t we have

Since δ ∈ [0 , 1] , we have the desired result.

Lemma 11. Using the same notations as in Lemma 10, for any δ ∈ [0 , 1] , the following holds

Proof. Using steps similar to the previous lemmas, we obtain

( r t are sub-poisson and independent)

Substituting c = log(1 -δ ) νγ and simplifing we get

Note that since log(1 -δ ) + δ is negative, we can upper bound the above expression by replacing ⟨ z, θ ∗ ⟩ with α .

Hence, the lemma stands proved.

C Regret Analysis of Algorithm 2

We will first define events E 1 and E 2 for each phase of the algorithm and show that they hold with high probability. We will use the events in the regret analysis.

Event E 1 : At the end of Part I, let ̂ θ be the unbiased estimator of θ ∗ and ˜ T be as defined in Algorithm 2. All arms x ∈ X with ⟨ x, θ ∗ ⟩ < 10 √ dν log ( T |X| ) T satisfy

In addition, all arms x ∈ X with ⟨ x, θ ∗ ⟩ ≥ 10 √ dν log ( T |X| ) T satisfy

Event E 2 : Let ˜ X denote the surviving set of arms at the start of a phase in Part II, and T ′ be as defined in Algorithm 2. For all phases and for all x ∈ ˜ X such that ⟨ x, θ ∗ ⟩ ≥ 10 √ dν log ( T |X| ) T , the estimator ̂ θ (calculated at the end of a phase) satisfies

C.1 Supporting Lemmas

Lemma 12 (Chernoff Bound) . Let Z 1 , . . . , Z n be independent Bernoulli random variables. Consider the sum S = ∑ n r =1 Z r and let µ = E [ S ] be its expected value. Then, for any ε ∈ [0 , 1] , we have

Lemma 13. During Part I , arms from D-optimal design are added to S at least ˜ T / 3 times with probability greater than 1 -1 T .

Proof. We use Lemma 12 with Z i as indicator random variables that take value one when an arm from A (the support of λ in the optimal design) is chosen. By setting ε = 1 3 and µ = ˜ T 2 , we obtain the required probability bound.

Lemma 14. Using the notations in Algorithm 1, if the event in Lemma 13 holds, then for each x ∈ X and each round t in Part I of the algorithm, we have

where X t is the arm pulled in round t .

Proof. Let U ( λ ) and λ denote the optimal design matrix (as defined in (4)) and the solution to the D-optimal design problem in Algorithm 1, respectively. That is, λ is the solution of the optimization problem stated in equation (5) and U ( λ ) = ∑ x ∈X λ x xx T . Lemma 3 implies that || x || U ( λ ) -1 ≤ √ d for all x ∈ X .

Next, note that the construction of the sequence S in Part I (Subroutine GenerateArmSequence ) and the event specified in Lemma 13 give us V ≻ ˜ T 3 U ( λ ) . Hence,

The next lemma lower bounds the probability of event E 1 (see equations (18), (19), and (20)). Lemma 15. Event E holds with probability at least 1 -6 .

1 T

Proof. First, consider all arms x ∈ X for which ⟨ x, θ ∗ ⟩ < 10 √ dν log ( T |X| ) T . Here, we invoke Lemma 10, with γ = 3 d ˜ T (as derived in Lemma 14), α = 10 √ dν log ( T |X| ) T , and δ = 1 , to obtain

Next, we consider arms x ∈ X such that ⟨ x, θ ∗ ⟩ ≥ 10 √ dν log ( T |X| ) T and for such arms establish equations (19) and (20). Towards this, we invoke Lemma 6, with parameters γ = 3 d ˜ T and δ =

3 √ dν log ( T |X| ) ⟨ x,θ ∗ ⟩ ˜ T . It is relevant to note that here δ ∈ [0 , 1] - this containment follows from the condition ⟨ x, θ ∗ ⟩ ≥ 10 √ dν log ( T |X| ) T and ˜ T = 3 √ T dν log( T |X| ) . Therefore,

For establishing equation (20), we invoke Lemma 6 again, now with γ = 3 d ˜ T and δ = 1 3 :

Similarly, with δ = 1 2 , Lemma 6 gives us

Finally, we combine (23), (24), (25) and (26), and apply a union bound over all arms in X . Then, conditioning on the event in Lemma 13 leads to the stated probability bound. The lemma stands proved.

The next lemma shows that event E 2 (see equations (21) and (22)) holds with high probability Lemma 16. Event E 2 holds with probability at least 1 -3 log T .

T

Proof. Consider any phase in Part II and let U ( λ ) be the optimal design matrix obtained after solving the D-optimal design problem at the start of the phase. By Lemma 3, for all x, z ∈ ˜ X we have

First, we address equation (21). In particular, we instantiate Lemma 6 with δ = 3 √ dν log ( T |X| ) ⟨ x,θ ∗ ⟩ T ′ and γ = d T ′ . Note that given the lower bound on ⟨ x, θ ∗ ⟩ and the inequality T ′ ≥ 2 √ T dν log( T |X| ) ensure that δ lies in [0 , 1] . Hence, substituting these values of δ and γ in Lemma 6, we obtain

Next, following a similar approach as in the proof of Lemma 15, we use Lemma 6 with δ = 1 3 and δ = 1 2 to establish the upper and lower bounds of equation (22), respectively. Applying a union bound across arms in ˜ X and over all-at most log T -phases, we obtain the desired probability bound of 1 -3 log T T .

Corollary 17.

Proof. From Lemma 15 we have P { E 1 } ≥ 1 -6 T . Furthermore, from Lemma 16 we have P { E 2 } ≥ 1 -3 log T T . Applying a union bound on the complements of these two events establishes the corollary.

Lemma 18. Consider any bandit instance with ⟨ x ∗ , θ ∗ ⟩ ≥ 192 √ dν log ( T |X| ) T . If event E 1 holds, then any arm with mean ⟨ x, θ ∗ ⟩ ≤ 10 √ dν log( T |X| ) T is eliminated after Part I of Algorithm 2.

Proof. We will show that in the given bandit instance and under the event E 1 , for each arm x ∈ X with mean ⟨ x, θ ∗ ⟩ ≤ 10 √ dν log( T |X| ) T the upper Nash confidence bound (see equation (7)) is less than the lower confidence bound of the optimal arm x ∗ . Hence, all such arms x are eliminated from consideration in Line 8 of Algorithm 2. This will establish the lemma.

The upper Nash confidence bound of arm x at the end of Part I is defined as

In the given bandit instance and under event E 1 , for the optimal arm x ∗ , we have

Therefore, the lower Nash confidence bound of x ∗ satisfies

Equations (29) and (27) imply

As mentioned previously, Line 8 in Algorithm 2 eliminates all arms x that satisfy inequality (30). Hence, the lemma stands proved

C.2 Proofs of Lemmas 7 and 8

Lemma 7. Consider any bandit instance in which for the optimal arm x ∗ ∈ X we have ⟨ x ∗ , θ ∗ ⟩ ≥ 192 √ d ν T log( T |X| ) . Then, with probability at least ( 1 -4 log T T ) , the optimal arm x ∗ always exists in the surviving set ˜ X in Part I and in every phase in Part II of Algorithm 2.

Proof. We will show that, under events E 1 and E 2 , throughout the execution of Algorithm 2 the UNCB of the optimal arm x ∗ is never less than the LNCB of any arm x . Hence, then the optimal arm x ∗ never satisfies the elimination criterion in Algorithm 2 and, hence, x ∗ always exists in the surviving set of arms.

First, we consider arms x with the property that ⟨ x, θ ∗ ⟩ < 10 √ dν log ( T |X| ) T . For any such arm x , at the end of Part I of the algorithm we have

Hence, at the end of Part I, arm x ∗ is not eliminated via the LNCB of any x which satisfies ⟨ x, θ ∗ ⟩ < 10 √ dν log ( T |X| ) T . Further, note that, under event E 1 , such arms are eliminated at the end of Part I (Lemma 18). Hence, the LNCB of such arms are not even considered in the phases of Part II. To complete the proof, we next show that the UNCB of the optimal arm x ∗ is at least the LNCB of all arms x which bear ⟨ x, θ ∗ ⟩ ≥ 10 √ dν log ( T |X| ) T . Below, we will consider the Nash confidence bounds for a general T ′ . Replacing T ′ by ˜ T gives us the desired confidence-bounds comparison for the end of Part I - this repetition is omitted.

Complementarily, for optimal arm x ∗ we have

Since ⟨ x ∗ , θ ∗ ⟩ ≥ ⟨ x, θ ∗ ⟩ for all arms x , inequalities (31) and (32) lead to the confidence-bounds comparison:

Hence, if events E 1 and E 2 hold, then the optimal arm x ∗ is never eliminated from Algorithm 2. Further, Corollary 17 ensures that the events E 1 and E 2 hold with probability at least 1 -4 log T T . Hence, the lemma stands proved.

Lemma 8. Consider any phase ℓ in Part II of Algorithm 2 and let ˜ X be the surviving set of arms at the beginning of that phase. Then, with ˜ T = √ dν T log( T |X| ) , we have

Here, ν is the sub-Poisson parameter of the stochastic rewards.

Proof. For the analysis, assume that events E 1 and E 2 hold. Lemma 7 ensures that the optimal arm is contained in the surviving set of arms ˜ X . Furthermore, if an arm x ∈ ˜ X at the beginning of the ℓ th phase, then it must be the case that arm x was not eliminated in the previous phase (which executed for T ′ / 2 rounds); in particular, we have UNCB( x, ̂ θ, T ′ / 2) ≥ LNCB( x ∗ , ̂ θ, T ′ / 2) . This inequality reduces to

Rearranging the terms, we obtain

Further, invoking equation (21) for x ∗ leads to

Substituting T ′ = 2 ℓ ˜ T / 3 , the above inequality reduces to the desired bound in (10). From Corollary 17, we have that the events E 1 and E 2 hold with probability at least 1 -4 log T T . Hence, the lemma stands proved.

C.3 Proof of Theorem 1

Theorem 1. For any given stochastic linear bandits problem with (finite) set of arms X ⊂ R d , time horizon T ∈ Z + , and ν -sub-Poisson rewards, Algorithm 2 achieves Nash regret

Here, β = max { 1 , ⟨ x ∗ , θ ∗ ⟩ log d } , with x ∗ ∈ X denoting the optimal arm and θ ∗ the (unknown) parameter vector.

Proof. We will assume, without loss of generality, that ⟨ x ∗ , θ ∗ ⟩ ≥ 192 √ d ν T log( T |X| ) , otherwise the stated Nash Regret bound directly holds (see equation (1)). Write E to denote the 'good' event identified in Lemma 8; the lemma ensures that P { E } ≥ 1 -4 log T T .

During Part I of Algorithm 2, the product of expected rewards, conditioned on E , satisfies

For analyzing Part II, we will utilize Lemma 8. Write B ℓ to denote all the rounds t that belong to ℓ th phase (in Part II). Also, let T ′ ℓ denote the associated phase-length parameter, i.e., T ′ ℓ = 2 ℓ ˜ T / 3 . Note that in each phase ℓ (i.e., in the for-loop at Line 12 of Algorithm 2), every arm a in Supp ( λ ) (the support of D-optimal design) is pulled ⌈ λ a T ′ ℓ ⌉ times. Given that | Supp ( λ ) | ≤ d ( d +1) / 2 , we have |B ℓ | ≤ T ′ ℓ + d ( d +1) 2 . By construction T ′ ℓ ≥ d ( d +1) 2 and, hence, |B ℓ | ≤ 2 T ′ ℓ . Since the phase length parameter, T ′ ℓ , doubles after each phase, the algorithm would have at most log T phases. Hence, the product of expected rewards in Part II satisfies

The last inequality follows from the fact that (1 -x ) r ≥ (1 -2 rx ) , for any r ∈ [0 , 1] and x ∈ [0 , 1 / 2] . Note that the term √ dν log ( T |X| ) ⟨ x ∗ ,θ ∗ ⟩ T ′ ℓ ≤ 1 / 2 , since ⟨ x ∗ , θ ∗ ⟩ ≥ 192 √ dν T log( T |X| ) along with T ′ ℓ ≥ 2 √ T dν log T |X| and T ≥ e 4 . We further simplify the expression as follows

Combining the lower bound for the expected rewards in the two parts we get

Therefore, the Nash Regret can be bounded as

Hence, with β = max { 1 , √ ⟨ x ∗ , θ ∗ ⟩ , ⟨ x ∗ , θ ∗ ⟩ log d } = max { 1 , ⟨ x ∗ , θ ∗ ⟩ log d } , from equation (34) we obtain the desired bound on Nash regret NR T = O ( β √ d ν T log( T |X| ) ) . The theorem stands proved.

D Algorithm 3 and Regret Analysis

Algorithm 3 LINNASH (Nash Confidence Bound Algorithm for Infinite Set of Arms)

Input: Arm set X and horizon of play T .

1: Initialize matrix V ← [0] d,d and number of rounds ˜ T = 3 √ T d 2 . 5 ν log( T ) . Part I

2: Generate arm sequence S for the first ˜ T rounds using Algorithm 1.

3: for t = 1 to ˜ T do

4: Pull the next arm X t from the sequence S .

5: Observe reward r t and update V ← V + X t X T t

6: end for

7: Set estimate ̂ θ := V -1 ( ∑ ˜ T t =1 r t X t )

8: Find γ = max z ∈X ⟨ z, ̂ θ ⟩

11: while end of time horizon T is reached do

12: Initialize V = [0] d,d to be an all zeros d × d matrix and s = [0] d to be an all-zeros vector. // Beginning of new phase.

13: Find the probability distribution λ ∈ ∆( ˜ X ) by maximizing the following objective

14: for each arm a in Supp ( λ ) do

15: Pull arm a for the next ⌈ λ a T ′ ⌉ rounds.

16: Observe rewards and Update V ← V + ⌈ λ a T ′ ⌉ · aa T

17: Observe ⌈ λ a T ′ ⌉ corresponding rewards z 1 , z 2 , . . . and update s ← s +( ∑ j z j ) a .

18: end for

19: Estimate ̂ θ = V -1 (∑ t ∈E r t X t )

20: Find γ = max z ∈X ⟨ z, ̂ θ ⟩

22: T ′ ← 2 × T ′ // End of phase.

23: end while

Instead of ensuring probability bounds on individual arms, we construct a confidence ellipsoid around θ ∗ . In the context of Algorithm 3, we define the following events for the regret analysis:

G 1 In Part I, arms from the D-optimal design are chosen at least ˜ T / 3 times. If ⟨ x ∗ , θ ∗ ⟩ ≥ 196 √ d 2 . 5 ν T log T , then ̂ θ calculated at the end of Part I satisfies

G 2 In Part II, for every phase, if ⟨ x ∗ , θ ∗ ⟩ ≥ 196 √ d 2 . 5 ν T log T , the estimators ̂ θ satisfy:

Without loss of generality, we assume throughout that ⟨ x ∗ , θ ∗ ⟩ ≥ 196 d 1 . 25 √ ν √ T log T . Otherwise, the regret bound in Theorem 2 trivially holds. Let B denote the unit ball in R d . We have

We construct an ε -net for the unit ball, denoted as C ε . For any y ∈ B , we define y ε := arg min b ∈C ε ∥ b -y ∥ 2 . We can now write

Rearranging, we obtain

In the following lemmas, we show that |⟨ y V 1 2 , θ -θ ∗ ⟩| is small for all values of y .

T |C ε |

ε ̂ ε Lemma 19. Let x 1 , x 2 , . . . , x n be a sequence of fixed arm pulls (from a set X ) such that each arm x in the support λ from D-optimal design (for X ) is pulled at least ⌈ λ x τ ⌉ times. Consider the matrix V = ∑ n j =1 x j x T j and let z be a vector such that ∥ z ∥ 2 ≤ 1 and ⟨ z V 1 2 , θ ∗ ⟩ ≥ 6 ν √ d τ log ( T |C ε | ) . Then, with probability greater than 1 -2 , we have,

Proof. We begin by utilizing Lemma 6. First, we determine the γ parameter in the lemma as follows, for any t ∈ [ n ] we have

Let A λ be the optimal design matrix. Since V ≻ τA λ , we have

Now, we use Corollary 9 with γ = √ d τ and δ = ( 3 √ d τ ν log ( T |C ε | ) ⟨ z V 1 2 ,θ ∗ ⟩ ) 1 2 . Note that δ ∈ [0 , 1] since

⟨

z

V

1

2

, θ

6

d

τ

ν

log (

T

|C

ε

|

)

. We obtain the following probability bound

Finally, we establish an upper bound on the term ⟨ z V 1 2 , θ ∗ ⟩ as follows

Substituting in (37) we get the lemma statement. This completes the proof of the lemma.

∗

√

Lemma 20. Consider the same notation as in Lemma 19. If ⟨ z V 1 2 , θ ∗ ⟩ ∈ [ 0 , 6 ν √ d τ log ( T |C ε | ) ] , then with probability greater than 1 -2 T |X| we have

Proof. Utilizing Lemma 10, with δ = 1 , α = 6 ν √ d τ log ( T |C ε | ) , and γ = √ d τ , we have ⟨ z V 1 2 , ̂ θ ⟩ ≤ 12 ν √ d τ log ( T |C ε | ) . Since ⟨ z V 1 2 , θ ∗ ⟩ ≥ 0 , it follows, with probability greater than 1 -1 T |X| , that

Next, applying Lemma 11 with δ = 1 and α = 6 ν √ d τ log ( T |C ε | ) , we have, with probability greater than 1 -1 T |X| ,

Hence, the lemma stands proved.

Lemma 21. If ⟨ x ∗ , θ ∗ ⟩ ≥ 196 √ d 2 . 5 ν T log T , then

Proof. First, we note (from Lemma 13) that arms from the solution of the D-optimal design problem are selected (with probability greater than 1 -1 T ) at least ˜ T / 3 times. Hence, we can use Lemmas 19 and 20 with τ = ˜ T / 3 .

Let us consider the case where ⟨ y ε V 1 2 , θ ∗ ⟩ ≥ 6 √ 3 d ˜ T log ( T |C ε | ) . We have that the following holds with probability greater than 1 -1 T |C ε | :

Next, we note that |C ε | ≤ ( 3 ε ) d [17], and by choosing ε = 1 / 2 we get

Taking a union bound over all elements in C ε gives a probability bound of 1 -1 T .

Now, for the case where ⟨ y ε V 1 2 , θ ∗ ⟩ ∈ [ 0 , 6 √ 3 d ˜ T log ( T |C ε | ) ] , substituting τ = ˜ T / 3 in Lemma 20 we have, with probability greater than 1 -1 T |C ε | ,

The last inequality is due to the fact that ⟨ x ∗ , θ ∗ ⟩ ≥ 196 √ d 2 . 5 ν T log T and ˜ T = 3 √ T νd 2 . 5 log T . We again take a union bound over all elements in C ε to get a probability bound of 1 -1 T . Finally, a union bound over the two cases and the event in Lemma 13 proves the lemma.

Lemma 22. If ⟨ x ∗ , θ ∗ ⟩ ≥ 196 √ d 2 . 5 ν T log T , then

Proof. To prove Lemma 22, we follow the same steps as in the proof of Lemma 21. Utilizing Lemma 19 and Lemma 20 with τ = T ′ , we establish that for any fixed phase, the following inequality holds with probability greater than 1 -1 T :

Taking a union bound over all - at most log T - phases in Part II of Algorithm 3 gives us the desired lower bound on P { G 2 } .

Corollary 23. If G 1 holds, then for all x ∈ X , ̂ θ calculated at the end of Part I satisfies

Consider any phase ℓ in Part II . If G 2 holds, then for every arm in the surviving arm set ˜ X , ̂ θ calculated at the end of the phase satisfies

Proof. First we use Hölder's inequality

Since G 1 holds, arms from the optimal design matrix are selected at least ˜ T / 3 times; we have by Lemma 3

Similarly, for every phase in Part II with T ′ = 2 ℓ ˜ T / 3 we have

Finally, using bounds on ∥ ∥ ∥ θ ∗ -̂ θ ∥ ∥ ∥ V from events G 1 and G 2 , and substituting in (40), we get the desired bound.

Proof. Since T ′ ≥ 2 ˜ T / 3 , via Corollary 23 any ̂ θ calculated in Part I or during any phase of Part II satisfies

We have

Now, for any x ∈ X ,

Hence, the lemma stands proved.

Lemma 25. If events G 1 and G 2 hold then the optimal arm x ∗ always exists in the surviving set ˜ X in every phase in Part II of Algorithm 3

Proof. Let τ = ˜ T / 3 for Part I and τ = T ′ for every phase of Part II. From Corollary 23 we have

Hence, the best arm will never satisfy the elimination criteria in Algorithm 3.

Lemma 26. Given that events G 1 and G 2 hold, consider any phase index ℓ in Part II of Alg. 3. For the surviving set of arms ˜ X at the beginning of that phase, and for ˜ T = √ d 2 . 5 ν T log( T ) , the following inequality holds for all x ∈ ˜ X

Proof. Lemma 25 ensures that the optimal arm is contained in the surviving set of arms ˜ X . Furthermore, if an arm x ∈ ˜ X is pulled in the ℓ th phase, then it must be the case that arm x was not eliminated in the previous phase (with a phase length parameter T ′ 2 ); in particular the arms x does not satisfy the inequality on Line 21 of Algorithm 3. This inequality reduces to

Substituting T ′ = 2 l ˜ T / 3 in the above inequality proves the Lemma.

Theorem 2. For any given stochastic linear bandits problem with set of arms X ⊂ R d , time horizon T ∈ Z + , and ν -sub-Poisson rewards, Algorithm 2 achieves Nash regret

Here, β = max { 1 , ⟨ x ∗ , θ ∗ ⟩ log d } , with x ∗ ∈ X denoting the optimal arm and θ ∗ the (unknown) parameter vector.

Proof. Without loss of generality, we assume that ⟨ x ∗ , θ ∗ ⟩ ≥ 196 √ d 2 . 5 ν T log T . Otherwise, the Nash Regret bound is trivially true. For Part I, the product of expected rewards satisfies

For Part II, we use Lemma 8. Let E i denote the time interval of the i th phase, and let T ′ i be the phase length parameter in that phase. Recall that |E i | ≤ T ′ i + d ( d +1) 2 . Also, the algorithm runs for at most log T phases. Hence, we have

The last inequality is due to the fact that (1 -x ) r ≥ (1 -2 rx ) where r ∈ [0 , 1] and x ∈ [0 , 1 / 2] . Note that the term √ d 2 . 5 ν log ( T ) ⟨ x ∗ ,θ ∗ ⟩ T ′ j ≤ 1 / 2 for ⟨ x ∗ , θ ∗ ⟩ ≥ 196 √ d 2 . 5 ν T log T , T ′ ≥ 2 √ T d 2 . 5 ν log T , and T ≥ e 6 . We can further simplify the expression as follows

Combining the lower bound for rewards in Part I and Part II of the algorithm, we obtain

Hence, the Nash Regret can be bounded as

The theorem stands proved.

E Experiments

We conduct experiments to compare the performance of our algorithm LINNASH with Thompson Sampling on synthetic data. For a comparison, we select Thompson Sampling (Algorithm 1 in

10000

15000

20000

25000

30000

35000

40000

45000

50000

Rounds

0.08

0.10

0.12

0.14

0.16

0.18

0.20

0.22

Nash Regret

LinNash

Thomson Sampling

0

10000

20000

30000

40000

50000

0.0

0.1

0.2

0.3

0.4

0.5

Round-wise Reward

LinNash

0

10000

20000

30000

40000

50000

Rounds

0.0

0.1

0.2

0.3

0.4

0.5

Round-wise Reward

Thompson Sampling

Rounds

[2]), instead of UCB/OFUL, since randomization is essential to achieve meaningful Nash Regret guarantees.

We fine-tune the parameters of both algorithms and evaluate their performance in the following experimental setup: We fix the ambient dimension d = 80 , the number of arms |X| = 10000 , and the number of rounds T = 50000 . Both the unknown parameter vector, θ ∗ , and the arm embeddings are sampled from a multivariate Gaussian distribution. Subsequently, the arm embeddings are shifted and normalized to ensure that all mean rewards are non-negative, with the maximum reward mean being set to 0 . 5 . Upon pulling an arm, we observe a Bernoulli random variable with a probability corresponding to its mean reward.

In this experimental setting, we observe a significant performance advantage of LINNASH over Thompson Sampling. We plot our results in Figure 1, which shows that the Nash regret of LINNASH decreases notably faster than that of Thompson Sampling.

Another notable advantage of LINNASH evident from the experiments is due to successive elimination. The variance in the quality of arms pulled decreases as the number of rounds progresses - see Figures 2 and 3. This is due to the bulk elimination of suboptimal arms at regular intervals. In contrast, Thompson Sampling incurs a large variance in quality of arms being pulled even after several rounds, since no arms are being eliminated at any point.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this research, the authors investigate the concept of Nash regret in the context of stochastic linear bandits. Nash regret represents the disparity between the optimal reward and the geometric mean of expected rewards, reflecting fairness considerations across different time instances within the Nash social welfare framework. The study assumes a fixed set of arms denoted as $\\mathcal{X}$, and an unknown vector $\\theta^*$. When an arm x is chosen, the agent observes a sub-Poisson $r_t$ with a mean of $<x,\\theta^*>$. The nash regret is defined as $max_x<x,\\theta^*>-(\\prod_t \\mathbb{E}[<X_t,\\theta^*>])^{1/T}$ where $X_t$ is a selected arm at time t.\n\nTo minimize Nash regret, the authors propose an algorithm that incorporates elimination phases. Firstly, to ensure the lower bound of expected rewards, they employ sampling via the Jon ellipsoid technique, as in Lemma 4. Additionally, they employ a G-optimal design to construct a probability distribution over arms, aiming to minimize the confidence interval width of estimated rewards. By combining these strategies, a sequence of arms is generated, and elimination phases are employed to remove arms with suboptimal mean rewards, utilizing upper confidence bounds (UCB) and lower confidence bounds (LCB) with a doubling time horizon for exploration.\nTo construct the UCB and LCB, a Nash confidence bound is utilized, which contains the mean reward value. Larger mean rewards correspond to larger confidence intervals, as defined in equation (8).\n\nThe theoretical analysis demonstrates that for a finite number of arms, the proposed algorithm achieves a nash regret of approximately $\\tilde{O}(\\sqrt{d/T})$. In the case of an infinite number of arms, the regret is $\\tilde{O}(d^{5/4}/\\sqrt{T})$ with a modified confidence width.\n\nSTRENGTHS:\n- This paper addresses the concept of nash regret in the context of stochastic linear bandits.\n- It presents a novel algorithm based on elimination phases and provides a detailed regret analysis.\n\nWEAKNESSES:\n- The paper does not discuss the computational aspects of finding the G-optimal distribution and sampling via the John ellipsoid method.\n- It does not include experimental results.\n- In the case of infinite arms, the achieved regret bound may not be sufficiently tight, as it involves a factor of $d^{5/4}$ instead of just $d$.\n\nQUESTIONS:\n- I'm wondering what is the reason for using an elimination-based algorithm rather than UCB.\n- wondering what is the reason to consider sub Poisson distribution rather than sub gaussian.\n- In line 504 (proof of Theorem 1 in appendix), when $\\langle x^*,\\theta^* \\rangle\\le \\sqrt{d\\nu/T}$, what is the reason that the NR bound, which contains $\\langle x^\\star,\\theta^\\star \\rangle$ in the bound,  trivially holds?\n\nLIMITATIONS:\n- This problem deals with a fixed arms set rather than a contextual linear setting.\n- The regret bound of $d^{5/4}$ in the case of infinite arms may not be tight, and it would be better to elaborate on why the factor of $5/4$ appears. ( $\\sqrt{d\\log|\\mathcal{X}|}$ regret bound for the finite case seems to imply $d$ when we consider $|\\mathcal{X}|=2^d$ for infinite case.)\n- There are no experimental results and discussion on the computational cost, which raises concerns about practical aspects\n\nMinor comments:\n\n- line 441 in Appendix: $d$-> $\\sqrt{d}$\n- line 449 in Appendix: $\\langle x,\\hat{\\theta}\\rangle \\le 20$ --> $\\langle x,\\hat{\\theta}\\rangle \\ge 20$\n- Lemma 7, equation (10) in Appendix: $P()\\le 4T^{-1}\\log T$ -->$P()\\ge 1-4T^{-1}\\log T$  ?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider stochastic linear bandits, and the goal is to minimize Nash regret, which is a stronger notion than the standard regret, by replacing the arithmetic mean reward of the learner over time with the geometric one in the definition.\n\nSTRENGTHS:\nThe paper is well written. I didn't check the proof but I didn't spot any critical errors in the main text, either. The setting, the algorithms, and the proof sketch all make sense to me. I agree that indeed in some scenarios we may need to consider fairness across rounds and the definition of Nash regret in this paper is probably a good notion to consider in this case.\n\nWEAKNESSES:\nThis setting is rarely known. It doesn't mean this problem is unimportant, but I think it would be better to provide more context for it to better evaluate the contributions. In the related work, the authors only mention that this setting differs from prior works because they all consider fairness across arms while this paper focuses on fairness across rounds. However, I would expect more discussions about this like whether previous techniques can be applied here. Also, there is no discussion about the Nash regret of multi-armed bandits, which is an important special case. Although this setup may not be meaningful in practice, it may provide insights into handling the linear case when designing algorithms or proving regret bounds.\n\nQUESTIONS:\n- The current algorithm requires $T$ as input. Do the authors think this is necessary? Part I in Algorithm 2 is somewhat non-standard for a successive elimination algorithm, so I guess it might be difficult to apply a doubling trick to remove the prior knowledge of $T$.\n- Does the sub-Poisson assumption include Gaussian as a special case? If not, can the analysis go through under sub-Gaussian?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends algorithmic techniques for the linear bandits problem to the Nash Social Welfare objective function which ensures a level of fairness rather than merely the utilitarian welfare. Their work provides essentially optimal regret guarantees as compared to the traditional linear bandits problem (as it is well known that maximizing the geometric mean is necessarily stronger than the arithmetic mean). These results work for both a finite and infinite set of arms with slightly different regret guarantees.\n\nSTRENGTHS:\nThe paper takes a novel approach to the linear bandits problem by adapting the objective function to one that ensures some level of fairness across agents in the given system. Moreover, all results appear sound and presented in a nice and compact form within the main text -- the sketches and intuition are clearly depicted with full proofs (not all of which were verified by this reviewer) in the appendix. In addition to near optimal regret guarantees on this modified problem instance, the authors provide various tail bounds that are of independent research to the community.\n\nWEAKNESSES:\nThe paper lacks a comprehensive discussion on the related work surrounding this problem. Though the problem at hand is novel, it does hurt the submission to not include such context for readers. Specifically, there is considerable work on online NSW maximization which is naturally a similar problem. Moreover, discussion of the linear bandits problem and the best known results there would be very useful (in addition to the brief mention of Nash regret being a stronger guarantee). Some discussion on these results for finite and infinite arms would also potentially alleviate the issue of the gap between upper and lower bounds in the latter setting. As noted by the paper, improving the guarantee for the infinite arm problem setting would naturally strengthen the submission considerably.\n\nQUESTIONS:\nCan you provide a brief sketch of the final theorem (infinite arms) regret guarantee? I understand that the submissions page limit prevents this but it would be helpful to have one in the extended version of the paper.\n\nHave you considered the regret guarantees of the contextual bandits problem with a NSW objective?\n\nHave you considered the multiplicative regret on this multiplicative objective function as well?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem minimize 'Nash regret' in linear bandit, where Nash regret here is defined as the difference between the optimum and the geometric mean of expected rewards induced by the bandit algorithm. The stochastic reward in this paper is assumed sub-Poisson. The authors give algorithm to achieve Nash regret of $O(\\sqrt{d/T}log(T/|\\mathcal{X}|)$ when the set of arms $\\mathcal{X}\\in \\mathbb{R}^d$ is finite and $O(\\frac{d^{5/4}}{\\sqrt{T}}logT)$ when the set of arms $\\mathcal{X}$ is infinite.\n\nSTRENGTHS:\n- This paper exhibits a good technical foundation. Specifically, in the finite-arm case, the upper bound is tightly bounded, even considering logarithmic terms.\n\n- The notion of regret explored in this paper is novel and is stronger compared to external regret.\n\nWEAKNESSES:\nThe paper lacks experimental results. It would be benefical to compare other bandit algorithm with the algorithm in the paper specifically in terms of this kind of Nash regret.\n\nQUESTIONS:\n- The term \"Nash regret\" has been previously used in [1] and [2], but with a different definition from the one presented in this paper. The current name \"Nash regret\" in this paper does not capture the characteristic of the geometric mean.\n\n- Regarding the definition of Nash regret in this paper, the expectation is taken within the geometric mean instead of first taking the geometric mean and then the expectation. Is there any specific reasons or motivations behind the choice of this order?\n\n- In equation (3), the solution obtained provides the optimal sequence of arm pulls. What is the meaning of 'optimal' here? Further explanation and intuition behind equation (3) can be provided by the authors to better understand the optimality criteria and its implications.\n\n- In line 439 in the appendix, there's a wrong ref (Algorithm ??)\n\n[1] Cui, Qiwen, et al. \"Learning in congestion games with bandit feedback.\" Advances in Neural Information Processing Systems 35 (2022): 11009-11022.\n\n[2] Tay, Sebastian Shenghong, et al. \"No-regret Sample-efficient Bayesian Optimization for Finding Nash Equilibria with Unknown Utilities.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2023.\n\nLIMITATIONS:\nThe authors have partially addressed the limitations of their work, though there is space for improvement (see the section Weaknesses and Questions).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach based on the Nash regret to extend the stochastic linear bandits framework with fairness and welfare considerations. The Nash regret relies on the geometric mean of expected rewards induced by the linear bandit algorithm, which tends to incentivize uniformly high rewards as opposed to the average regret which promotes cumulatively high rewards. In this setting, the authors derived tight upper bounds on this strengthened notion of regret, thereby providing fairness guarantees as the geometric mean of rewards corresponds to the Nash social welfare (NSW) function.\n\nBuilding upon a novel sampling arms strategy based on John ellipsoid and optimal design, as well as tailored concentration bounds, the authors propose an algorithm that achieves $O\\left(\\sqrt{\\frac{d\\nu}{T}}log(T|\\chi|)\\right)$, for a finite set of arms $\\chi$, and $O\\left(\\frac{d^{5/4}\\nu^{1/2}}{\\sqrt{T}} log(T)\\right)$, in the infinite case.\n\nSTRENGTHS:\nThe paper is technically solid and well-motivated. I really like the idea of inherently imposing fairness to the bandit algorithm via the Nash regret. To the best of my knowledge, the authors' result is new and provides valuable insight into incorporating fairness and welfare considerations into bandit algorithms. In addition, the authors propose tail bounds that may be of broader interest.\n\nWEAKNESSES:\nAlthough the paper is technically well-set, the contribution seems incremental. The paper is also suffering from a lack of comparisons with existing approaches. The related work section seems a bit short. Having a richer picture of the literature would be beneficial for understanding the context of the paper’s contribution.\n\nLastly, I feel that the paper could benefit from numerical experiments that support the derived theoretical results.\n\nQUESTIONS:\nWhat is the computational efficiency of the LinNash algorithm?\n\nLIMITATIONS:\nThe limitations of the proposed framework are not entirely clear.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}