SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ayush Sawarni
Indian Institute of Science Bangalore
sawarniayush@gmail.com
Google Research Bangalore
soumyabrata@google.com
Siddharth Barman
barman@iisc.ac.in
We obtain essentially tight upper bounds for a strengthened notion of regret in the stochastic linear bandits framework. The strengthening—referred to as Nash regret—is defined as the difference between the (a priori unknown) optimum and the geometric mean of expected rewards accumulated by the linear bandit algorithm. Since the geometric mean corresponds to the well-studied Nash social welfare (NSW) function, this formulation quantifies the performance of a bandit algorithm as the collective welfare it generates across rounds. NSW is known to satisfy fairness axioms and, hence, an upper bound on Nash regret provides a principled fairness guarantee.
We consider the stochastic linear bandits problem over a horizon of  $\mathsf{T}$  rounds and with set of arms  $\mathcal{X}$  in ambient dimension  $d$ . Furthermore, we focus on settings in which the stochastic reward—associated with each arm in  $\mathcal{X}$ —is a non-negative,  $\nu$ -sub-Poisson random variable. For this setting, we develop an algorithm that achieves a Nash regret of  $O\left(\sqrt{\frac{d\nu}{\mathsf{T}}}\log (\mathsf{T}|\mathcal{X}|)\right)$ . In addition, addressing linear bandit instances in which the set of arms  $\mathcal{X}$  is not necessarily finite, we obtain a Nash regret upper bound of  $O\left(\frac{d^{\frac{5}{4}}\nu^{\frac{1}{2}}}{\sqrt{\mathsf{T}}}\log (\mathsf{T})\right)$ . Since bounded random variables are sub-Poisson, these results hold for bounded, positive rewards. Our linear bandit algorithm is built upon the successive elimination method with novel technical insights, including tailored concentration bounds and the use of sampling via John ellipsoid in conjunction with the Kiefer-Wolffowitz optimal design.
Bandit optimization is a prominent framework for sequential decision making and has several applications across multiple domains, such as healthcare [26, 23, 27] and advertising [22]. In this framework, we have a set of arms (possible actions) with unknown means and a time horizon. The goal is to sequentially pull the arms such that the regret—which is a notion of loss defined over the bandit instance—is minimized.
We consider settings wherein the stochastic rewards generated by a sequential algorithm induces welfare across a population of agents. Specifically, there are  $\mathsf{T}$  agents, arriving one per round; in particular, the reward accrued at each round  $t\in [\mathsf{T}]$  corresponds to the value accrued by the  $t^{\mathrm{th}}$  agent. Indeed, such a welfarist connection exists in various applications of the bandit framework. Consider, for instance, the classic context of drug trials [24]: Suppose there are  $\mathsf{T}$  patients and several available drugs. In each round  $t\in [\mathsf{T}]$ , one of the available drugs is administered to the  $t^{\mathrm{th}}$  patient. Subsequently, the reward accrued at the  $t^{\mathrm{th}}$  round corresponds to the efficacy of the administered drug to the  $t^{\mathrm{th}}$  patient. In such a setting, fairness is a fundamental consideration. That is, in addition to cumulative efficacy, individual effectiveness of the drugs is quite important.
A central notion in the bandit literature is that of average regret, defined as the difference between the (a priori unknown) optimum and the arithmetic mean of the expected rewards (accumulated by the algorithm) [4]. However, average regret fails to capture the fairness criterion that the rewards should be balanced (across the agents) and not just cumulatively high. From a welfarist viewpoint, the standard notion of (average) regret equates the algorithm's performance to the social welfare it induces. Social welfare is defined as the sum of agent's rewards [19] and can be high among a set of agents even if a fraction of them receive indiscriminately low rewards. For instance, in the drug-trials example provided above, high average efficacy (i.e., high social welfare) does not rule out a severely ineffective outcome for a subset of agents.
Given that average regret is defined using the sum of expected rewards, this notion inherits this utilitarian limitation of social welfare. In summary, in welfare-inducing contexts, a bandit algorithm with low average regret is not guaranteed to induce fair outcomes across rounds.
Addressing this issue and with the overarching aim of achieving fairness across rounds (i.e., across agents that receive rewards from a bandit algorithm), the current work considers a strengthened notion of regret. The strengthening—referred to as Nash regret—is defined as the difference between the (a priori unknown) optimum and the geometric mean of expected rewards induced by the bandit algorithm. It is relevant to note that the geometric mean (of rewards) corresponds to the Nash social welfare (NSW) function [19]. This welfare function has been extensively studied in mathematical economics (see, e.g., [19]) and is known to satisfy fundamental fairness axioms, including the Pigou-Dalton transfer principle, scale invariance, and independence of unconcerned agents. Hence, by definition, Nash regret quantifies the performance of a bandit algorithm as the NSW it generates.
Quantitatively speaking, in order for the geometric mean (i.e., the NSW) to be large, the expected reward at every round should be large enough. The AM-GM inequality also highlights that Nash regret is a more demanding objective that average regret.
We obtain novel results for Nash regret in the stochastic linear bandits framework. In this well-studied bandit setup each arm corresponds to a  $d$ -dimensional vector  $x$  (an arm-specific context) and the unknown arm means are modelled to be a linear function of  $x$ . With a focus on average regret, stochastic linear bandits have been extensively studied in the past decade [1, 9, 21]. The current paper extends the line of work on linear bandits with fairness and welfare considerations.
Note that an ostensible approach for minimizing Nash regret is to take the logarithm of the observed rewards and, then, solve the average regret problem. However, this approach has the following shortcomings: (i) Taking log implies the modified rewards can have a very large range possibly making the regret vacuous, and (ii) This approach leads to a multiplicative guarantee and not an additive one. In a recent work of [5], the authors study Nash regret in the context of stochastic multi-armed bandits (with bounded rewards) and provide optimal guarantees. The current work notably generalizes this prior work to linear bandits.
1.1 Our Contributions and Techniques
We consider the stochastic linear bandits setting with a set of arms  $\mathcal{X}$  over a finite horizon of  $\mathsf{T}$  rounds. Since we consider the welfarist viewpoint, we assume that the rewards across all the rounds are positive and, in particular, model the distribution of the arm rewards to be  $\nu$ -sub-Poisson, for parameter  $\nu \in \mathbb{R}_{+}$ . Our goal is to minimize the Nash regret  $\mathrm{NR}_{\mathsf{T}}$ .
We develop a novel algorithm LINNASH that obtains essentially optimal Nash regret guarantees for this setting. Specifically, for a finite set of arms  $\mathcal{X} \subset \mathbb{R}^d$ , our algorithm LINNASH achieves Nash regret  $\mathrm{NR}_{\mathrm{T}} = O\left(\sqrt{\frac{d\nu}{\mathrm{T}}}\log (\mathrm{T}|\mathcal{X}|)\right)$ . For infinite sets of arms, a modified version of LINNASH achieves Nash regret  $\mathrm{NR}_{\mathrm{T}} = O\left(\frac{d^{\frac{5}{4}}\nu^{\frac{1}{2}}}{\sqrt{\mathrm{T}}}\log (\mathrm{T})\right)$ .
Recall that Nash regret is a strengthening of the average regret; the AM-GM inequality implies that, for any bandit algorithm, the Nash regret is at least as much as its average regret. Hence, in the linear bandits context, the known  $\Omega\left(d / \sqrt{\overline{T}}\right)$  lower bound on average regret (see [17], Chapter 24) holds
for Nash regret as well. $^{1}$  This observation implies that, up to a logarithmic factor, our upper bound on Nash regret is tight with respect to the number of rounds  $\mathsf{T}$ . We also note that for instances in which the number of arms  $|\mathcal{X}| = \omega(2^{d})$ , the Nash-regret dependence on  $d$  has a slight gap. Tightening this gap is an interesting direction of future work.
We note that bounded, positive random variables are sub-Poisson (Lemma 1). Hence, our results hold for linear bandit instances wherein the stochastic rewards are bounded and positive. This observation also highlights the fact that the current work is a generalization of the result obtained in [5]. In addition, notice that, by definition, Poisson distributions are 1-sub-Poisson. Hence, our guarantees further hold of rewards that are not necessarily sub-Gaussian. Given the recent interest in obtaining regret guarantees beyond sub-Gaussian rewards [18, 3], our study of sub-Poisson rewards is interesting in its own right.[2]
Our linear bandit algorithm, LINNASH, has two parts. In the first part, we develop a novel approach of sampling arms such that in expectation the reward obtained is a linear function of the center of John Ellipsoid [13]. Such a strategy ensures that the expected reward in any round of the first part is sufficiently large. The second part of LINNASH runs in phases of exponentially increasing length. In each phase, we sample arms according to a distribution that is obtained as a solution of a concave optimization problem, known as D-optimal design. We construct confidence intervals at each phase and eliminate sub-optimal arms. A key novelty in our algorithm and analysis is the use of confidence widths that are estimate dependent. We define these widths considering multiplicative forms of concentration bounds and crucially utilize the sub-Poisson property of the rewards. The tail bounds we develop might be of independent interest.
There has been a recent surge in interest to achieve fairness guarantees in the context of multi-armed bandits; see, e.g., [14, 7, 20, 6, 12]. However, these works mostly consider fairness across arms and, in particular, impose fairness constraints that require each arm to be pulled a pre-specified fraction of times. By contrast, our work considers fairness across rounds.
Alternative Regret Formulations. In the current work, for the welfare computation, each agent  $t$ 's value is considered as the expected reward in round  $t$ . One can formulate stronger notions of regret by, say, considering the expectation of the geometric mean of the rewards, rather than the geometric mean of the expectations. However, as discussed in [5], it is not possible to obtain non-trivial guarantees for such reformulations in general: every arm must be pulled at least once. Hence, if one considers the realized rewards (and not their expectations), even a single pull of a zero-reward arm will render the geometric mean zero.
We will write  $[m]$  to denote the set  $\{1,2,\ldots,m\}$ . For a matrix  $\mathbf{X}$ , let  $\operatorname{Det}(\mathbf{X})$  to denote the determinant of  $\mathbf{X}$ . For any discrete probability distribution  $\lambda$  with sample space  $\Omega$ , write  $\operatorname{Supp}(\lambda) \triangleq \{x \in \Omega : \operatorname{Pr}_{X \sim \lambda} \{X = x\} > 0\}$  to denote the points for which the probability mass assigned by  $\lambda$  is positive. For a vector  $\mathbf{a} \in \mathbb{R}^d$  and a positive definite matrix  $\mathbf{V} \in \mathbb{R}^{d \times d}$ , we will denote  $||a||_{\mathbf{V}} := \sqrt{a^T\mathbf{V}a}$ . Finally, let  $\mathcal{B} := \{x \in \mathbb{R}^d | ||x||_2 = 1\}$  be the  $d$ -dimensional unit ball.
We address the problem of stochastic linear bandits with a time horizon of  $\mathsf{T} \in \mathbb{Z}_{+}$  rounds. Here, an online algorithm (decision maker) is given a set of arms  $\mathcal{X} \subset \mathbb{R}^d$ . Each arm corresponds to a  $d$ -dimensional vector. Furthermore, associated with each arm  $x \in \mathcal{X}$ , we have a stochastic reward  $r_x \in \mathbb{R}_+$ . In the linear bandits framework, the expected value of the reward  $r_x$  is modeled to be a linear function of  $x \in \mathbb{R}^d$ . In particular, there exists an unknown parameter vector  $\theta^{*} \in \mathbb{R}^{d}$  such that, for each  $x \in \mathcal{X}$ , the associated reward's expected value  $\mathbb{E}[r_x] = \langle x, \theta^{*} \rangle$ . Given the focus on welfare contexts, we will, throughout, assume that the rewards are positive,  $r_x > 0$ , for all  $x \in \mathcal{X}$ .
The online algorithm (possibly randomized) must sequentially select an arm  $X_{t}$  in each round  $t \in [\mathsf{T}]$  and, then, it observes the corresponding (stochastic) reward  $r_{X_t} > 0$ . For notational convenience,
This lower bound on average regret is obtained for instances in which the set of arms  $\mathcal{X}$  are the corners of a hypercube [17].
An intersection between sub-Gaussian and sub-Poisson distribution classes is identified in Lemma 2.
Note that, for a randomized online algorithm, the selected arm  $X_{t}$  is a random variable.
we will write  $r_t$  to denote  $r_{X_t}$ . In particular, if in round  $t$  the selected arm  $X_t = x$ , then the expected reward is  $\langle x, \theta^* \rangle$ , i.e.,  $\mathbb{E}[r_t \mid X_t = x] = \langle x, \theta^* \rangle$ . We will, throughout, use  $x^*$  to denote the optimal arm,  $x^* = \arg \max_{x \in \mathcal{X}} \langle x, \theta^* \rangle$  and  $\widehat{\theta}$  to denote estimator of  $\theta^*$ .
In the stochastic linear bandits framework, our overarching objective is to minimize the Nash regret, defined as follows:
$$
\mathrm {N R} _ {\top} := \max  _ {x \in \mathcal {X}} \left\langle x, \theta^ {*} \right\rangle - \left(\prod_ {t = 1} ^ {\top} \mathbb {E} [ \left\langle X _ {t}, \theta^ {*} \right\rangle ]\right) ^ {1 / \top} \tag {1}
$$
Note that the definition of Nash regret is obtained by applying the Nash social welfare (geometric mean) onto ex ante rewards,  $\mathbb{E}[\langle X_t,\theta^*\rangle ]$ , accrued across the T rounds.
2.1 Sub-Poisson Rewards
In order to model the environment with positive rewards  $(r_x > 0)$ , we assume that the rewards  $r_x$  associated with the arms  $x \in \mathcal{X}$  are  $\nu$ -sub Poisson, for some parameter  $\nu > 0$ . Formally, their moment-generating function satisfies the following bound
$$
\mathbb {E} \left[ e ^ {\lambda r _ {x}} \right] \leq \exp \left(\nu^ {- 1} \mathbb {E} [ r _ {x} ] (e ^ {\nu \lambda} - 1)\right) = \exp \left(\nu^ {- 1} \langle x, \theta^ {*} \rangle (e ^ {\nu \lambda} - 1)\right) \text {f o r a l l} \lambda \in \mathbb {R}. \tag {2}
$$
Note that a Poisson random variable is 1-sub Poisson. To highlight the generality of  $\nu$ -sub-Poisson distributions, we note that bounded, non-negative random variables are sub-Poisson (Lemma 1). Further, in Lemma 2, we establish a connection between non-negative sub-Gaussian and sub-Poisson random variables.
Lemma 1. Any non-negative random variable  $X \in [0, \mathsf{B}]$  is  $\mathsf{B}$ -sub-Poisson, i.e., if mean  $\mathbb{E}[X] = \mu$ , then for all  $\lambda \in \mathbb{R}$ , we have  $\mathbb{E}[e^{\lambda X}] \leq \exp(B^{-1}\mu(e^{B\lambda} - 1))$ .
Lemma 2. Let  $X$  be a non-negative sub-Gaussian random variable  $X$  with mean  $\mu = \mathbb{E}[X]$  and sub-Gaussian norm  $\sigma$ . Then,  $X$  is also  $\left(\frac{\sigma^2}{\mu}\right)$ -sub-Poisson.
The proofs of Lemmas 1 and 2 appear in Appendix A. Lemma 2 has useful instantiations. In particular, the lemma implies that the half-normal random variable, with variance of  $\sigma$ , is also a  $(C\sigma)$ -sub-Poisson, where  $C$  is a constant (independent of distribution parameters). Similarly, for other well-studied, positive sub-Gaussian random variables (including truncated and folded normal distributions), the sub-Poisson parameter is small.
Next, we discuss the necessary preliminaries for our algorithm and analysis.
2.2 Optimal Design.
Write  $\Delta(\mathcal{X})$  to denote the probability simplex associated with the set of arms  $\mathcal{X}$ . Let  $\lambda \in \Delta(\mathcal{X})$  be such a probability distribution over the arms, with  $\lambda_x$  denoting the probability of selecting arm  $x$ . The following optimization problem, defined over the set of arms  $\mathcal{X}$ , is well-known and is referred to as the G-optimal design problem.
$$
\text {M i n i m i z e} g (\lambda) \triangleq \max  _ {x \in \mathcal {X}} \| x \| _ {\mathbf {U} (\lambda) ^ {- 1}} ^ {2}, \text {w h e r e} \lambda \in \Delta (\mathcal {X}) \text {a n d} \mathbf {U} (\lambda) = \sum_ {x \in \mathcal {X}} \lambda_ {x} x x ^ {T} \tag {3}
$$
The solution to (3) provides the optimal sequence of arm pulls (for a given budget of rounds) to minimize the confidence width of the estimated rewards for all arms  $x \in \mathcal{X}$ . The G-optimal design problem connects to the following optimization problem (known as D-optimal design problem):
$$
\text {M a x i m i z e} f (\lambda) \triangleq \log \operatorname {D e t} (\mathbf {U} (\lambda)), \text {w h e r e} \lambda \in \Delta (\mathcal {X}) \text {a n d} \mathbf {U} (\lambda) = \sum_ {x \in \mathcal {X}} \lambda_ {x} x x ^ {T} \tag {4}
$$
The lemma below provides an important result of Kiefer and Wolfowitz [15].
Lemma 3 (Kiefer-Wolffowitz). If the set  $\mathcal{X}$  is compact and  $\mathcal{X}$  spans  $\mathbb{R}^d$ , then there exists  $\lambda^{*} \in \Delta(\mathcal{X})$  supported over at most  $d(d + 1)/2$  arms such that  $\lambda^{*}$  minimizes the objective in equation (3) with  $g(\lambda^{*}) = d$ . Furthermore,  $\lambda^{*}$  is also a maximizer of the D-optimal design objective, i.e.,  $\lambda^{*}$  maximizes the function  $f(\lambda) = \log \operatorname{Det}(\mathbf{U}(\lambda))$  subject to  $\lambda \in \Delta(\mathcal{X})$ .
Here, the expectation is with respect to the random variable  $X_{t}$
Algorithm 1 GenerateArmSequence (Subroutine to generate Arm Sequence)
Input: Arm set  $\mathcal{X}$  and sequence length  $\widetilde{\mathrm{T}}\in \mathbb{Z}_{+}$    
1: Find the probability distribution  $\lambda \in \Delta (\mathcal{X})$  by maximizing the following objective log Det(U(λo)) subject to  $\lambda_0\in \Delta (\mathcal{X})$  and  $\operatorname {Supp}(\lambda_0)\leq d(d + 1) / 2$  (5)   
2: Initialize multiset  $S = \emptyset$  and set  $\mathcal{A} = \operatorname {Supp}(\lambda)$  . Also, initialize count  $c_{z} = 0$  , for each arm  $z\in \mathcal{A}$    
3: Compute distribution  $U$  as described in Section 3.1.   
4: for  $i = 1$  to T do   
5: With probability  $1 / 2$  set flag  $=$  SAMPLE-U, otherwise, set flag  $= \mathrm{D / G - OPT}$    
6: if flag  $=$  SAMPLE-U or  $\mathcal{A} = \emptyset$  then   
7: Sample an arm  $\widehat{x}$  from the distribution  $U$  , and update multiset  $S\gets S\cup \{\widehat{x}\}$    
8: else if flag  $=$  D/G-OPT then   
9: Pick the next arm  $z$  in  $\mathcal{A}$  (round robin).   
10: Update multiset  $S\gets S\cup \{z\}$  and increment count  $c_{z}\leftarrow c_{z} + 1$    
11: If  $c_{z}\geq \lceil \lambda_{z}\widetilde{\mathrm{T}} /3\rceil$  , then update A  $\leftarrow$  A\{z}.   
12: end if   
13: end for   
14: return multiset  $S$
At several places in our algorithm, our goal is to find a probability distribution that minimizes the non-convex optimization problem (3). However, instead we will maximize the concave function  $f(\lambda) = \log \operatorname{Det}(\mathbf{U}(\lambda))$  over  $\lambda \in \Delta(\mathcal{X})$ . The Frank-Wolfe algorithm, for instance, can be used to solve the D-optimal design problem (4) and compute  $\lambda^*$  efficiently ([17], Chapter 21). Lemma 3 ensures that this approach works, since the G-optimal and the D-optimal design problems have the same optimal solution  $\lambda^* \in \Delta(\mathcal{X})$ , which satisfies  $\operatorname{Supp}(\lambda^*) \leq d(d + 1) / 2$ .<sup>5</sup>
2.3 John Ellipsoid.
For any convex body  $K \subset \mathbb{R}^d$ , a John ellipsoid is an ellipsoid with maximal volume that can be inscribed within  $K$ . It is known that  $K$  itself is contained within the John Ellipsoid dilated by a factor of  $d$ . Formally,
Lemma 4 ([11]). Let  $K \subset \mathbb{R}^d$  be a convex body (i.e., a compact, convex set with a nonempty interior). Then, there exists an ellipsoid  $E$  (called the John ellipsoid) that satisfies  $E \subseteq K \subseteq c + d(E - c)$ . Here,  $c \in \mathbb{R}^d$  denotes the center of  $E$  and  $c + d(E - c)$  refers to the (dialated) set  $\{c + d(x - c) : x \in E\}$ .
3 Our Algorithm LINNASH and Main Results
In this section, we detail our algorithm LINNASH (Algorithm 2), and establish an upper bound on the Nash regret achieved by this algorithm. Subsection 3.1 details Part I of LINNASH and related analysis. Then, Subsection 3.2 presents and analyzes Part II of the algorithm. Using the lemmas from these two subsections, the regret bound for the algorithm is established in Subsection 3.3.
3.1 Part I: Sampling via John Ellipsoid and Kiefer-Wolffowitz Optimal Design
As mentioned previously, Nash regret is a more challenging objective than average regret: if in any round  $t \in [\mathsf{T}]$ , the expected reward  $\mathbb{E}[r_t]$  is zero (or very close to zero), then geometric mean  $(\prod_{t=1}^{\mathsf{T}} \mathbb{E}[r_{X_t}])^{1/\mathsf{T}}$  goes to zero, even if the expected rewards in the remaining rounds are large. Hence, we need to ensure that in every round  $t \in [\mathsf{T}]$ , specifically the rounds in the beginning of the algorithm, the expected rewards are bounded from below. In [5], this problem was tackled for stochastic multi-armed bandits (MAB) by directly sampling each arm uniformly at random in the initial rounds. Such a sampling ensured that, in each of those initial rounds, the expected reward is bounded from below by the average of the expected rewards. While such a uniform sampling
Even though the two optimization problems (3) and (4) share the optimal solution, the optimal objective function values can be different.
The ellipsoid  $E$  considered in Lemma 4 is also the ellipsoid of maximal volume contained in  $K$  [11].
Here, the expectation is over randomness in algorithm and the reward noise.