SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2 LINNASH (Nash Regret Algorithm for Finite Set of Arms)
Input: Arm set  $\mathcal{X}$  and horizon of play  $T$ .  
1: Initialize matrix  $\mathbf{V} \gets [0]_{d,d}$  and number of rounds  $\widetilde{\mathsf{T}} = 3\sqrt{\mathsf{T}d\nu\log(\mathsf{T}|\mathcal{X}|)}$ .  
Part I  
2: Generate arm sequence  $\mathcal{S}$  for the first  $\widetilde{\mathsf{T}}$  rounds using Algorithm 1.  
3: for  $t = 1$  to  $\widetilde{T}$  do  
4: Pull the next arm  $X_{t}$  from the sequence  $\mathcal{S}$ , observe corresponding reward  $r_t$ , and update  $\mathbf{V} \gets \mathbf{V} + X_t X_t^T$   
5: end for  
6: Set estimate  $\widehat{\theta} := \mathbf{V}^{-1}\left(\sum_{t=1}^{\widetilde{\mathsf{T}}} r_t X_t\right)$   
7: Compute confidence bounds LNCB  $(x, \widehat{\theta}, \widetilde{\mathsf{T}}/3)$  and UNCB  $(x, \widehat{\theta}, \widetilde{\mathsf{T}}/3)$ , for all  $x \in \mathcal{X}$  (see equation (7))  
8: Set  $\widetilde{\mathcal{X}} = \left\{x \in \mathcal{X}: \mathrm{UNCB}(x, \widehat{\theta}, \widetilde{\mathsf{T}}/3) \geq \max_{z \in \mathcal{X}} \mathrm{LNCB}(z, \widehat{\theta}, \widetilde{\mathsf{T}}/3)\right\}$  and initialize  $\mathsf{T}' = \frac{2}{3} \widetilde{\mathsf{T}}$   
Part II  
9: while end of time horizon  $\mathsf{T}$  is reached do  
10: Initialize  $V = [0]_{d,d}$  to be an all zeros  $d \times d$  matrix and  $s = [0]_d$  to be an all-zeros vector. // Beginning of new phase.  
11: Find the probability distribution  $\lambda \in \Delta(\widetilde{\mathcal{X}})$  by maximizing the following objective  
\[
\log \operatorname{Det}(\mathbf{U}(\lambda_0)) \text{ subject to } \lambda_0 \in \Delta(\widetilde{\mathcal{X}}) \text{ and } \operatorname{Supp}(\lambda_0) \leq d(d+1)/2.
\]  
12: for each arm  $a$  in  $\operatorname{Supp}(\lambda)$  do  
13: Pull arm  $a$  for the next  $[\lambda_a T']$  rounds. Update  $\mathbf{V} \gets \mathbf{V} + [\lambda_a T'] \cdot a a^T$ .  
14: Observe  $[\lambda_a T']$  corresponding rewards  $z_1, z_2, \ldots$  and update  $s \gets s + (\sum_j z_j)a$ .  
15: end for  
16: Set estimate  $\widehat{\theta} = \mathbf{V}^{-1}s$  and compute LNCB  $(x, \widehat{\theta}, T')$  and UNCB  $(x, \widehat{\theta}, T')$ , for all  $x \in \mathcal{X}$  (see equation (7))  
17: Set  $\widetilde{\mathcal{X}} = \left\{x \in \widetilde{\mathcal{X}}: \mathrm{UNCB}(x, \widehat{\theta}, T') \geq \max_{z \in \mathcal{X}} \mathrm{LNCB}(z, \widehat{\theta}, T')\right\}$ . // End of phase.  
18: Update  $\mathsf{T}' \gets 2\mathsf{T}'$ .  
19: end while
strategy is reasonable for the MAB setting, it can be quite unsatisfactory in the current context of linear bandits. To see this, consider a linear bandit instance in which, all—except for one—arms in  $\mathcal{X}$  are orthogonal to  $\theta^{*}$ . Here, a uniform sampling strategy will lead to an expected reward of  $\langle x^{*},\theta^{*}\rangle /|\mathcal{X}|$ , which can be arbitrarily small for large cardinality  $\mathcal{X}$ .
To resolve this issue we propose a novel approach in the initial  $\widetilde{\mathsf{T}}\coloneqq 3\sqrt{\mathsf{T}d\nu\log(\mathsf{T}|\mathcal{X}|)}$  rounds. In particular, we consider the convex hull of the set of arms  $\mathcal{X}$  denoted as  $\operatorname {cvh}(\mathcal{X})$  and find the center  $c\in \mathbb{R}^d$  of the John ellipsoid  $E$  for the convex hull  $\operatorname {cvh}(\mathcal{X})$ . Since  $E\subseteq \operatorname {cvh}(\mathcal{X})$ , the center  $c$  of the John ellipsoid is contained within  $\operatorname {cvh}(\mathcal{X})$  as well. Furthermore, via Carathéodory's theorem [10], we can conclude that the center  $c$  can be expressed as a convex combination of at most  $(d + 1)$  points in  $\mathcal{X}$ . Specifically, there exists a size-  $(d + 1)$  subset  $\mathcal{V}\coloneqq \{y_1,\ldots ,y_{d + 1}\} \subseteq \mathcal{X}$  and convex coefficients  $\alpha_{1},\dots ,\alpha_{d + 1}\in [0,1]$  such that  $c = \sum_{i = 1}^{d + 1}\alpha_iy_i$  with  $\sum_{i = 1}^{d + 1}\alpha_{i} = 1$ . Therefore, the convex coefficients induce a distribution  $U\in \Delta (\mathcal{X})$  of support size  $d + 1$  and with  $\mathbb{E}_{x\sim U}[x] = c$ .
Lemma 5 below asserts that sampling according to the distribution  $U$  leads to an expected reward that is sufficiently large. Hence,  $U$  is used in the subroutine GenerateArmSequence (Algorithm 1).
In particular, the purpose of the subroutine is to carefully construct a sequence (multiset) of arms  $S$ , with size  $|S| = \widetilde{T}$  and to be pulled in the initial  $\widetilde{T}$  rounds. The sequence  $S$  is constructed such that (i) upon pulling arms from  $S$ , we have a sufficiently large expected reward in each pull, and (ii) we obtain an initial estimate of the inner product of the unknown parameter vector  $\theta^*$  with all arms in  $\mathcal{X}$ . Here, objective (i) is achieved by considering the above-mentioned distribution  $U$ . Now, towards the objective (ii), we compute distribution  $\lambda \in \Delta(\mathcal{X})$  by solving the optimization problem (also known as the D-optimal design problem) stated in equation (5).
We initialize sequence  $S = \emptyset$  and run the subroutine GenerateArmSequence for  $\widetilde{T}$  iterations. In each iteration (of the for-loop in Line 4), with probability  $1/2$ , we sample an arm according to the distribution  $U$  (Line 7) and include it in  $S$ . Also, in each iteration, with remaining probability  $1/2$ , we consider the computed distribution  $\lambda$  and, in particular, pick arms  $z$  from the support of  $\lambda$  in a round-robin manner. We include such arms  $z$  in  $S$  while ensuring that, at the end of the subroutine,
each such arm  $z \in \operatorname{Supp}(\lambda)$  is included at least  $\lceil \lambda_z \widetilde{\mathsf{T}} / 3 \rceil$  times. We return the curated sequence of arms  $\mathcal{S}$  at the end of the subroutine.
Our main algorithm LINNASH (Algorithm 2) first calls subroutine GenerateArmSequence to generate the sequence  $S$ . Then, the algorithm LINNASH sequentially pulls the arms  $X_{t}$  from  $S$ , for  $1 \leq t \leq \widetilde{T}$  rounds. For these initial  $\widetilde{T} = |S|$  rounds, let  $r_t$  denote the noisy, observed rewards. Using these  $\widetilde{T}$  observed rewards, the algorithm computes the ordinary least squares (OLS) estimate  $\widehat{\theta}$  (see Line 6 in Algorithm 2); in particular,  $\widehat{\theta} := (\sum_{t=1}^{\widetilde{T}} X_t X_t^T)^{-1} (\sum_{t=1}^{\widetilde{T}} r_t X_t)$ . The algorithm uses the OLS estimate  $\widehat{\theta}$  to eliminate several low rewarding arms (in Lines 7 and 8 in Algorithm 2). This concludes Part I of the algorithm LINNASH.
Before detailing Part II (in Subsection 3.2), we provide a lemma to be used in the analysis of Part I of LINNASH.
Lemma 5. Let  $c \in \mathbb{R}^d$  denote the center of a John ellipsoid for the convex hull  $\operatorname{cvh}(\mathcal{X})$  and let  $U \in \Delta(\mathcal{X})$  be a distribution that satisfies  $\mathbb{E}_{x \sim U} x = c$ . Then, it holds that
$$
\mathbb {E} _ {x \sim U} [ \langle x, \theta^ {*} \rangle ] \geq \frac {\langle x ^ {*} , \theta^ {*} \rangle}{(d + 1)}.
$$
Proof. Lemma 4 ensures that there exists a positive definite matrix  $\mathbf{H}$  with the property that
$$
\left\{x \in \mathbb {R} ^ {d}: \sqrt {(x - c) ^ {T} \mathbf {H} (x - c)} \leq 1 \right\} \subseteq \operatorname {c v h} (\mathcal {X}) \subseteq \left\{x \in \mathbb {R} ^ {d}: \sqrt {(x - c) ^ {T} \mathbf {H} (x - c)} \leq d \right\}.
$$
Now, write  $y \coloneqq c - \frac{x^{*} - c}{d}$  and note that
$$
\sqrt {(y - c) ^ {T} \mathbf {H} (y - c)} = \sqrt {\frac {(x ^ {*} - c) ^ {T} \mathbf {H} (x ^ {*} - c)}{d ^ {2}}} \leq 1 \quad (\text {s i n c e} x ^ {*} \in \operatorname {c v h} (\mathcal {X}))
$$
Therefore,  $y \in \operatorname{cvh}(\mathcal{X})$ . Recall that, for all arms  $x \in \mathcal{X}$ , the associated reward  $(r_x)$  is nonnegative and, hence, the rewards' expected value satisfies  $\langle x, \theta^* \rangle \geq 0$ . This inequality and the containment  $y \in \operatorname{cvh}(\mathcal{X})$  give us  $\langle y, \theta^* \rangle \geq 0$ . Substituting  $y = c - \frac{x^* - c}{d}$  in the last inequality leads to  $\langle c, \theta^* \rangle \geq \langle x^*, \theta^* \rangle / (d + 1)$ . Given that  $\mathbb{E}_{x \sim U}[x] = c$ , we obtain the desired inequality  $\mathbb{E}_{x \sim U}\langle x, \theta^* \rangle = \langle c, \theta^* \rangle \geq \frac{\langle x^*, \theta^* \rangle}{(d + 1)}$ .
Note that at each iteration of the subroutine GenerateArmSequence, with probability  $1/2$ , we insert an arm into  $\mathcal{S}$  that is sampled according to  $U$ . Using this observation and Lemma 5, we obtain that, for any round  $t \in [\widetilde{\mathsf{T}}]$  and for the random arm  $X_{t}$  pulled from the sequence  $\mathcal{S}$  according to our procedure, the observed reward  $r_{X_{t}}$  must satisfy  $\mathbb{E}[r_{X_{t}}] \geq \frac{\langle x^{*}, \theta^{*} \rangle}{2(d + 1)}$ .
Further, recall that in the subroutine GenerateArmSequence, we insert arms  $x \in \mathrm{Supp}(\lambda)$  at least  $\lceil \lambda_x \widetilde{T} / 3 \rceil$  times, where  $\lambda$  corresponds to the solution of D-optimal design problem defined in equation (5). Therefore, we can characterize the confidence widths of the estimated rewards for each arm in  $\mathcal{X}$  computed using the least squares estimate  $\widehat{\theta}$  computed in Line 7 in Algorithm 2.
Broadly speaking, we can show that all arms with low expected reward (less than a threshold) also have an estimated reward at most twice the true reward. On the other hand, high rewarding arms must have an estimated reward to be within a factor of 2 of the true reward. Thus, based on certain high probability confidence bounds (equation (7)), we can eliminate arms in  $\mathcal{X}$  with true expected reward less than some threshold, with high probability.
3.2 Part II: Phased Elimination via Estimate Dependent Confidence Widths
Note that while analyzing average regret via confidence bound algorithms, it is quite common to use, for each arm  $x$ , a confidence width (interval) that does not depend on  $x$ 's estimated reward. This is a reasonable design choice for bounding average regret, since the regret incurred at each round is the sum of confidence intervals that grow smaller with the round index and, hence, this choice
Here, the expectation is over both the randomness in including arm  $X_{t}$  in  $S$  and the noise in the reward.
leads to a small average regret. However, for the analysis of the Nash regret, a confidence width that is independent of the estimated reward can be highly unsatisfactory: the confidence width might be larger than the optimal  $\langle x^{*},\theta^{*}\rangle$ . This can in turn allow an arm with extremely low reward to be pulled leading to the geometric mean going to zero. In order to alleviate this issue, it is vital that our confidence intervals are reward dependent. This in turn, requires one to instantiate concentration bounds similar to the multiplicative version of the standard Chernoff bound. In general, multiplicative forms of concentration bounds are much stronger than the additive analogues [16]. In prior work [5] on Nash regret for the stochastic multi-armed bandits setting, such concentration bounds were readily available through the multiplicative version of the Chernoff bound. However, in our context of linear bandits, the derivation of analogous concentration bounds (and the associated confidence widths) is quite novel and requires a careful use of the sub-Poisson property.
In particular, we use the following confidence bounds (with estimate dependent confidence widths) in our algorithm. We define the lower and upper confidence bounds considering any arm  $x$ , any least squares estimator  $\phi$  (of  $\theta^*$ ), and  $t$  the number of observations used to compute the estimator  $\phi$ . That is, for any triple  $(x, \phi, t) \in \mathcal{X} \times \mathbb{R}^d \times [\mathsf{T}]$ , we define Lower Nash Confidence Bound (LNCB) and Upper Nash Confidence Bound (UNCB) as follows:
$$
\operatorname {L N C B} (x, \phi , t) := \langle x, \phi \rangle - 6 \sqrt {\frac {\langle x , \phi \rangle \nu d \log (\mathsf {T} | \mathcal {X} |)}{t}}
$$
$$
\operatorname {U N C B} (x, \phi , t) := \langle x, \phi \rangle + 6 \sqrt {\frac {\langle x , \phi \rangle \nu d \log (\mathsf {T} | \mathcal {X} |)}{t}}. \tag {7}
$$
As mentioned previously, the confidence widths in equation (7) are estimate dependent.
Next, we provide a high level overview of Part II in Algorithm 2. This part is inspired from the phased elimination algorithm for the average regret ([17], Chapter 21); a key distinction here is to use the Nash confidence bounds defined in (7). Part II in Algorithm 2 begins with the set of arms  $\widetilde{\mathcal{X}} \subseteq \mathcal{X}$  obtained after an initial elimination of low rewarding arms in Part I. Subsequently, Part II runs in phases of exponentially increasing length and eliminates sub-optimal arms in every phase.
Suppose at the beginning of the  $\ell^{\mathrm{th}}$  phase,  $\widetilde{\mathcal{X}}$  is the updated set of arms. We solve the D-optimal design problem (see (6)) corresponding to  $\widetilde{\mathcal{X}}$  to obtain a distribution  $\lambda \in \Delta (\widetilde{\mathcal{X}})$ . For the next  $O(d^{2} + 2^{\ell}\widetilde{\mathsf{T}})$  rounds, we pull arms  $a$  in the support of  $\lambda$  (Line 13): each arm  $a\in \operatorname {Supp}(\lambda)$  is pulled  $\lceil \lambda_a\mathsf{T}'\rceil$  times where  $\mathsf{T}' = O(2^{\ell}\widetilde{\mathsf{T}})$ . Using the data covariance matrix and the observed noisy rewards, we recompute: (1) an improved estimate  $\widehat{\theta}$  (of  $\theta^{*}$ ) and (2) improved confidence bounds for every surviving arm. Then, we eliminate arms based on the confidence bounds and update the set of surviving arms (Lines 16 and 17).
The following lemma provides the key concentration bound for the least squares estimate.
Lemma 6. Let  $x_{1}, x_{2}, \ldots, x_{s} \in \mathbb{R}^{d}$  be a fixed set of vectors and let  $r_{1}, r_{2}, \ldots, r_{s}$  be independent  $\nu$ -sub-Poisson random variables satisfying  $\mathbb{E}r_{s} = \langle x_{s}, \theta^{*} \rangle$  for some unknown  $\theta^{*}$ . Further, let matrix  $\mathbf{V} = \sum_{j=1}^{s} x_{j}x_{j}^{T}$  and  $\widehat{\theta} = \mathbf{V}^{-1}\left(\sum_{j} r_{j}x_{j}\right)$  be the least squares estimator of  $\theta^{*}$ . Consider any  $z \in \mathbb{R}^{d}$  with the property that  $z^{T}\mathbf{V}^{-1}x_{j} \leq \gamma$  for all  $j \in [s]$ . Then, for any  $\delta \in [0,1]$  we have
$$
\mathbb {P} \left\{\langle z, \widehat {\theta} \rangle \geq (1 + \delta) \langle z, \theta^ {*} \rangle \right\} \leq \exp \left(- \frac {\delta^ {2} \langle z , \theta^ {*} \rangle}{3 \nu \gamma}\right) a n d \tag {8}
$$
$$
\mathbb {P} \left\{\langle z, \widehat {\theta} \rangle \leq (1 - \delta) \langle z, \theta^ {*} \rangle \right\} \leq \exp \left(- \frac {\delta^ {2} \langle z , \theta^ {*} \rangle}{2 \nu \gamma}\right). \tag {9}
$$
Lemma 6 is established in Appendix B. Using this lemma, we can show that the optimal arm  $x^{*}$  is never eliminated with high probability.
Lemma 7. Consider any bandit instance in which for the optimal arm  $x^{*} \in \mathcal{X}$  we have  $\langle x^{*}, \theta^{*} \rangle \geq 192\sqrt{\frac{d\nu}{T}}\log(\mathsf{T}|\mathcal{X}|)$ . Then, with probability at least  $\left(1 - \frac{4\log\mathsf{T}}{\mathsf{T}}\right)$ , the optimal arm  $x^{*}$  always exists in the surviving set  $\widetilde{\mathcal{X}}$  in Part I and in every phase in Part II of Algorithm 2.
Finally, using Lemmas 6 and 7, we show that, with high probability, in every phase of Part II all the surviving arms  $x \in \widetilde{\mathcal{X}}$  have sufficiently high reward means.
Lemma 8. Consider any phase  $\ell$  in Part II of Algorithm 2 and let  $\widetilde{\mathcal{X}}$  be the surviving set of arms at the beginning of that phase. Then, with  $\widetilde{\mathsf{T}} = \sqrt{d\nu\mathsf{T}\log(\mathsf{T}|\mathcal{X}|)}$ , we have
$$
\Pr \left\{\langle x, \theta^ {*} \rangle \geq \langle x ^ {*}, \theta^ {*} \rangle - 2 5 \sqrt {\frac {3 d \nu \langle x ^ {*} , \theta^ {*} \rangle \log (\mathrm {T} | \mathcal {X} |)}{2 ^ {\ell} \cdot \widetilde {\mathrm {T}}}} f o r a l l x \in \widetilde {\mathcal {X}} \right\} \geq 1 - \frac {4 \log \mathrm {T}}{\mathrm {T}} \tag {10}
$$
Here,  $\nu$  is the sub-Poisson parameter of the stochastic rewards.
The proofs of the Lemmas 7 and 8 are deferred to Appendix C.
3.3 Main Result
This section states and provides a proof sketch of the main theorem.
Theorem 1. For any given stochastic linear bandits problem with (finite) set of arms  $\mathcal{X} \subset \mathbb{R}^d$ , time horizon  $\mathsf{T} \in \mathbb{Z}_+$ , and  $\nu$ -sub-Poisson rewards, Algorithm 2 achieves Nash regret
$$
\mathrm {N R} _ {\mathsf {T}} = O \left(\beta \sqrt {\frac {d \nu}{\mathsf {T}}} \log (\mathsf {T} | \mathcal {X} |)\right).
$$
Here,  $\beta = \max \{1,\langle x^{*},\theta^{*}\rangle \log d\}$ , with  $x^{*}\in \mathcal{X}$  denoting the optimal arm and  $\theta^{*}$  the (unknown) parameter vector.
Note that, in Theorem 1, lower the value of the optimal expected reward,  $\langle x^{*},\theta^{*}\rangle$ , stronger is the Nash regret guarantee. In particular, with a standard normalization assumption that  $\langle x^{*},\theta^{*}\rangle \leq 1$  and for 1-sub Poisson rewards, we obtain a Nash regret of  $O\left(\sqrt{\frac{d}{\mathbb{T}}}\log (\mathsf{T}|\mathcal{X}|)\right)$ . Also, observe that the regret guarantee provided in Theorem 1 depends logarithmically on the size of  $\mathcal{X}$ . Hence, the Nash regret is small even when  $|\mathcal{X}|$  is polynomially large in  $d$ .
Proof Sketch of Theorem 1: We condition on the event  $\mathcal{E}$  defined as the intersection of the (high probability) events defined in Lemmas 7 and 8, respectively. Union bound implies that event  $\mathcal{E}^c$  holds with probability at most  $O(\mathsf{T}^{-1})$ . For Part I, in the first  $\tilde{\overline{T}} = 3\sqrt{\mathsf{T}d\nu\log(\mathsf{T}|\mathcal{X}|)}$  rounds, we bound from below the product of the expected rewards, using Lemma 5, as follows
$$
\prod_ {t = 1} ^ {\widetilde {\mathsf {T}}} \mathbb {E} [ \langle X _ {t}, \theta^ {*} \rangle \mid \mathcal {E} ] ^ {\frac {1}{\mathsf {T}}} \geq \left(\frac {\langle x ^ {*} , \theta^ {*} \rangle}{2 (d + 1)}\right) ^ {\frac {\widetilde {\mathsf {T}}}{\mathsf {T}}} \geq \langle x ^ {*}, \theta^ {*} \rangle^ {\frac {\widetilde {\mathsf {T}}}{\mathsf {T}}} \left(1 - \frac {\widetilde {\mathsf {T}} \log (2 (d + 1))}{\mathsf {T}}\right).
$$
For remaining rounds, we invoke Lemma 8 (specifically, equation (10)) to obtain
$$
\prod_ {t = \widetilde {\mathsf {T}} + 1} ^ {\mathsf {T}} \mathbb {E} [ \langle X _ {t}, \theta^ {*} \rangle | \mathcal {E} ] ^ {\frac {1}{\mathsf {T}}} \geq \langle x ^ {*}, \theta^ {*} \rangle^ {\frac {\mathsf {T} - \widetilde {\mathsf {T}}}{\mathsf {T}}} \prod_ {\ell = 1} ^ {\log \mathsf {T}} \left(1 - 5 0 \frac {| \widetilde {\mathsf {T}} \cdot (2 ^ {\ell} / 3) + 2 d ^ {2} |}{\mathsf {T}} \sqrt {\frac {3 d \nu \log (\mathsf {T} | \mathcal {X} |)}{\langle x ^ {*} , \theta^ {*} \rangle 2 ^ {\ell} \cdot \widetilde {\mathsf {T}}}}\right).
$$
The above equations reduce to the following bound on the geometric mean of the expected rewards (across the T rounds):
$$
\prod_ {t = 1} ^ {\tau} \mathbb {E} [ \langle X _ {t}, \theta^ {*} \rangle ] ^ {\frac {1}{\tau}} \geq \prod_ {t = 1} ^ {\tau} \left(\mathbb {E} [ \langle X _ {t}, \theta^ {*} \rangle \mid \mathcal {E} ]\right) ^ {\frac {1}{\tau}} \Pr (\mathcal {E}) \geq \langle x ^ {\star}, \theta^ {*} \rangle \left(1 - 1 0 0 \sqrt {\frac {d \nu}{\mathsf {T} \langle x ^ {*} , \theta^ {*} \rangle}} \log (\mathsf {T} | \mathcal {X} |)\right).
$$
Therefore, the Nash regret of LINNASH satisfies
$$
\mathrm {N R} _ {\mathsf {T}} = \langle x ^ {*}, \theta^ {*} \rangle - \left(\prod_ {t = 1} ^ {\mathsf {T}} \mathbb {E} [ \langle X _ {t}, \theta^ {*} \rangle ]\right) ^ {1 / \mathsf {T}} = O \left(\sqrt {\frac {d \nu \langle x ^ {*} , \theta^ {*} \rangle}{\mathsf {T}}} \log (\mathsf {T} | \mathcal {X} |)\right).
$$
This completes the proof sketch.
Due to space restrictions, the complete analysis is deferred to Appendix C
Computational Efficiency of LINNASH. We note that Algorithm 2 (LINNASH) executes in polynomial time. In particular, the algorithm calls the subroutine GENERATEARMSEQUENCE in Part I for computing the John Ellipsoid. Given a set of arm vectors as input, this ellipsoid computation can be performed efficiently (see Chapter 3 in [25]). In fact, for our purposes an approximate version of the John Ellipsoid suffices, and such an approximation can be found much faster [8]; specifically, in time  $O(|\mathcal{X}|^2 d)$ . Furthermore, the algorithm solves the D-optimal design problem, once in Part I and at most  $O(\log \mathsf{T})$  times in Part II. The D-optimal design is a concave maximization problem, which can be efficiently solved using, say, the Frank-Wolfe algorithm with rank-1 updates. Each iteration takes  $O(|\mathcal{X}|^2)$  time, and the total number of iterations is at most  $O(d)$  (see, e.g., Chapter 21 of [17] and Chapter 3 in [25]). Overall, we get that LINNASH is a polynomial-time algorithm.
4 Extension of Algorithm LINNASH for Infinite Arms
The regret guarantee in Theorem 1 depends logarithmically on  $|\mathcal{X}|$ . Such a dependence makes the guarantee vacuous when the set of arms  $\mathcal{X}$  is infinitely large (or even  $|\mathcal{X}| = \Omega(2^{\sqrt{\mathsf{T}}d^{-1}})$ ). To resolve this limitation, we extend LINNASH with a modified confidence width that depends only on the largest estimated reward  $\gamma \coloneqq \max_{x\in \mathcal{X}}\langle x,\widehat{\theta}\rangle$ . Specifically, we consider the confidence width  $16\sqrt{\frac{\gamma d^{\frac{5}{2}}\nu\log(\mathsf{T})}{\mathsf{T}^{\prime}}}$ , for all the arms, and select the set of surviving arms in each phase (of Part II of the algorithm for infinite arms) as follows:
$$
\widetilde {\mathcal {X}} = \left\{x \in \mathcal {X}: \langle x, \widehat {\theta} \rangle \geq \gamma - 1 6 \sqrt {\frac {\gamma d ^ {\frac {5}{2}} \nu \log (\mathrm {T})}{\mathrm {T} ^ {\prime}}} \right\} \tag {11}
$$
See Algorithm 3 in Appendix D for details. The theorem below is the main result of this section.
Theorem 2. For any given stochastic linear bandits problem with set of arms  $\mathcal{X} \subset \mathbb{R}^d$ , time horizon  $\mathsf{T} \in \mathbb{Z}_+$ , and  $\nu$ -sub-Poisson rewards, Algorithm 2 achieves Nash regret
$$
\mathrm {N R} _ {\mathsf {T}} = O \left(\beta \frac {d ^ {\frac {5}{4}} \sqrt {\nu}}{\sqrt {\mathsf {T}}} \log (\mathsf {T})\right),
$$
Proof of Theorem 2 and a detailed regret analysis of Algorithm 3 can be found in Appendix D.
Fairness and welfare considerations have emerged as a central design objectives in online decision-making contexts. Motivated, broadly, by such considerations, the current work addresses the notion of Nash regret in the linear bandits framework. We develop essentially tight Nash regret bounds for linear bandit instances with a finite number of arms.
In addition, we extend this guarantee to settings wherein the number of arms is infinite. Here, our regret bound scales as  $d^{5/4}$ , where  $d$  is the ambient dimension. Note that, for linear bandits with infinite arms, [1] obtains a bound of  $d / \sqrt{T}$  for average regret. We conjecture that a similar dependence should be possible for Nash regret as well and pose this strengthening as a relevant direction of future work. Another important direction would be to study Nash regret for more other bandit frameworks (such as contextual bandits and combinatorial bandits) and Markov Decision Processes (MDPs).
Siddharth Barman's reserach is supported by a SERB Core research grant (CRG/2021/006165).