SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The LWS Algorithm. Recall that we want LWS at each batch  $m$  to select  $d$  vectors  $\{a_i\} \subseteq \mathcal{A}'$  such that, by playing each  $n_m(i)$  times, we can create a least-squares estimate  $\theta_{m+1}$  of  $\theta_*$  that allows an accurate estimate of the product  $\langle a, \theta_\star \rangle$  for all  $a \in \mathcal{A}$ . It is well-known (see [24]) that the error in estimating  $\langle a, \theta_\star \rangle$  is proportional to  $\|a\|_{\mathbf{V}_m^{-1}}$ , where  $\mathbf{V}_m = \mathbf{I} + \sum_{i=1}^{T_m} a_i a_i^\top$  is the least squares matrix we used to estimate  $\theta_{m+1}$ . Thus, essentially we want LWS to select  $d$  vectors  $\{a_i\}$  that maintain a small  $\|a\|_{\mathbf{V}_m^{-1}}$  for all actions  $a \in \mathcal{A}$ . We can do so using what is called a G-optimal design [23].
Definition 2. (G-optimal design) For any set  $\mathcal{A} \subseteq \mathbb{R}^d$ , a subset  $S \subseteq \mathcal{A}$ , together with a distribution  $\pi$  over  $\mathcal{S}$  is said to be a  $C$ -approximate optimal design for  $\mathcal{A}$  if for any  $a \in \mathcal{A}$
$$
\left\| a \right\| _ {\mathbf {V} _ {\pi} ^ {- 1}} ^ {2} \leq C d, \tag {5}
$$
where  $\mathbf{V}_{\pi} = \sum_{a_i\in S}\pi (i)a_ia_i^{\top 4}$ . When  $C = 1$  this is referred to as a G-optimal design.
Notice that if we were to play each action  $a_{i}$  for  $n\pi (i)$  times, then  $\mathbf{V}_{\pi}$  would be (approximately) a normalized least squares matrix since  $\mathbf{V}_{\pi} + \mathbf{I} / n = \mathbf{V} / n$ , and hence,  $\| a\|_{\mathbf{V}^{-1}}^2\leq \| a\|_{\mathbf{V}_{\pi}^{-1}}^2 /n$
It is well-known that for any compact set, there exists a 1-approximate optimal design [23] with  $|S| = d$ . However, computing an 1-approximate optimal design is NP-hard in general [14, 35], even for small action sets. Computing a 2-approximate optimal design can be done in polynomial time [38], but the complexity scales linearly with the size of the action set. Instead, we adopt an approach introduced in [7], which efficiently constructs an  $O(\sqrt{d})$ -approximate optimal design using only a linear optimization oracle. This relies on the concept of a barycentric spanner, which we define next.
Definition 3. (Barycentric spanner) For any set  $\mathcal{A} \subseteq \mathbb{R}^d$ , a subset  $\mathcal{S} = \{a_1, \dots, a_d\} \subseteq \mathcal{A}$  is said to be a  $C$ -approximate barycentric spanner for  $\mathcal{A}$  if any  $a \in \mathcal{A}$  can be expressed as a linear combination of vectors in  $\mathcal{S}$  with coefficients in  $[-C, C]$ .
It is easy to see that a  $C$ -approximate barycentric spanner together with a uniform distribution  $\pi(i) = 1 / d$  results in a  $C\sqrt{d}$ -approximate optimal design [19, 41]. And importantly, a  $C$ -approximate barycentric spanner for a set  $\mathcal{A}$  can be constructed using at most  $O(d^2 \log_C d)$  calls of a linear optimization oracle over the set  $\mathcal{A}$  [19].
However, this is still not sufficient for us. Even though we can efficiently construct a  $C\sqrt{d}$ -approximate optimal design for  $\mathcal{A}$ , we do not want to pull these arms according to a uniform distribution; we want to pull action  $a_i$  with estimated gap  $\Delta_m(a_i)$  for  $n_m(i) = \lceil \pi (i)T_m / (1 + \sqrt{T_{m - 1}}\Delta_m(a) / (8\gamma))^2 \rceil$  times to control the regret (which can be thought of as using a weighted distribution). But if we do not use the uniform distribution, the resulting least squares matrix  $\mathbf{V}_m$  may not satisfy that  $\| a\|_{\mathbf{V}_m^{-1}}$  is sufficiently small for all actions  $a$ .
To account for this, instead of finding a  $C$ -approximate barycentric spanner for the set  $\mathcal{A}$ , at each batch  $m$  we consider a virtual action set  $\tilde{\mathcal{A}}_m$ , which we define as
$$
\tilde {\mathcal {A}} _ {m} = \left\{\phi_ {m} (a) \mid a \in \mathcal {A} \right\}, \phi_ {m} (a) = \frac {a}{1 + \eta_ {m} \Delta_ {m} (a)}, \tag {6}
$$
where  $\eta_{m} = \sqrt{T_{m - 1}} /(8\gamma)$  and find actions  $\{a_1,\dots ,a_d\} \in \mathcal{A}$  such that  $\{\phi_m(a_i)\}_{i = 1}^d$  forms a  $C$ -approximate barycentric spanner for  $\tilde{\mathcal{A}}_m$ . The least squares matrix at batch  $m$  can be bounded as
$$
\mathbf {V} _ {m} = \mathbf {I} + \sum_ {i = 0} ^ {d} \tilde {n} _ {m} (i) a _ {i} a _ {i} ^ {\top} \geq \sum_ {i = 1} ^ {d} \frac {\pi (i) T _ {m} / 8}{(1 + \eta_ {m} \Delta_ {m} (a _ {i})) ^ {2}} a _ {i} a _ {i} ^ {\top} = \sum_ {i = 1} ^ {d} \pi (i) \frac {T _ {m}}{8} \phi_ {m} (a _ {i}) \phi_ {m} (a _ {i}) ^ {\top} \tag {7}
$$
with high probability  $\bar{0}$ , where  $\tilde{n}_m(i)$  is the number of times action  $a_i$  is played in batch  $m$  and  $a_0 = a_m^{\star}$ . That is, playing actions  $\{a_1, \dots, a_d\} \in \mathcal{A}$  for  $n_m(i)$  times each, can equivalently
3Adding reward samples from the estimated best action  $a_{m}^{\star}$  can only improve the least squares estimator.
This summation assumes finiteness of the set  $S$  which suffices for our application.
The technique of inverse gap weighting was employed in [2, 41], albeit with a different weighting approach using inverse gap instead of squared inverse gap, as utilized in our proposed schemes.
<sup>6</sup>We show in the proof of Theorem [1] that  $\sum_{i=1}^{d} n_m(i) \leq T_m \forall m \in [M]$  with high probability, hence, all the required  $n_m(i)$  action pulls can be finished within the batch.
Algorithm 2 Linear Weighted Spanner (LWS) Algorithm
1: Input: set of actions  $\mathcal{A}$ , parameter  $\eta$ , estimated best action  $\hat{a}$ , estimated parameter  $\hat{\theta}$ .  
2: Initialize:  $\tilde{a}_i = e_i$ , where  $e_i$  is the  $i$ -th basis vector of dimension  $d$ . Let  $\mathbf{A} = [\tilde{a}_1, \dots, \tilde{a}_d]$ .  
3: Let  $C = \exp(1)$ ,  $\Delta(a) = \langle \hat{a} - a, \hat{\theta} \rangle$ ,  $\phi(a) = a / (1 + \eta \Delta(a))$ .  
4: for  $i = 1, \dots, d$  do  
5: Find  $\theta$  with  $\langle \theta, \tilde{a} \rangle = \det(\tilde{a}, \mathbf{A}_{-i})$ ,  $\forall \tilde{a} \in \mathbb{R}^d$ .  
6:  $a^+ = \mathrm{LW - ArgMax}(\mathcal{A}; \frac{\theta}{\|\theta\|_2}, \eta, \hat{a}, \hat{\theta})$ ,  $a^- = \mathrm{LW - ArgMax}(\mathcal{A}; \frac{\theta}{\|\theta\|_2}, \eta, \hat{a}, -\hat{\theta})$ .  
7:  $a_i = \arg \max_{b \in \{a^+, a^-\}} |\langle \phi(b), \theta \rangle|$ ,  $\tilde{a}_i = \phi(a_i)$ .  
8:  
9: for  $i = 1, \dots, d$  do  
10: Find  $\theta$  with  $\langle \theta, \tilde{a} \rangle = \det(\tilde{a}, \mathbf{A}_{-i})$ ,  $\forall \tilde{a} \in \mathbb{R}^d$ .  
11:  $a^+ = \mathrm{LW - ArgMax}(\mathcal{A}; \frac{\theta}{\|\theta\|_2}, \eta, \hat{a}, \hat{\theta})$ ,  $a^- = \mathrm{LW - ArgMax}(\mathcal{A}; \frac{\theta}{\|\theta\|_2}, \eta, \hat{a}, -\hat{\theta})$ .  
12:  $a = \arg \max_{b \in \{a^+, a^-\}} |\langle \phi(b), \theta \rangle|$ .  
13: if  $|\det((\phi(a), \mathbf{A}_{-i}))| \geq C|det(\mathbf{A})|$  then  
14:  $a_i = a, \tilde{a}_i = \phi(a)$ .  
15: go to line 8.  
16: Return:  $a_1, \dots, a_d$ .
be thought of as playing actions  $\{\phi(a_1), \dots, \phi(a_d)\} \in \tilde{\mathcal{A}}_m$  for  $\pi(i)T_m$  times each; and since  $\{\phi(a_1), \dots, \phi(a_d)\}$  form an approximate optimal design (through a barycentric spanner) for the set  $\tilde{\mathcal{A}}_m$ , the resulting least squares matrix will lead to small  $\| \phi_m(a) \|_{\mathbf{V}_m^{-1}}$  values. In our proofs we show that a small enough  $\| \phi_m(a) \|_{\mathbf{V}_m^{-1}}$  implies  $\| a \|_{\mathbf{V}_m^{-1}} = O(\Delta_a)$  as a result of the scaling in  $\phi(a)$ . We prove in Lemma 5 in App. ① that this allows to estimate  $\Delta_a$  within a constant factor, which is all we need.
Intuitively, the virtual set  $\tilde{\mathcal{A}}_m$  weighs the actions inversely proportional to the estimated gap  $\Delta_m(a)$  and batch length  $\sqrt{T_{m - 1}}$ : the larger the gap and  $T_{m - 1}$ , the smaller magnitude the corresponding action has; this implements a form of soft elimination (shaping) of the action set, where the actions closest to the optimal become increasingly dominant as the batch length increases while the remaining fade out to zero. As a result, as  $m$  increases, the span of the optimal design focuses on the space where actions have small gaps, allowing to better distinguish among them.
To complete SoftBach (Algorithm 1), one last step is missing. LWS (Algorithm 2) follows standard steps (in Algorithm 2, see [7] for detailed explanation) to calculate the C-approximate barycentric spanner for  $\tilde{\mathcal{A}}_m$ . But to follow these steps, it requires the ability to solve the non-linear optimization problem  $\sup_{a \in A} \langle \phi_m(a), \theta \rangle$ , since  $\phi_m(a) = a / (1 + \sqrt{T_{m-1}} \Delta_m(a) / (8\gamma))$  is nonlinear in  $a$ . To do so, we will use an approximate oracle with multiplicative error, that we term LW-ArgMax and describe next.
LW-ArgMax Algorithm. LW-ArgMax (Algorithm 3) constructs an approximate oracle with  $(1 - \alpha)$ -multiplicative error for the optimization  $\sup_{a\in \mathcal{A}}\langle \phi_m(a),\theta \rangle$ . This is sufficient: we show in Lemma 2 that Algorithm 2 can use LW-ArgMax to compute a  $C / \alpha$ -approximate barycentric spanner for  $\tilde{\mathcal{A}}_m$ .
Recall that, before providing the action set  $\mathcal{A}$  to Algorithm 2, SoftBatch extends to  $\mathcal{A}' = \mathcal{A} \cup \mathcal{B}_{1/T}$ . This guarantees that:  $\mathcal{A}'$  spans  $\mathbb{R}^d$  (required to find a barycentric spanner [7]), and  $\sup_{a \in \mathcal{A}'} \langle a, \theta \rangle \geq 1/T$  for all  $\theta$  with  $\| \theta \|_2 = 1$  which implies that any approximate optimization oracle with additive error less than  $1/(2T)$  has multiplicative error of at most  $1/2$ . The extension of the set  $\mathcal{A}$  results in the barycentric spanner possibly containing points not in  $\mathcal{A}$ . However, we show that removing these points only affects  $\sup_{a \in \mathcal{A}} \| a \|_{\mathbf{V}^{-1}}$  by a constant factor, since  $\mathcal{B}_{1/T}$  has a small radius. Extending the
${}^{7}$  A related problem was faced in [41],but with a different function hence, the resulting strategy does not apply in our case. Both [41] and our solution use the standard idea of line search, albeit with different steps and different number of iterations. The proof that our line search provides an approximate optimization oracle turns out to be much more involved than that of [41].
The linear optimization problem  $\max_{a\in \mathcal{A}'}\langle a,\theta \rangle$  can be solved by comparing  $\max_{a\in \mathcal{A}}\langle a,\theta \rangle$  and  $\max_{a\in \mathcal{B}_{1 / T}}\langle a,\theta \rangle = 1 / T$
Algorithm 3 LW-ArgMax Algorithm
1: Input: set of actions  $\mathcal{A}$ ,  $\theta \in S_1$ , parameter  $\eta$ , estimated best action  $\hat{a}$ , estimate  $\hat{\theta}$ , horizon T.
2: Let  $\Delta(a) = \langle \hat{a} - a, \hat{\theta} \rangle$ ,  $\phi(a) = a / (1 + \eta \Delta(a))$ .
3: Let  $W = 3\log T$ ,  $N = 36W\log^2 (T)$ ,  $s = 1 - 1 / 6\log T$ ,  $\epsilon' = (1 - \exp (-1)) / (12T^{7 + 12\log T})$ .
4: Initialize  $z = 2^W$ .
5: for  $i = 1, \dots, N + 1$  do
6:  $\tilde{\theta} = (1 + 1 / W)z\theta + z^{1 + 1 / W}\eta \hat{\theta}$
7:  $a_{i} = \mathcal{O}_{\epsilon^{\prime}}^{+}(\mathcal{A};\tilde{\theta} /\| \tilde{\theta}\|_{2}).$
8:  $z\gets zs$
9: Return: arg max  $a_{\in \{a_i\}_{i = 1}^N}\langle \phi (a),\theta \rangle$
set  $\mathcal{A}$  to  $\mathcal{A}'$  also handles the case where the span of  $\mathcal{A}$  is smaller than  $\mathbb{R}^d$ , that was typically handled in literature by constructing a basis of  $\mathcal{A}$  which can be complicated for some sets.
LW-ArgMax then builds on the following observation (proved as part of the proof of Lemma I):
$$
\arg \max  _ {a \in \mathcal {A} ^ {\prime}} \langle \phi (a), \theta \rangle (\langle a, \theta \rangle) ^ {1 / W} = \arg \max  _ {a \in \mathcal {A} ^ {\prime}} \sup  _ {z > 0} L _ {z} (a), \tag {8}
$$
where  $L_{z}(a) = z\cdot (1 + 1 / W)\cdot \langle a,\theta \rangle -z^{1 + 1 / W}(1 + \eta \Delta (a))$  and  $\Delta (a) = \sup_{b\in \mathcal{A}}\langle b - a,\theta_{\star}\rangle \forall a\in \mathcal{A}^{\prime}$
By choosing  $W$  to be large enough, the left hand side of (8) becomes a good approximation for  $\langle \phi(a), \theta \rangle$ . For a fixed  $z$ , the supremum on the right hand side of (8) reduces to a linear optimization over the set  $\mathcal{A}'$  (that we solve using an approximate linear optimization oracle). Although the optimal value of  $z$  is not known, it can be bounded (see equation (25) in App. B); thus LW-ArgMax scans between upper and lower bounds on the optimal  $z$  with a constant multiplicative step. The pseudo-code is provided in Algorithm 3
3.3 Proof Outline for Theorem 1
We start by proving that LW-ArgMax is an approximate linear optimization oracle for the set  $\tilde{\mathcal{A}}$  with  $1 - \exp(-3)$  multiplicative error. The result is stated in Lemma 1 and proved in App. B.
Lemma 1. Let  $T \geq 3, \eta \in \mathbb{R}, \hat{a} \in \mathbb{R}^d, \hat{\theta} \in \mathcal{B}_T$  be given parameters, and  $\mathcal{A}$  be a given set. Let  $\Delta(a), \phi(a)$  denote  $\Delta(a) = \langle \hat{a} - a, \hat{\theta} \rangle$ ,  $\phi(a) = a / (1 + \eta \Delta(a))$ . If  $\mathcal{B}_{1/T} \subseteq \mathcal{A} \subseteq \mathcal{B}_1, |\eta| \leq T$  and  $1/2 \leq 1 + \eta \Delta(a) \leq T^2$ ,  $\forall a \in \mathcal{A}$ , then for any  $\theta \in S_1$ , LW-ArgMax outputs an element  $a \in \mathcal{A}$  such that
$$
\langle \phi (a), \theta \rangle \geq \exp (- 3) \sup  _ {b \in \left\{\phi \left(b ^ {\prime}\right) \mid b ^ {\prime} \in \mathcal {A} \right\}} \langle b, \theta \rangle . \tag {9}
$$
The conditions of Lemma  $\boxed{1}$  are easy to verify for all batches  $m$ ; namely,  $\mathcal{B}_{1 / T} \subseteq \mathcal{A} \subseteq \mathcal{B}_1$  holds as we extend the set of actions by adding  $\mathcal{B}_{1 / T}$  before feeding it into Algorithm  $\boxed{3}$  and the condition  $1 / 2 \leq 1 + \eta \Delta(a) \leq T^2$ ,  $\forall a \in \mathcal{A}$  is proved in Theorem  $\boxed{1}$  for all the inputs fed into Algorithm  $\boxed{3}$ .
Given the result of Lemma  $\boxed{1}$ , we next show that Algorithm  $\boxed{2}$  finds a  $C / \alpha$ -approximate barycentric spanner of the set  $\tilde{\mathcal{A}}_m$ ,  $\forall m \in [M]$ . This is done by slightly adapting the proof of Proposition 2.5 in  $\boxed{7}$  to work with approximate linear optimization oracles instead of exact oracles. The result is stated in the following theorem and the proof is provided in App.  $\boxed{C}$  for completeness.
Lemma 2. Let  $\eta \in \mathbb{R},\hat{a}\in \mathbb{R}^d,\hat{\theta}\in \mathbb{R}^d$  be given parameters, and  $\mathcal{A}$  be a given set. Let  $\Delta (a),\phi (a)$  denote  $\Delta (a) = \langle \hat{a} -a,\hat{\theta}\rangle ,\phi (a) = a / (1 + \eta \Delta (a))$  . Suppose that  $\langle \phi (LW\text{-ArgMax} (\theta)),\theta \rangle \geq$ $\alpha \sup_{a\in \mathcal{A}}\langle \phi (a),\theta \rangle$  then Algorithm computes a  $C / \alpha$  -approximate barycentric spanner for the set  $\tilde{\mathcal{A}} = \{\phi (a)|a\in \mathcal{A}\}$  with at most  $O(d^{2}\log_{C}(d / \alpha))$  calls to LW-ArgMax.
To build our regret bounds, we essentially prove that a number of pulls of  $\propto \frac{\pi(i)}{\Delta_m(a_i)^2}$  for action  $a_{i}$  enables to estimate the gap  $\Delta_m(a_i)$  within a constant factor of the real gap  $\Delta_{a_i}$ . To do so, we start by providing an error bound for estimating  $\langle \phi_m(a),\theta_\star\rangle$  using standard sub-Gaussian concentration inequalities. Then, through mathematical induction, we extend this bound to the error of the action mean estimates  $\langle a,\theta_m\rangle$ . Intuitively, if we believe that  $\langle a,\theta_m\rangle$  is a good estimate of  $\langle a,\theta_\star\rangle$  for all
actions, which implies  $\Delta_m(a)$  is a good estimate of  $\Delta_a$  at batch  $m$ , then even though the scale in  $\phi_{m + 1}$  by  $\Delta_m(a)$ , this property will continue to hold at batch  $m + 1$ . The constants multiplying  $\Delta_m(a)$  in  $\phi_m$  are carefully designed to enable this. Finally, we show that the inverse squared gap weighting of the distribution enables to tightly upper bound the regret.
4 Algorithm for Contextual Linear Bandits
Our algorithm for contextual linear bandits is based on a technique proposed in [18], which reduces the contextual linear to a linear bandit setting. However, we cannot directly apply the reduction from [18] in Algorithm 1 as the reduction is not necessarily computationally efficient. Instead, we build a new algorithm (see Algorithm 4 in App. G) that incorporates reduction steps from [18] within Algorithm 1. One challenge we encounter is the introduction of a large, non-convex action set through the reduction process. To address this, we construct a linear optimization oracle over the new action set in order to ensure the efficiency of Algorithm 4. Additionally, the reduction requires estimating the expected value of a function (explained next and in App. G), and we carefully design the batch lengths to perform this estimation effectively. The following theorem describes our main result.
Theorem 3. Consider a contextual linear bandit instance with  $\mathcal{A}_t$  generated from an unknown distribution  $\mathcal{D}$ . There exists a universal constant  $C$  and choice for batch lengths such that Algorithm 4 finishes in  $O(\log \log T)$  batches with regret upper bounded as
$$
R _ {T} \leq C \gamma \sqrt {T} \log \log T
$$
with probability at least  $1 - \delta$ , where  $\gamma = 10\sqrt{C_Ld(\log(16M / \delta) + 57d\log^2(6T))}$ . Moreover, the running time and space complexity are  $\tilde{O}(d^4 + \mathcal{T}_{opt}d^3 T)$ ,  $\tilde{O}(d^2 + \mathcal{M}_{opt})$  respectively.
We next briefly review the reduction and refer the reader to [18] for a detailed description. The basic idea in [18] is to establish a linear bandit action for each possible parameter vector  $\theta$  of the contextual bandit instance. This is achieved through the use of the function  $g: \mathbb{R}^d \to \mathbb{R}^d$ , which computes the expected best action under the context distribution  $\mathcal{D}$  with respect to the parameter  $\theta: g(\theta) = \mathbb{E}_{\mathcal{A} \sim \mathcal{D}}[\mathcal{O}(\mathcal{A}; \theta)]$ , where  $\mathcal{O}$  is an optimization oracle (see Definition 1). A key insight, as stated in Theorem 1 of [18], is that if  $a_t = \mathcal{O}(\mathcal{A}_t; \theta_t)$  for some  $\theta_t \in \mathbb{R}^d$ , then the reward generated by the contextual bandit instance can be expressed as  $r_t = \langle g(\theta_t), \theta_\star \rangle + \eta_t'$ , where  $\eta_t'$  is noise with zero mean conditioned on the history. Consequently, the reward can be viewed as generated by pulling action  $g(\theta_t)$  in a linear bandit instance with an action set  $\mathcal{X} = \{g(\theta)|\theta \in \Theta\}$ . Moreover, the same theorem demonstrates that if a linear bandit algorithm is employed to choose  $g(\theta_t) \in \mathcal{X}$  at round  $t$  and thus play action  $a_t = \mathcal{O}(\mathcal{A}_t; \theta_t)$ , then  $|R_T - R_T^L| = \tilde{O}(\sqrt{T})$  with high probability, where  $R_T^L = \sum_{t=1}^{T} \sup_{\theta \in \Theta} \langle g(\theta) - g(\theta_t), \theta_\star \rangle$  is the regret of the algorithm on the linear bandit instance.
To estimate  $g$ , which depends on the unknown context distribution  $\mathcal{D}$ , [18] proposes using contexts observed in previous batches. Specifically, the function  $g$  is replaced by  $g^{(m + 1)}(\theta) = \frac{1}{|H_m|}\sum_{t\in H_m}\mathcal{O}(\mathcal{A}_t;\theta)$  for all  $\theta \in \Theta'$ , where  $H_{m}$  is the set of indices for rounds in batch  $m$  and  $\Theta' = [\theta ]_q|\theta \in \Theta$  is a discretization of  $\Theta$ ,  $[\theta ]_q = q\lfloor \theta \sqrt{d} /q\rfloor /\sqrt{d}$  and  $q$  is the discretization parameter. The action set at batch  $m$  is correspondingly modified as  $\mathcal{X}_m = \{g^{(m)}(\theta)|\theta \in \Theta '\}$ . It is also shown in [18] that  $g^{(m)}$  is a good estimate of  $g$  for all  $\theta \in \Theta^{\prime}$  with high probability.
To leverage this reduction, we modify Algorithm  $\boxed{1}$  by adapting the action set in each batch based on the estimate of  $g$ , i.e., the set  $\mathcal{X}_m$  (note that we do not need to explicitly calculate the sets  $\mathcal{X}_m$ ). However, an issue arises where the estimate of  $\theta_{\star}$  at batch  $m$  depends on the approximate optimal design from batch  $m - 1$ , which employs the action set  $\mathcal{X}_{m - 1}$  estimated from the contexts of batch  $m - 2$ . In the proof of Theorem  $\boxed{3}$ , we demonstrate that this leads to regret proportional to  $T_{m} / \sqrt{T_{m - 2}}$ . If the batch lengths grow rapidly, significant regret may occur. To mitigate this, we reduce the growth rate of batch lengths by allowing them to increase only when  $m$  is odd (a similar technique was employed in [18]). The pseudo-code is in Algorithm  $\boxed{4}$
To implement Algorithm4 efficiently we need: (i) an approximate linear optimization oracle for the set  $\mathcal{X}_m$  with additive error at most  $\epsilon = (1 - \exp(-1)) / (12T^{7 + 12\log T})$ : we show in Lemma8 in App. F that  $g^{(m)}([\theta]_q)$  can be used as our approximate oracle for  $q \leq \epsilon/2$ ; and (ii) an inverse of the function  $g^{(m)}$  to find  $\theta_t$  associated with  $g^{(m)}(\theta_t)$  to play the action  $a_t = \mathcal{O}(\mathcal{A}_t; \theta_t)$ : we observe
that all actions played by our algorithms (Algorithm 1 and 4) are the output of the approximate optimization oracle for some  $\theta$ ; namely, for Algorithm 4 any pulled action is of the form  $g^{(m)}([\theta]_q)$  for some input to the approximate oracle  $\theta$ . Hence, the inversion of  $g^{(m)}$  for the actions pulled by Algorithm 4 can be performed by storing  $[\theta]_q$  whenever the action  $g^{(m)}([\theta]_q)$  is stored. This increases both the space and time complexity only by a constant factor.
Gap-dependent regret bounds for contextual linear bandits. The main difficulty in extending the gap-dependent regret bounds in Theorem  $\mathbb{I}$  to the contextual case is that a large minimum action gap in the original action sets  $\mathcal{A}_t$  does not imply a large gap in the reduced action set  $\mathcal{X}$ . As a simple example consider  $d = 1$ ,  $\theta_{\star} = 1$ , and two action sets  $\mathcal{A}_1 = \{-1,1\}$ , and  $\mathcal{A}_2 = \{-1\}$ . At each iteration the learner receives the action set  $\mathcal{A}_1$  with probability  $p$  and  $\mathcal{A}_2$  with probability  $1 - p$  independently from other iterations. Recall that the action set in the reduced instance  $\mathcal{X} = \{g(\theta)|\theta \in [-1,1]\}$ , where  $g(\theta) = \mathbb{E}_{\mathcal{A} \sim \mathcal{D}}[\arg \max_{a \in \mathcal{A}}\langle a,\theta\rangle]$ . For  $\theta \geq 0$  we have that  $g(\theta) = p(1) + (1 - p)(-1) = 2p - 1$ , while for  $\theta < 0$  we have  $g(\theta) = p(-1) + (1 - p)(-1) = -1$ . Then  $\mathcal{X} = \{-1,2p - 1\}$ . Therefore, the suboptimality gap is  $\Delta_{\min} = (2p - 1)(1) - (-1)(1) = 2p$  which can be arbitrarily small depending on  $p$ . Note that in the original contextual bandit instance, the minimum gap is at least 2 for both action sets.
While it may be possible to provide gap dependent regret bounds for our algorithm in the contextual case, this will require more sophisticated regret analysis that does not only rely on the reduced linear bandit instance.
Numerical results. In App. [1] we provide a numerical example to compare the computational complexity of computing the exploration policy of our algorithm versus the complexity of computing the policy in [40].