SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Parnian Kassraie<sup>1</sup> Nicolas Emmenegger<sup>1</sup> Andreas Krause<sup>1</sup> Aldo Pacchiano<sup>2,3</sup>  
<sup>1</sup>ETH Zurich <sup>2</sup>Broad Institute of MIT and Harvard <sup>3</sup>Boston University  
{pkassraie, nikolaem, krausea}@ethz.ch pacchian@bu.edu
Model selection in the context of bandit optimization is a challenging problem, as it requires balancing exploration and exploitation not only for action selection, but also for model selection. One natural approach is to rely on online learning algorithms that treat different models as experts. Existing methods, however, scale poorly (poly  $M$ ) with the number of models  $M$  in terms of their regret. Our key insight is that, for model selection in linear bandits, we can emulate full-information feedback to the online learner with a favorable bias-variance trade-off. This allows us to develop ALEXP, which has an exponentially improved  $(\log M)$  dependence on  $M$  for its regret. ALEXP has anytime guarantees on its regret, and neither requires knowledge of the horizon  $n$ , nor relies on an initial purely exploratory stage. Our approach utilizes a novel time-uniform analysis of the Lasso, establishing a new connection between online learning and high-dimensional statistics.
When solving bandit problems or performing Bayesian optimization, we need to commit to a reward model a priori, based on which we estimate the reward function and build a policy for selecting the next action. In practice, there are many ways to model the reward by considering different feature maps or hypothesis spaces, e.g., for optimizing gene knockouts [Gonzalez et al., 2015, Pacchiano et al., 2022] or parameter estimation in biological dynamical systems [Ulmasov et al., 2016, Imani et al., 2019]. It is not known a priori which model is going to yield the most sample efficient bandit algorithm, and we can only select the right model as we gather empirical evidence. This leads us to ask, can we perform adaptive model selection, while simultaneously optimizing for reward?
In an idealized setting with no sampling limits, given a model class of size  $M$ , we could initialize  $M$  bandit algorithms (a.k.a agents) in parallel, each using one of the available reward models. Then, as the algorithms run, at every step we can select the most promising agent, according to the cumulative rewards that are obtained so far. Model selection can then be cast into an online optimization problem on a  $M$ -dimensional probability simplex, where the probability of selecting an agent is dependent on its cumulative reward, and the optimizer seeks to find the distribution with the best return in hindsight. This approach is impractical for large model classes, since at every step, it requires drawing  $M$  different samples in parallel from the environment so that the reward for each agent is realized.
In realistic applications of bandit optimization, we can only draw one sample at a time, and so we need to design an algorithm which allocates more samples to the agents that are deemed more promising. Prior work [e.g., Maillard and Munos, 2011, Agarwal et al., 2017] propose to run a single meta algorithm which interacts with the environment by first selecting an agent, and then selecting an action according to the suggestion of that agent. The online model selection problem can still be emulated in this setup, however this time the optimizer receives partial feedback, coming from only one agent. Consequently, many agents need to be queried, and the overall regret scales with poly  $M$ , again restricting this approach to small model classes. In fact, addressing the limited scope of such algorithms, Agarwal et al. [2017] raise an open problem on the feasibility of obtaining a log  $M$  dependency for the regret.
We show that this rate is achievable, in the particular case of linearly parametrizable rewards. We develop a technique to "hallucinate" the reward for every agent that was not selected, and run the online optimizer with emulated full-information feedback. This allows the optimizer to assess the quality of the agents, without ever having queried them. As a result, our algorithm ALEXP satisfies a regret of rate  $\mathcal{O}(\max \{\sqrt{n\log^3M},n^{3 / 4}\sqrt{\log M}\})$ , with high probability, simultaneously for all  $n\geq 1$  (Theorem 1). Our key idea, leading to  $\log M$  dependency, is to employ the Lasso as a low-variance online regression oracle, and estimate the reward for the agents that were not chosen. This trick is made possible through our novel time-uniform analysis of online Lasso regression (Theorem 3). Consequently, ALEXP is horizon-independent, and explores adaptively without requiring an initial exploration stage. Empirically we find that ALEXP consistently outperforms prior work across a range of environments.
Table 1: Overview of literature on online model selection for bandit optimization
MS Technique	Regret	MS Guarantee	adaptive exploration	anytime
Sparse Linear Bandits	Lasso	log M	X	X	X
MS for Black-Box Bandits	OMD with bandit feedback	polyM	✓	✓	X
MS for Linear Bandits (Ours)	EXP4 with full-information	log M	✓	✓	✓
Online Model selection (MS) for bandits considers combining a number of agents in a master algorithm, with the goal of performing as well as the best agent [Maillard and Munos, 2011, Agarwal et al., 2017, Pacchiano et al., 2020, Luo et al., 2022]. This literature operates on black-box model classes of size  $M$  and uses variants of Online Mirror Descent (OMD) to sequentially select the agents. The optimizer operates on importance-weighted estimates of the agents' rewards, which has high variance  $(\mathrm{poly}M)$  and is non-zero only for the selected agent. Effectively, the optimizer receives partial (a.k.a. bandit) feedback and agents are at risk of starvation, since at every step only the selected agent gets the new data point. These works assume knowledge of the horizon, however, we suspect that this may be lifted with a finer analysis of OMD.
Sparse linear bandits use sparsity-inducing methods, often Lasso [Tibshirani, 1996], for estimating the reward in presence of many features, as an alternative to model selection. Early results are in the data-rich regime where the stopping time  $n$  is known and larger than the number of models, and some suffer from poly  $M$  regret dependency [e.g., Abbasi-Yadkori et al., 2012]. Recent efforts often consider the contextual case, where at every step only a finite stochastic subset of the action domain is presented to the agent, and that the distribution of these points is i.i.d. and sufficiently diverse [Li et al., 2022, Bastani and Bayati, 2020, Kim and Paik, 2019, Oh et al., 2021, Cella and Pontil, 2021]. We do not rely on such assumptions. Most sparse bandit algorithms either start with a purely exploratory phase [Kim and Paik, 2019, Li et al., 2022, Hao et al., 2020, Jang et al., 2022], or rely on a priori scheduled exploration [Bastani and Bayati, 2020]. The exploration budget is set according to the horizon  $n$ . Therefore, such algorithms inherently require the knowledge of  $n$  and can be made anytime only via the doubling trick [Auer et al., 1995]. Table 2 presents an in-depth overview.
ALEXP inherits the best of both worlds (Table 1): its regret enjoys the log  $M$  dependency of sparse linear bandits even on compact domains, and it has adaptive probabilistic exploration with anytime guarantees. In contrast to prior literature, we perform model selection with an online optimizer (EXP4), which hallucinates full-information feedback using a low-variance Lasso estimator instead of importance-weighted estimates. Moreover, our anytime approach lifts the horizon dependence and the exploration requirement of sparse linear bandits.
Our work is inspired by and contributes to the field of Learning with Expert Advice, which analyzes incorporating the advice of  $M$  oblivious (non-adaptive) experts, with bandit or full-information feedback [Haussler et al., 1998, Auer et al., 2002b, McMahan and Streeter, 2009]. The idea of employing an online optimizer for learning stems from this literature, and has been used in various applications of online learning [Foster et al., 2017, Singla et al., 2018, Muthukumar et al., 2019, Karimi et al., 2021, Liu et al., 2022]. In particular, we are inspired by Foster and Rakhlin [2020] and Moradipari et al. [2022], who apply EXP4 to least squares estimates, for arm selection in  $K$ -armed
contextual bandits. However, their algorithms are not anytime, and due to the choice of estimator, the corresponding regret scales with  $\mathcal{O}(\min (\sqrt{M},\sqrt{K\log M}))$
3 Problem Setting
We consider a bandit problem where a learner interacts with the environment in rounds. At step  $t$  the learner selects an action  $\pmb{x}_t\in \mathcal{X}$ , where  $\mathcal{X}\subset \mathbb{R}^{d_0}$  is a compact domain and observes a noisy reward  $y_{t} = r(\pmb{x}_{t}) + \varepsilon_{t}$  such that  $\varepsilon_t$  is an i.i.d. zero-mean sub-Gaussian variable with parameter  $\sigma^2$ . We assume the reward function  $r:\mathcal{X}\to \mathbb{R}$  is linearly parametrizable by some unknown feature map, and that the model class  $\{\phi_j:\mathbb{R}^{d_0}\to \mathbb{R}^d,j = 1,\ldots ,M\}$  contains the set of plausible feature maps. We consider the setting where  $M$  can be very large, and while the set  $\{\phi_j\}$  may include misspecified feature maps, it contains at least one feature map that represents the reward function, i.e., there exists  $j^{\star}\in [M]$  such that  $r(\cdot) = \pmb{\theta}_{j^{\star}}^{\top}\pmb{\phi}_{j^{\star}}(\cdot)$ . We assume the image of  $\phi_j$  spans  $\mathbb{R}^d$ , and no two feature maps are linearly dependent, i.e. for any  $j,j'\in [M]$ , there exists no  $\alpha \in \mathbb{R}$  such that  $\phi_j(\cdot) = \alpha \phi_{j'}(\cdot)$ . This assumption, which is satisfied by design in practice, ensures that the features are not ill-posed and we can explore in all relevant directions. We assume that the concatenated feature map  $\phi (\pmb {x})\coloneqq (\phi_1(\pmb {x}),\dots ,\phi_M(\pmb {x}))$  is normalized  $\| \phi (\pmb {x})\| \leq 1$  for all  $\pmb {x}\in \mathcal{X}$  and that  $\| \pmb {\theta}_{j^{\star}}\| \leq B$ , which implies  $|r(x)|\leq B$  for all  $\pmb {x}\in \mathcal{X}$ .
We will model this problem in the language of model selection where a meta algorithm aims to optimize the unknown reward function by relying on a number of base learners. In order to interact with the environment the meta algorithm selects an agent that in turn selects an action. In our setting we thinking of each of these  $M$  feature maps as controlled by a base agent running its own algorithm. Base agent  $j$  uses the feature map  $\phi_j$  for modeling the reward. At step  $t$  of the bandit problem, each agent  $j$  is given access to the full history  $H_{t-1} \coloneqq \{(\pmb{x}_1, y_1), \dots, (\pmb{x}_{t-1}, y_{t-1})\}$ , and uses it to locally estimate the reward as  $\hat{\beta}_{t-1,j}^{\top} \phi_j(\cdot)$ , where  $\beta_{t-1,j} \in \mathbb{R}^d$  is the estimated coefficients vector. The agent then uses this estimate to develop its action selection policy  $p_{t,j} \in \mathcal{M}(\mathcal{X})$ . Here,  $\mathcal{M}$  denotes the space of probability measures defined on  $\mathcal{X}$ . The condition on existence of  $j^\star$  will ensure that there is at least one agent which is using a correct model for the reward, and thereby can solve the bandit problem if executed in isolation. We refer to agent  $j^\star$  as the oracle agent.
Our goal is to find a sample-efficient strategy for iterating over the agents, such that their suggested actions maximize the cumulative reward, achieved over any horizon  $n \geq 1$ . This is equivalent to minimizing the cumulative regret  $R(n) = \sum_{t=1}^{n} r(\pmb{x}^*) - r(\pmb{x}_t)$ , where  $\pmb{x}^*$  is a global maximizer of the reward function. We neither fix  $n$ , nor assume knowledge of it.
As warm-up, consider an example with deterministic agents, i.e., when  $p_{t,j}$  is a Dirac measure on a specific action  $x_{t,j}$ . Suppose it was practically feasible to draw the action suggested by every agent and observe the corresponding reward vector  $\boldsymbol{r}_t = (r_{t,j})_{j=1}^M$ . In this case, model selection becomes a full-information online optimization problem, and we can design a minimax optimal algorithm as follows. We assign a probability distribution  $\boldsymbol{q}_t = (q_{t,j})_{j=1}^M$  to the models, and update it such that the overall average return  $\sum_{t=1}^n \boldsymbol{q}_t^\top \boldsymbol{r}_t$  is competitive to the best agent's average return  $\sum_{t=1}^n r_{t,j^*}$ . At every step, we update  $q_{t+1,j} \propto \exp(\sum_{s=1}^t r_{s,j})$ , since such exponential weighting is known to lead to an optimal solution for this classic online learning problem [Cesa-Bianchi and Lugosi, 2006]. In our setting however, the agents are stochastic, and we do not have access to the full  $\boldsymbol{r}_t$  vector.
We propose the Anytime Exponential weighting algorithm based on Lasso reward estimates (ALEXP), summarized in Algorithm 1. At step  $t$  we first sample an agent  $j_{t}$ , and then sample an action  $x_{t}$  according to the agent's policy  $p_{t,j_t}$ . Let  $\Delta_M$  denote the  $M$ -dimensional probability simplex. We maintain a probability distribution  $q_{t}\in \Delta_{M}$  over the agents, and update it sequentially as we accumulate evidence on the performance of each agent. Ideally, we would have adjusted  $q_{t,j}$  according to the average return of model  $j$ , that is,  $\mathbb{E}_{\boldsymbol{x}\sim p_{t,j}}r(\boldsymbol {x})$ . However, since  $r$  is unknown, we estimate the average reward with some  $\hat{r}_{t,j}$ . We then update  $q_{t}$  for the next step via,
$$
q _ {t + 1, j} = \frac {\exp (\eta_ {t} \sum_ {s = 1} ^ {t} \hat {r} _ {s , j})}{\sum_ {i = 1} ^ {M} \exp (\eta_ {t} \sum_ {s = 1} ^ {t} \hat {r} _ {s , i})}
$$
for all  $j = 1, \dots, M$ , where  $\eta_t$  is the learning rate, and controls the sensitivity of the updates. This rule allows us to imitate the full-information example that we mentioned above. By utilizing  $\hat{r}_{t,j}$  and hallucinating feedback from all agents, we can reduce the probability of selecting a badly performing agent, without ever having sampled them (c.f. Fig. 4). It remains to design the estimator  $\hat{r}_{t,j}$ . We concatenate all feature maps, and, knowing that many features are redundant, use a sparsity inducing estimator over the resulting coefficients vector. Mainly, let  $\pmb{\theta} = (\pmb{\theta}_1, \dots, \pmb{\theta}_M) \in \mathbb{R}^{Md}$  be the concatenated coefficients vector. We then solve
$$
\hat {\boldsymbol {\theta}} _ {t} = \underset {\boldsymbol {\theta} \in \mathbb {R} ^ {M d}} {\arg \min } \mathcal {L} (\boldsymbol {\theta}; H _ {t}, \lambda_ {t}) = \underset {\boldsymbol {\theta} \in \mathbb {R} ^ {M d}} {\arg \min } \frac {1}{t} \| \boldsymbol {y} _ {t} - \Phi_ {t} \boldsymbol {\theta} \| _ {2} ^ {2} + 2 \lambda_ {t} \sum_ {j = 1} ^ {M} \| \boldsymbol {\theta} _ {j} \| _ {2} \tag {1}
$$
where  $\Phi_t = [\phi^\top(\boldsymbol{x}_s)]_{s \leq t} \in \mathbb{R}^{t \times Md}$  is the feature matrix,  $\boldsymbol{y}_t \in \mathbb{R}^t$  is the concatenated reward vector, and  $\lambda_t$  is an adaptive regularization parameter. Problem (1) is the online variant of the group Lasso [Lounici et al., 2011]. The second term is the loss is the mixed (2, 1)-norm of  $\boldsymbol{\theta}$ , which can be seen as the  $\ell_1$ -norm of the vector  $(\|\pmb{\theta}_1\|, \dots, \|\pmb{\theta}_M\|) \in \mathbb{R}^M$ . This norm induces sparsity at the group level, and therefore, the sub-vectors  $\hat{\pmb{\theta}}_{t,j} \in \mathbb{R}^d$  that correspond to redundant feature maps are expected to be  $\mathbf{0}$ , i.e. the null vector. We then estimate the average return of each model by simply taking an expectation  $\hat{r}_{t,j} = \mathbb{E}_{\boldsymbol{x} \sim p_{t+1,j}}[\hat{\pmb{\theta}}_t^\top \phi(\boldsymbol{x})]$ . This quantity is the average return of the agent's policy  $p_{t+1,j}$ , according to our Lasso estimator. In Section 5.2 we explain why the particular choice of Lasso is crucial for obtaining a log  $M$  rate for the regret.
For action selection, with probability  $\gamma_{t}$ , we sample agent  $j$  with probability  $q_{t,j}$  and draw  $\mathbf{x}_t\sim p_{t,j}$  as per suggestion of the agent. With probability  $1 - \gamma_{t}$ , we choose the action according to some exploratory distribution  $\pi \in \mathcal{M}(\mathcal{X})$  which aims to sample informative actions. This can be any design where  $\mathrm{supp}(\pi) = \mathcal{X}$ . We mix  $p_{t,j}$  with  $\pi$ , to collect sufficiently diverse data for model selection. We are not restricting the agents' policy, and therefore can not rely on them to explore adequately. In Theorem 1, we choose a decreasing sequence of  $(\gamma_t)_{t\geq 1}$  the probabilities of exploration at step  $t\geq 1$ , since less exploration will be required as data accumulates. To conclude, the action selection policy of ALEXP is formally described as the mixture
$$
p _ {t} (\boldsymbol {x}) = \gamma_ {t} \pi (\boldsymbol {x}) + (1 - \gamma_ {t}) \sum_ {j = 1} ^ {M} q _ {t, j} p _ {t, j} (\boldsymbol {x}).
$$
5 Main Results
For the regret guarantee, we consider specific choices of base agents and exploratory distribution. Our analysis may be extended to include other policies, since ALEXP can be wrapped around any bandit agent that is described by some  $p_{t,j}$ , and allows for random exploration with any distribution  $\pi$ .
Base Agents. We assume that the oracle agent has either a UCB [Abbasi-Yadkori et al., 2011] or a GREEDY [Auer et al., 2002a] policy, and all other agents are free to choose any arbitrary policy. Similar treatment can be applied to cases where the oracle uses other (sublinear) polices for solving linear or contextual bandits [e.g., Thompson, 1933, Kaufmann et al., 2012, Agarwal et al., 2014]. In either case, agent  $j^{\star}$  calculates a ridge estimate of the coefficients vector based on the history  $H_{t}$
$$
\hat {\boldsymbol {\beta}} _ {t, j ^ {\star}} := \underset {\boldsymbol {\beta} \in \mathbb {R} ^ {d}} {\arg \min } \| \boldsymbol {y} _ {t} - \Phi_ {t, j ^ {\star}} \boldsymbol {\beta} \| _ {2} ^ {2} + \tilde {\lambda} \| \boldsymbol {\beta} \| _ {2} ^ {2} = \left(\Phi_ {t, j ^ {\star}} ^ {\top} \Phi_ {t, j ^ {\star}} + \tilde {\lambda} \boldsymbol {I}\right) ^ {- 1} \Phi_ {t, j ^ {\star}} ^ {\top} \boldsymbol {y} _ {t}.
$$
Here  $\Phi_{t,j^{\star}}\in \mathbb{R}^{t\times d}$  is the feature matrix, where each row  $s$  is  $\phi_{j^{\star}}^{\top}(\pmb{x}_s)$  and  $\tilde{\lambda}$  is the regularization constant. Then at step  $t$ , a GreEDy oracle suggests the action which maximizes the reward estimate  $\hat{\beta}_{t - 1,j^{\star}}^{\top}\phi_{j^{\star}}(\pmb{x})$ , and a UCB oracle queries  $\arg \max u_{t - 1,j^{\star}}(\pmb{x})$  where  $u_{t - 1,j^{\star}}(\cdot)$  is the upper confidence bound that this agent calculates for  $r(\cdot)$ . Proposition 21 shows that the sequence  $(u_{t,j^{\star}})_{t\geq 1}$  is in fact an anytime valid upper bound for  $r$  over the entire domain.
Exploratory policy. Performance of ALEXP depends on the quality of the samples that  $\pi$  suggests. The eigenvalues of the covariance matrix  $\Sigma (\pi ,\phi)\coloneqq \mathbb{E}_{\boldsymbol {x}\sim \pi}\phi (\boldsymbol {x})\phi^{\top}(\boldsymbol {x})$  reflect how diverse the data is, and thus are a good indicator for data quality. van de Geer and Buhlmann [2009] present a survey on the notions of diversity defined based on these eigenvalues. Let  $\lambda_{\mathrm{min}}(A)$  denote the minimum eigenvalue of a matrix  $A$ . Similar to Hao et al. [2020], we assume that  $\pi$  is the maximizer of the problem below and present our regret bounds in terms of
$$
C _ {\min } = C _ {\min } (\mathcal {X}, \phi) := \max  _ {\pi \in \mathcal {M} (\mathcal {X})} \lambda_ {\min } (\Sigma (\pi , \phi)), \tag {2}
$$
Algorithm 1 ALEXP
Inputs:  $\pi, (\gamma_t, \eta_t, \lambda_t)$  for  $t \geq 1$
Let  $q_{1}\gets \mathrm{Unif}(M)$  and initialize base agents  $(p_{1,1},\ldots ,p_{1,M})$
for  $t\geq 1$  do
Draw  $\alpha_{t}\sim$  Bernoulli  $(\gamma_{t})$
$\triangleright$  Decide to explore or exploit
if  $\alpha_{t} = 1$  then
Choose action  $x_{t}$  randomly according to  $\pi$
Draw  $j_{t}\sim \pmb{q}_{t}$
$\triangleright$  Select an agent
Draw  $\boldsymbol{x}_t \sim p_{t,j_t}$ .
$\triangleright$  Select the action suggested by the agent
Observe  $y_{t} = r(\pmb{x}_{t}) + \varepsilon_{t}$
Receive reward
[ H_{t} = H_{t - 1}\cup \{(\pmb{x}_{t},y_{t})\} . ]
$\triangleright$  Append history
$\pmb{\theta}_{t}\gets \arg \min \mathcal{L}(\pmb {\theta};H_{t},\lambda_{t})$
> Update the parameter estimate
Report  $H_{t}$  to all agents, and get updated policies  $p_{t + 1,1},\ldots ,p_{t + 1,M}$
$\triangleright$  Update agents
Update estimated average return of every agent
$$
\hat {r} _ {t, j} \leftarrow \mathbb {E} _ {\boldsymbol {x} \sim p _ {t + 1, j}} \left[ \hat {\boldsymbol {\theta}} _ {t} ^ {\top} \phi (\boldsymbol {x}) \right], \quad j = 1, \dots , M
$$
Update agent probabilities
$$
q _ {t + 1, j} \gets \frac {\exp (\eta_ {t} \sum_ {s = 1} ^ {t} \hat {r} _ {s , j})}{\sum_ {i = 1} ^ {M} \exp (\eta_ {t} \sum_ {s = 1} ^ {t} \hat {r} _ {s , i})}
$$
which is greater than zero under the conditions specified in our problem setting. Prior works in the sparse bandit literature all require a similar or stronger assumption of this kind, and Table 2 gives an overview. Alternatively, one can work with an arbitrary  $\pi$ , e.g.,  $\mathrm{Unif}(\mathcal{X})$ , as long as  $\lambda_{\min}(\Sigma(\pi, \phi))$  is bounded away from zero. Appendix C.1 reviews some configurations of  $(\phi, \mathcal{X}, \pi)$  that lead to a nonzero minimum eigenvalue, and Corollary 12 bounds the regret of ALEXP with uniform exploration.
For this choice of agents and exploratory distribution, Theorem 1 presents an informal regret bound. Here, we have used the  $\mathcal{O}$  notation, and only included the fastest growing terms. The inequality is made exact in Theorem 14, up to constant multiplicative factors.
Theorem 1 (Cumulative Regret of ALEXP, Informal). Let  $\delta \in (0,1]$  and set  $\pi$  to be the maximizer of (2). Choose learning rate  $\eta_t = \mathcal{O}(C_{\min}t^{-1/2} / C(M,\delta,d))$ , exploration probability  $\gamma_t = \mathcal{O}(t^{-1/4})$  and Lasso regularization parameter  $\lambda_t = \mathcal{O}(C_{\min}t^{-1/2}C(M,\delta,d))$ , where
$$
C (M, \delta , d) = \mathcal {O} \left(\sqrt {1 + \log (M / \delta) + (\log \log d) _ {+} + \sqrt {d (\log (M / \delta) + (\log \log d) _ {+})}}\right).
$$
Then ALEXP satisfies the cumulative regret
$$
\begin{array}{l} R (n) = \mathcal {O} \left(n ^ {3 / 4} B + n ^ {3 / 4} C (M, \delta , d) + C _ {\min } ^ {- 1} \sqrt {n} C (M, \delta , d) \log M \right. \\ \left. + C _ {\min } ^ {- 1 / 2} n ^ {5 / 8} \sqrt {d \log (n) + \log (1 / \delta)}\right) \\ \end{array}
$$
simultaneously for all  $n \geq 1$ , with probability greater than  $1 - \delta$ .
In this bound, the first term is the regret incurred at exploratory steps (when  $\alpha_{t} = 1$ ), the second term is due to the estimation error of Lasso (i.e.,  $||\pmb{\theta} - \hat{\pmb{\theta}}_t||$ ), and the third term is the regret of the exponential weights sub-algorithm. The fourth term, is the regret bound for the oracle agent  $j^{\star}$ , when run within the ALEXP framework. It does not depend on the agent's policy (greedy or optimistic), and is worse than the minimax optimal rate of  $\sqrt{nd\log n}$ . This is because the oracle is suggesting actions based on the history  $H_{t}$ , which includes uninformative action-reward pairs queried by other, potentially misspecified, agents. In Corollary 12, we provide a regret bound independent of  $C_{\mathrm{min}}$  for orthogonal feature maps, and show that the same  $\mathcal{O}(\max \{\sqrt{n\log^3M},n^{3 / 4}\sqrt{\log M}\})$  rate may be achieved even with the simple choice  $\pi = \operatorname {Unif}(\mathcal{X})$ .
5.1 Proof Sketch