SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2 compares recent work on sparse linear bandits based on a number of important factors. In this table, the ETC algorithms follow the general format of exploring, performing parameter estimation once at  $t = n_0$ , and then repeatedly suggesting the same action which maximizes  $\hat{\theta}_{n_0}^\top \phi(\boldsymbol{x})$ . Explore-then-( $\epsilon$ )Greedy takes a similar approach, however it does not settle on  $\hat{\theta}_{n_0}$ , rather it continues to update the parameter estimate and select  $\boldsymbol{x}_t = \arg \max \hat{\theta}_t^\top \phi(\boldsymbol{x})$ . The UCB algorithms iteratively update upper confidence bound, and choose actions which maximize them. The regret bounds in Table 2 are simplified to the terms with largest rate of growth, the reader should check the corresponding papers for rigorous results. Some of the mentioned bounds depend on problem-dependent parameters (e.g.  $c_K$ ), which may not be treated as absolute constants and have complicated forms. To indicate such parameters we use  $\tau$  in Table 2, following the notation of Hao et al. [2020]. Note that  $\tau$  varies across the rows of the table, and is just an indicator for existence of other terms.
Abbasi-Yadkori et al. [2012] use the SEQSEW online regression oracle [Gerchinovitz, 2011] for estimating the parameter vector, together with a UCB policy. The regression oracle is an exponential weights algorithm, which runs on the squared error loss. This subroutine, and thereby the algorithm proposed by Abbasi-Yadkori et al. [2012] are not computationally efficient, and this is believed to be unavoidable. This work considers the data-rich regime and shows  $R(n) = \mathcal{O}(\sqrt{dMn})$ , matching the lower bound of Theorem 24.3 in Lattimore and Szepesvári [2020].
Carpentier and Munos [2012] assume that the action set is a Euclidean ball, and that the noise is directly added to the parameter vector, i.e.  $y_{t} = \boldsymbol{x}^{\top}(\boldsymbol{\theta} + \boldsymbol{\varepsilon}_{t})$ . Roughly put, linear bandits with parameter noise are "easier" to solve than stochastic linear bandits with reward noise, since the noise is scaled proportionally to the features  $x_{i}$  and does less "damage" [Chapter 29.3 Lattimore and Szechesváři, 2020]. In this setting, Carpentier and Munos [2012] present a  $\mathcal{O}(d\sqrt{n})$  regret bound.
Recent work considers contextual linear bandits, where at every step  $\mathcal{A}_t$ , a stochastic finite subset of size  $K$  from  $\mathcal{A}$ , is presented to the agent. It is commonly assumed that members of  $\mathcal{A}_t$  are i.i.d., and the sampling distribution is diverse and time-independent. The diversity assumption is often in the
form of a restricted eigenvalue condition (Definition 2) on the covariance of the context distribution [e.g. in, Kim and Paik, 2019, Bastani and Bayati, 2020]. Li et al. [2022] require a stronger condition which directly assumes that  $\lambda_{\mathrm{min}}(\Phi_t)$  the minimum eigenvalue of the empirical covariance matrix is lower bounded. This is generally not true, but may hold with high probability. Hao et al. [2020] assume that the action set spans  $\mathbb{R}^{dM}$ . We believe that this assumption is the weakest in the literature, and conjecture that it is necessary for model selection. If not met, the agent can not explore in all relevant directions, and may not identify the relevant features. Our diversity assumption is similar to Hao et al. [2020], adapted to our problem setting. Mainly, we consider reward functions which are linearly parametrizable, i.e.  $\pmb{\theta}^{\top}\pmb{\phi}(\pmb{x})$ , as oppose to linear rewards, i.e.  $\pmb{\theta}^{\top}\pmb{x}$ .
A key distinguishing factor between ALEXP and existing work on sparse linear bandit is that ALEXP is horizon-independent and does not rely on a forced exploration schedule. As shown on Table 2, majority of prior work relies either on an initial exploration stage, the length of which is determined according to  $n$  [e.g., Carpentier and Munos, 2012, Kim and Paik, 2019, Li et al., 2022, Hao et al., 2020, Jang et al., 2022], or on a hand crafted schedule, which is again designed for a specific horizon [Bastani and Bayati, 2020]. Oh et al. [2021], which analyzes  $K$ -armed contextual bandits, does not require explicit exploration, and instead imposes restrictive assumptions on the diversity of context distribution, e.g. relaxed symmetry and balanced covariance. Regardless, the regret bounds hold in expectation, and are not time-uniform.
Table 2: Overview of recent work on high-dimensional Bandits. Parameter  $\tau$  shows existence of other problem-dependent terms which are not constants, and varies across different rows. The regret bounds are simplified and are not rigorous.
|A_t|	data-poor	adap.exp.	any-time	action selection policy	MS algo	context or action assumpt.	Regret
Abbasi-Yadtori et al.	∞	X	✓	✓	UCB	EXP4 on Sqrd error	A is compact	√dMn, w.h.p.
Foster et al.	K	✓	✓	X	UCB	EXP4 on Sqrd error	λmin(Σ) ≥ cλ	(Mn)3/4K1/4 + √KdMn w.h.p
Carpentier and Munos	∞	✓	X	X	UCB	Hard Thresh.	A is a ball param. noise	d√n, w.h.p.
Bastani and Bayati	K	X	X	X	Explore then Greedy	Lasso	κ(Σ) > cK	τKd2(log n + log M)2 w.h.p.
Kim and Paik	K	X	X	X	Explore then ε-Greedy	Lasso	κ(Σ) > cK	τd√n log(Mn), w.h.p.
Oh et al.	K	✓	✓	X	Greedy	Lasso	κ(Σ) > cκ + other assumes.	τd√n log(Mn) in expectation
Li et al.	K	✓	X	X	ETC	Lasso	λmin(Σ) > cλ	τ(n2d)1/3√log Mn in expectation
Hao et al.	∞	✓	X	X	ETC	Lasso	A spans RdM + is compact	(ndCmin-1)2/3(log M)1/3 w.h.p.
Jang et al.	∞	✓	X	X	ETC	Hard Thresh.	A ⊂ [−1, 1]Md + spans RdM	(nd)2/3(Cmin-1) log M)1/3 w.h.p.
ALEXP (Ours)	∞	✓	✓	✓	Greedy or UCB	EXP4 on reward est.	Im(φj) spans Rd A is compact	√n log M(n1/4 + Cmin-1 log M) w.h.p
B Time Uniform Lasso Analysis
We start by showing that the sum of squared sub-gaussian variables is a sub-Gamma process (c.f. Definition 22).
Lemma 4 (Empirical Process is sub-Gamma). For  $t \geq 1$ , suppose  $\xi_t$  are a sequence conditionally standard sub-Gaussians adapted to the filtration  $\mathcal{F}_t = \sigma(\xi_1, \ldots, \xi_t)$ . Let  $v_t \in \mathbb{R}$ , and  $Z_t := \xi_t^2 - 1$ . Define the processes  $S_t := \sum_{i=1}^{t} Z_i v_i$  and  $V_t := 4 \sum_{i=1}^{t} v_i^2$ . Then  $(S_t)_{t=0}^\infty$  is sub-Gamma with variance process  $(V_t)_{t=0}^\infty$  and scale parameter  $c = 4 \max_{i \geq 1} v_i$ .
Proof of Lemma 4. By definition [c.f. Definition 1, Howard et al., 2021],  $S_{t}$  is sub-Gamma if for each  $\lambda \in [0,1 / c)$ , there exists a supermartingale  $(M_t(\lambda))_{t = 0}^{\infty}$  w.r.t.  $\mathcal{F}_t$ , such that  $\mathbb{E}M_0 = 1$  and for all  $t\geq 1$ :
$$
\exp \left\{\lambda S _ {t} - \frac {\lambda^ {2}}{2 (1 - c \lambda)} V _ {t} \right\} \leq M _ {t} (\lambda) \quad a. s.
$$
We show the above holds in equality by proving that the left hand side itself, is a supermartingale w.r.t.  $\mathcal{F}_t$ . We define,  $M_t(\lambda) \coloneqq \exp \left\{\lambda S_t - \lambda^2 V_t / 2(1 - c\lambda)\right\}$ , therefore,
$$
\begin{array}{l} \mathbb {E} \left[ M _ {t} \mid \mathcal {F} _ {t - 1} \right] \leq \mathbb {E} \left[ \exp \left(\lambda S _ {t - 1} - \frac {\lambda^ {2}}{2 (1 - c \lambda)} V _ {t - 1} + \lambda Z _ {t} v _ {t} - \frac {2 \lambda^ {2} v _ {t} ^ {2}}{(1 - c \lambda)}\right) \mid \mathcal {F} _ {t - 1} \right] \\ = \mathbb {E} \left[ M _ {t - 1} \mid \mathcal {F} _ {t - 1} \right] \mathbb {E} \left[ \exp \left(\lambda Z _ {t} v _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \exp \left(- \frac {2 \lambda^ {2} v _ {t} ^ {2}}{1 - c \lambda}\right) \\ = M _ {t - 1} \mathbb {E} \left[ \exp \left(\lambda Z _ {t} v _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \exp \left(- \frac {2 \lambda^ {2} v _ {t} ^ {2}}{1 - c \lambda}\right). \\ \end{array}
$$
Note that  $Z_{t}$  is  $\mathcal{F}_{t-1}$ -measurable, conditionally centered and conditionally sub-exponential with parameters  $(\nu, \alpha) = (2, 4)$  (c.f. Vershynin [2018, Lemma 2.7.6] and Wainwright [2019, Example 2.8]). Therefore, for  $\lambda < 1/c$ ,
$$
\mathbb {E} \left[ \exp \left(\lambda v _ {t} Z _ {t}\right) \mid \mathcal {F} _ {t - 1} \right] \leq \exp \left(2 \lambda^ {2} v _ {t} ^ {2}\right) \leq \exp \left(\frac {2 \lambda^ {2} v _ {t} ^ {2}}{1 - c \lambda}\right),
$$
where the last inequality holds due to the fact that  $0 \leq 1 - c\lambda < 1$ . Therefore,
$$
\mathbb {E} \left[ M _ {t} \mid \mathcal {F} _ {t - 1} \right] \leq M _ {t - 1} \exp \left(\frac {2 \lambda^ {2} v _ {t} ^ {2}}{1 - c \lambda}\right) \exp \left(- \frac {2 \lambda^ {2} v _ {t} ^ {2}}{1 - c \lambda}\right) = M _ {t - 1}.
$$
for  $\lambda \in [0,1 / c)$ , concluding the proof.
We now construct a self-normalizing martingale sequence based on  $\ell_2$ -norm of the empirical process error term, and recognize that it is a sub-gamma process. We then employ our curved Bernstein bound Lemma 25 to control the boundary. This step will allow us to "ignore" the empirical process error term later in the lasso analysis.
Lemma 5 (Empirical Process is dominated by regularization.). Let
$$
A _ {j} = \left\{\forall t \geq 1: \left\| \left(\Phi_ {t} ^ {\top} \boldsymbol {\varepsilon} _ {t}\right) _ {j} \right\| _ {2} / t \leq \lambda_ {t} / 2 \right\}.
$$
Then, for any  $0 \leq \delta < 1$ , the event  $A = \cap_{j=1}^{M} A_j$  happens with probability  $1 - \delta$ , if for all  $t \geq 1$ ,
$$
\lambda_ {t} \geq \frac {2 \sigma}{\sqrt {t}} \sqrt {1 + \frac {5}{\sqrt {2}} \sqrt {d (\log (2 M / \delta) + (\log \log d) _ {+})} + \frac {1 2}{\sqrt {2}} (\log (2 M / \delta) + (\log \log d) _ {+})}.
$$
Proof of Lemma 5. This proof includes a treatment of the empirical process similar to Lemma B.1 in Kassraie et al. [2022], but adapts it to our time-uniform setting. Since  $\varepsilon_{i}$  are zero-mean sub-gaussian variables, as driven in Lemma 3.1 [Lounici et al., 2011], it holds that
$$
A _ {j} ^ {c} = \left\{\exists t: \frac {1}{t ^ {2}} \pmb {\varepsilon} _ {t} ^ {T} \Phi_ {t, j} \Phi_ {t, j} ^ {\top} \pmb {\varepsilon} _ {t} \geq \frac {\lambda^ {2}}{4} \right\} = \left\{\exists t: \frac {\sum_ {i = 1} ^ {t} v _ {i} (\xi_ {i} ^ {2} - 1)}{\sqrt {2} \| \pmb {v} _ {t} \|} \geq \alpha_ {t, j} \right\}
$$
where  $\xi_{i}$  are sub-gaussian variables with variance proxy 1, scalar  $v_{i}$  denotes the  $i$ -th eigenvalue of  $\Phi_{t,j} \Phi_{t,j}^{\top} / t$  with the concatenated vector  $\mathbf{v}_t = (v_1, \ldots, v_t)$ , and
$$
\alpha_ {t, j} = \frac {t ^ {2} \lambda^ {2} / (4 \sigma^ {2}) - \mathrm {t r} (\Phi_ {t , j} ^ {\top} \Phi_ {t , j})}{\sqrt {2} \left\| \Phi_ {t , j} ^ {\top} \Phi_ {t , j} \right\| _ {\mathrm {F r}}}.
$$
We can apply Lemma 25 to control the probability of event  $A_{j}^{c}$  by tuning  $\lambda$ . Mainly, for  $A_{j}^{c}$  to happen with probability less than  $\delta / M$ , Lemma 25 states that the following must hold for all  $t$ ,
$$
\sqrt {2} \| \boldsymbol {v} _ {t} \| _ {2} \alpha_ {t, j} \geq \frac {5}{2} \sqrt {\max  \left\{4 \| \boldsymbol {v} _ {t} \| _ {2} ^ {2} , 1 \right\} \omega_ {\delta / M} (\| \boldsymbol {v} _ {t} \| _ {2})} + 1 2 \omega_ {\delta / M} (\| \boldsymbol {v} _ {t} \| _ {2}) \max  _ {t \geq 1} v _ {t} \tag {4}
$$
Recall that w.l.o.g. feature maps are bounded everywhere  $\| \phi_j(\cdot)\| _2\leq 1$ , and  $\mathrm{rank}(\Phi_j)\leq d$  which allows for the following matrix inequalities,
$$
\operatorname {t r} \left(\Phi_ {t, j} ^ {\top} \Phi_ {t, j}\right) = \operatorname {t r} \left(\Phi_ {t, j} \Phi_ {t, j} ^ {\top}\right) = \sum_ {i = 1} ^ {t} \phi_ {j} ^ {\top} \left(x _ {i}\right) \phi_ {j} \left(x _ {i}\right) \leq t
$$
$$
\left\| \Phi_ {t, j} \Phi_ {t, j} ^ {\top} \right\| \leq \operatorname {t r} \left(\Phi_ {t, j} \Phi_ {t, j} ^ {\top}\right) \leq t
$$
$$
\left\| \Phi_ {t, j} \Phi_ {t, j} ^ {\top} \right\| \leq \left\| \Phi_ {t, j} \Phi_ {t, j} ^ {\top} \right\| _ {\mathrm {F r}} \leq \sqrt {d} \left\| \Phi_ {t, j} \Phi_ {t, j} ^ {\top} \right\| \leq t \sqrt {d}
$$
$$
\| \boldsymbol {v}_{t}\| = \bigl\|\Phi_{t,j}\Phi_{t,j}^{\top}\bigr\|_{\mathrm{Fr}} / t\leq \sqrt{d},\quad \max_{t\geq 1}v_{t} = \max_{t\geq 1}\bigl\|\Phi_{t,j}\Phi_{t,j}^{\top}\bigr\| / t\leq 1.
$$
For Eq. (4) to hold, is suffices that for all  $t \geq 1$ ,
$$
\lambda \geq \frac {2 \sigma}{\sqrt {t}} \sqrt {1 + \frac {5}{2 \sqrt {2}} \sqrt {4 d (\log (2 M / \delta) + (\log \log d) _ {+})} + \frac {1 2}{\sqrt {2}} (\log (2 M / \delta) + (\log \log d) _ {+})}.
$$
Therefore, if  $\lambda_{t}$  are chosen to satisfy the above inequality, each  $A_{j}^{c}$  happens with probability less than  $\delta /M$ . Then by applying union bound,  $\cup_{j = 1}^{M}A_{j}^{c}$  happens with probability less than  $\delta$ .
Proof of Theorem 3. The theorem statement requires that the regularization parameter  $\lambda_{t}$  is chosen such that condition of Lemma 5 is met, and therefore event  $A$  happens with probability  $1 - \delta$ . Throughout this proof, which adapts the analysis of Theorem 3.1. Lounici et al. [2011] to the time-uniform setting, we condition on  $A$  happening, and later incorporate the probability.
Step 1. Let  $\hat{\theta}_t$  be a minimizer of  $\mathcal{L}$  and  $\pmb{\theta}$  be the true coefficients vector, then  $\mathcal{L}(\hat{\theta}_t;H_t,\lambda_t)\leq \mathcal{L}(\pmb {\theta};H_t,\lambda_t)$ . Writing out the loss and re-ordering the inequality we obtain,
$$
\frac {1}{t} \left\| \Phi_ {t} (\hat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta}) \right\| _ {2} ^ {2} \leq \frac {2}{t} \varepsilon_ {t} ^ {T} \Phi_ {t} (\hat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta}) + 2 \lambda_ {t} \sum_ {j = 1} ^ {M} \left(\left\| \boldsymbol {\theta} _ {j} \right\| _ {2} - \left\| \hat {\boldsymbol {\theta}} _ {t, j} \right\| _ {2}\right).
$$
which is often referred to as the Basic inequality [Buhlmann and Van De Geer, 2011]. By Cauchy-Schwarz and conditioned on event  $A$ ,
$$
\varepsilon_ {t} ^ {T} \Phi_ {t} (\hat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta}) \leq \sum_ {j = 1} ^ {M} \left\| \left(\Phi_ {t} ^ {T} \varepsilon_ {t}\right) _ {j} \right\| _ {2} \left\| \hat {\boldsymbol {\theta}} _ {t, j} - \boldsymbol {\theta} _ {j} \right\| _ {2} \leq \frac {t \lambda}{2} \sum_ {j = 1} ^ {M} \left\| \hat {\boldsymbol {\theta}} _ {t, j} - \boldsymbol {\theta} _ {j} \right\| _ {2}
$$
then adding  $\lambda_t\sum_{j = 1}^M\left\| \hat{\pmb{\theta}}_{t,j} - \pmb {\theta}_j\right\|$  to both sides, applying the triangle inequality, and recalling from Section 3 that  $\pmb {\theta}_j = 0$  for  $j\neq j^{\star}$  gives
$$
\begin{array}{l} \frac {1}{t} \left\| \Phi_ {t} (\hat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta}) \right\| _ {2} ^ {2} + \lambda_ {t} \sum_ {j = 1} ^ {M} \left\| \hat {\boldsymbol {\theta}} _ {t, j} - \boldsymbol {\theta} _ {j} \right\| _ {2} \leq 2 \lambda_ {t} \sum_ {j = 1} ^ {M} \left\| \hat {\boldsymbol {\theta}} _ {t, j} - \boldsymbol {\theta} _ {j} \right\| _ {2} + 2 \lambda_ {t} \sum_ {j = 1} ^ {M} \left(\left\| \boldsymbol {\theta} _ {j} \right\| _ {2} - \left\| \hat {\boldsymbol {\theta}} _ {t, j} \right\| _ {2}\right) \\ \leq 4 \lambda_ {t} \left\| \hat {\boldsymbol {\theta}} _ {t, j ^ {*}} - \boldsymbol {\theta} _ {j ^ {*}} \right\| _ {2}. \\ \end{array}
$$
Since each term on the left hand side is positive, then each is also individually smaller than the right hand side, and we obtain,
$$
\frac {1}{t} \left\| \Phi_ {t} \left(\hat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta}\right) \right\| _ {2} ^ {2} \leq 4 \lambda_ {t} \left\| \hat {\boldsymbol {\theta}} _ {t, j ^ {\star}} - \boldsymbol {\theta} _ {j ^ {\star}} \right\| _ {2} \tag {5}
$$
$$
\sum_ {\substack {j = 1 \\ j \neq j ^ {\star}}} ^ {M} \left\| \hat {\boldsymbol {\theta}} _ {t, j} - \boldsymbol {\theta} _ {j} \right\| _ {2} \leq 3 \left\| \hat {\boldsymbol {\theta}} _ {t, j ^ {\star}} - \boldsymbol {\theta} _ {j ^ {\star}} \right\| _ {2} \tag{6}
$$
Step 2. Consider a sequence  $(c_{1},\ldots ,c_{k},\ldots)$ , where  $c_{1}\geq \dots \geq c_{k}\ldots$ , then
$$
c _ {k} \leq \frac {1}{k} \left(k c _ {k} + \sum_ {i > k} c _ {i}\right) \leq \sum_ {i \geq 1} \frac {c _ {i}}{k}. \tag {7}
$$
Define  $J_{1} = \{j^{\star}\}$  and  $J_{2} = \{j^{\star}, j'\}$  where
$$
j^{\prime} = \operatorname *{arg  max}_{\substack{j\in [M]\\ j\neq j^{\star}}}\bigg\|\hat{\boldsymbol{\theta}}_{t,j} - \boldsymbol{\theta}_{j}\bigg\|_{2}.
$$
For any  $J \subset [M]$  the complementing set is denoted as  $J^c = [M] \setminus J$ . For simplicity let  $c_j = \left\| \hat{\pmb{\theta}}_{t,j} - \pmb{\theta}_j \right\|_2$ , and let  $\pi(k)$  denote the index of the  $k$ -th largest element of  $\{c_j : j \in J_1^c\}$ . By definition of  $J_2^c$  we have,
$$
\begin{array}{l} \sum_{j\in J_{2}^{c}}\bigg\|\hat{\boldsymbol{\theta}}_{t,j} - \boldsymbol{\theta}_{j}\bigg\|_{2}^{2} = \sum_{\substack{k > 1\\ \pi (k)\in J_{1}^{c}}}c_{k}^{2}\overset {(7)}{\leq}\sum_{\substack{k > 1\\ \pi (k)\in J_{1}^{c}}}\frac{(\sum_{i\in J_{1}^{c}}c_{i})^{2}}{k^{2}} \\ \leq \big(\sum_{i\in J_{1}^{c}}c_{i}\big)^{2}\sum_{\substack{k > 1\\ \pi (k)\in J_{1}^{c}}}\frac{1}{k^{2}}\stackrel {(6)}{\leq}9c_{j^{\star}}^{2} \\ \leq 9 \left(c _ {j ^ {\star}} ^ {2} + c _ {j ^ {\prime}} ^ {2}\right) = 9 \sum_ {j \in J _ {2}} \left\| \hat {\boldsymbol {\theta}} _ {t, j} - \boldsymbol {\theta} _ {j} \right\| _ {2} ^ {2}, \\ \end{array}
$$
which, in turn, gives
$$
\left\| \hat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta} \right\| _ {2} = \sqrt {\sum_ {j = 1} ^ {M} \left\| \hat {\boldsymbol {\theta}} _ {t , j} - \boldsymbol {\theta} _ {j} \right\| _ {2} ^ {2}} \leq \sqrt {1 0 \sum_ {j \in J _ {2}} \left\| \hat {\boldsymbol {\theta}} _ {t , j} - \boldsymbol {\theta} _ {j} \right\| _ {2} ^ {2}}. \tag {8}
$$
Step 3. On the other hand, due to (6), and by definition of  $J_{2}$  it also holds that
$$
\sum_ {j \in J _ {2} ^ {c}} \left\| \hat {\boldsymbol {\theta}} _ {t, j} - \boldsymbol {\theta} _ {j} \right\| _ {2} \leq 3 \sum_ {j \in J _ {2}} \left\| \hat {\boldsymbol {\theta}} _ {t, j} - \boldsymbol {\theta} _ {j} \right\| _ {2}.
$$
From the theorem assumptions and Definition 2, we know that there exists  $0 < \kappa(\Phi_t, 2)$ , therefore by Definition 2, the feature matrix  $\Phi_t$  satisfies,
$$
\begin{array}{l} \sum_ {j \in J _ {2}} \left\| \hat {\boldsymbol {\theta}} _ {t, j} - \boldsymbol {\theta} _ {j} \right\| _ {2} ^ {2} \leq \frac {1}{t \kappa^ {2} (\Phi_ {t} , 2)} \left\| \Phi_ {t} (\hat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta}) \right\| _ {2} ^ {2} \\ \stackrel {(5)} {\leq} \frac {1}{\kappa^ {2} (\Phi_ {t} , 2)} 4 \lambda_ {t} \left\| \hat {\boldsymbol {\theta}} _ {t, j ^ {\star}} - \boldsymbol {\theta} _ {j ^ {\star}} \right\| _ {2} \leq \frac {1}{\kappa^ {2} (\Phi_ {t} , 2)} 4 \lambda_ {t} \sqrt {\sum_ {j \in J _ {2}} \left\| \hat {\boldsymbol {\theta}} _ {t , j} - \boldsymbol {\theta} _ {j} \right\| _ {2} ^ {2}}. \\ \end{array}
$$
From here, by applying (8) we get,
$$
\left\| \hat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta} \right\| _ {2} \leq \frac {4 \sqrt {1 0} \lambda_ {t}}{\kappa^ {2} (\Phi_ {t} , 2)}.
$$
If  $\lambda_{t}$  are chosen according to Lemma 5, event  $A$  and, in turn, the inequality above hold with probability greater than  $1 - \delta$ .
C Results on Exploration
In this section we present lower-bounds on the eigenvalues of the covariance matrix  $\Phi_t\Phi_t^\top$ , as it is later used in our regret analysis. In particular, we show that the feature matrix  $\Phi_t$  satisfies the restricted eigenvalue condition (Definition 2) required for valid Lasso confidence set (Theorem 3), and calculate a lower bound on  $\kappa(\Phi_t,2)$ . The lower bound is later used by Lemma 19 and Lemma 20 to develop the model selection regret. We show this bound in three steps.
Equivalent to Definition 2, we write  $\kappa(\Phi_t, s) = \inf_{b \in \Xi_s} \| \Phi_t b \|_2 / \sqrt{t}$  where
$$
\Xi_ {s} := \left\{\boldsymbol {b} \in \mathbb {R} ^ {d} \backslash \{0 \} \mid \sum_ {j \notin J} \| \boldsymbol {b} _ {j} \| _ {2} \leq 3 \sum_ {j \in J} \| \boldsymbol {b} _ {j} \| _ {2}, \sqrt {\sum_ {j \in J} \| \boldsymbol {b} _ {j} \| _ {2} ^ {2}} \leq 1 \text {s . t .} J \subset \{1, \dots , M \}, | J | \leq s. \right\}. \tag {9}
$$
For simplicity in notation, we further define
$$
\tilde {\kappa} (A, s) := \min  _ {b \in \Xi_ {s}} b ^ {\top} A b. \tag {10}
$$
since  $\tilde{\kappa} (\frac{\Phi_t^\top\Phi_t}{t},s) = \kappa^2 (\Phi_t,s)$
Step I. Consider the exploratory steps at which  $\alpha_{t} = 1$ . Let  $\Phi_{\pi, t}$  be a sub-matrix of  $\Phi_{t}$  where only rows from exploratory steps are included. Note that  $\Phi_{\pi, t} \in \mathbb{R}^{t' \times dM}$  is a random matrix, where the number of rows  $t'$  are also random. We show that  $\kappa^2(\Phi_t, s)$  is lower bounded by  $\kappa^2(\Phi_{t,\pi}, s)$ .
Lemma 6. Suppose  $\Phi_{\pi ,t}$  has  $t^\prime$  rows. Then,
$$
\kappa^ {2} (\Phi_ {t}, s) \geq \frac {t ^ {\prime}}{t} \kappa^ {2} (\Phi_ {\pi , t}, s)
$$
Proof of Lemma 6. Let  $\Psi^{(t)} = \phi (\pmb {x}_t)\phi^\top (\pmb {x}_t)\in \mathbb{R}^{dM\times dM}$  for all  $t = 1,\ldots ,n$ . Note that  $\Psi^{(t)}$  is positive semi-definite by construction. We have,
$$
\| \Phi_ {t} b \| _ {2} ^ {2} = \sum_ {s = 1} ^ {t} b ^ {\top} \Psi^ {(s)} b = \sum_ {s \in T _ {\pi}} ^ {t} b ^ {\top} \Psi^ {(s)} b + \sum_ {s \notin T _ {\pi}} ^ {t} b ^ {\top} \Psi^ {(s)} b
$$
where the set  $T_{\pi}$  contains the indices of the exploratory steps at which the action is selected according to  $\pi$ . Therefore,
$$
\begin{array}{l} \kappa^ {2} (\Phi_ {t}, s) = \frac {1}{t} \min  _ {b \in \Xi_ {s}} \| \Phi_ {t} \boldsymbol {b} \| _ {2} ^ {2} \\ = \min  _ {b \in \Xi_ {s}} \frac {1}{t} \sum_ {s \in T _ {\pi}} b ^ {\top} \Psi^ {(s)} b + \frac {1}{t} \sum_ {s \notin T _ {\pi}} b ^ {\top} \Psi^ {(s)} b \\ \geq \min  _ {b \in \Xi_ {s}} \frac {1}{t} \sum_ {s \in T _ {\pi}} b ^ {\top} \Psi^ {(s)} b \\ \end{array}
$$
where the last inequality holds due to  $\Psi^{(s)}$  being PSD. Then we have,
$$
\kappa^ {2} (\Phi_ {t}, s) \geq \min _ {b \in \Xi_ {s}} b ^ {\top} \left(\frac {1}{t} \sum_ {s \in T _ {\pi}} ^ {t} \Psi^ {(s)}\right) b = \frac {| T _ {\pi} |}{t} \kappa^ {2} (\Phi_ {\pi , t}, s)
$$
While the number of rows of  $\Phi_{\pi ,t}$  is a random variable, we continue to condition on the event that  $\Phi_{\pi ,t}$  has  $t^\prime$  rows, and investigate the distribution of its restricted eigenvalues.
Step II. The restricted eigenvalues of the exploratory submatrix are well bounded away from zero.
Lemma 7. Let  $\pi$  be the solution to (2), and  $s\in \mathbb{N}$ . Suppose  $\Phi_{\pi ,t}$  has  $t^\prime$  rows. Then for all  $\delta >0$
$$
\mathbb {P} \left(\forall t ^ {\prime}: \kappa^ {2} (\Phi_ {\pi , t}, s) \geq \tilde {\kappa} (\Sigma , s) - \frac {8 0 s}{\sqrt {t ^ {\prime}}} \sqrt {\log (2 M d / \delta) + (\log \log 4 t ^ {\prime}) _ {+}}\right) \geq 1 - \delta
$$
where  $\Sigma = \Sigma (\pi ,\phi)\coloneqq \mathbb{E}_{\pmb {x}\sim \pi}\phi (\pmb {x})\phi^{\top}(\pmb {x})$  and  $\tilde{\kappa}$  is defined in (10).
Step III. Remains to combine the two above lemmas and incorporate a high probability bound on  $t'$ , showing that it is close to  $\sum_{s=1}^{t} \gamma_s$ .
Lemma 8. There exist absolute constants  $C_1, C_2$  which satisfy,
$$
\mathbb {P} \left(\forall t \geq 1: \kappa^ {2} (\Phi_ {t}, 2) \geq C _ {1} \tilde {\kappa} (\Sigma , 2) t ^ {- 1 / 4} - C _ {2} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \geq 1 - \delta
$$
if  $\gamma_t = \mathcal{O}(t^{-1/4})$ . Let  $\pi$  be the solution to (2), then it further holds that
$$
\mathbb {P} \left(\forall t \geq 1: \kappa^ {2} (\Phi_ {t}, 2) \geq C _ {1} C _ {\min } t ^ {- 1 / 4} - C _ {2} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \geq 1 - \delta
$$