SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The regret analysis of Hao et al. [2020] also relies on connecting  $\kappa(\Phi_t, s)$  to  $C_{\mathrm{min}}$ , and for this, they use Theorem 2.4 of Javanmard and Montanari [2014]. This theorem states that there exists a problem-dependent constant  $C_1$  for which  $\kappa^2(\Phi_t, s) \geq C_1 C_{\mathrm{min}}$  with high probability, if  $t \geq n_0$  and roughly  $n_0 = \mathcal{O}\left(\sqrt[3]{n^2 \log M}\right)$ . We highlight that Lemma 8, presents a lower-bound which holds for all  $t \geq 1$ , however this comes at the cost of getting a looser lower bound than the result of Javanmard and Montanari [2014] for the larger time steps  $t$ . In fact, due to the sub-optimal dependency of Lemma 8 on  $t$ , we later obtain sub-optimal dependency on the horizon for the case when  $n \gg M$ . It is unclear to us if this rate can be improved without assuming knowledge of  $n$ , or that  $n \geq n_0$ .
For the last lemma in this section we show that the empirical sub-matrices  $\Phi_{t,j}$  are also bounded away from zero. This will be required later to prove Lemma 15.
Lemma 9 (Base Model  $\lambda_{\mathrm{min}}$  Bound). Assume  $\pi$  is the maximizer of Eq. (2). Then, with probability greater than  $1 - \delta$ , simultaneously for all  $j = 1,\dots ,M$  and  $t\geq 1$
$$
\lambda_ {\min } \left(\Phi_ {t, j} ^ {\top} \Phi_ {t, j}\right) \geq C _ {1} C _ {\min } t ^ {3 / 4} - C _ {2} t ^ {3 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}
$$
$$
i f \gamma_ {t} = \mathcal {O} (t ^ {- 1 / 4}).
$$
C.1 ALEXP with Uniform Exploration
We presented our main regret bound (Theorem 1) in terms of  $C_{\mathrm{min}}$ , which only depends on properties of the feature maps and the action domain. We give a lower-bound on  $C_{\mathrm{min}}$  for a toy scenario which corresponds to the problem of linear feature selection over convex action sets.
Proposition 10 (Invertible Features). Suppose  $\phi(\pmb{x}) \coloneqq A\pmb{x} : \mathbb{R}^d \to \mathbb{R}^d$  is an invertible linear map, and  $\mathcal{X} \in \mathbb{R}^d$  is a convex body. Then,
$$
C _ {\min } \geq \frac {\lambda_ {\min } (A)}{\lambda_ {\max } ^ {2} (T)} > 0
$$
where  $T$  is the transformation which maps  $\mathcal{X}$  to an isotropic body.
The lower-bound of Proposition 10 is achieved by simply exploring via  $\pi = \mathrm{Unif}(\mathcal{X})$ . Inspired by Schur et al. [2023, Lemma E.13], we show that even for non-convex action domains and orthogonal feature maps, the uniform exploration yields a constant lower-bound on restricted eigenvalues.
Proposition 11 (Orthonormal Features). Suppose  $\phi_j: \mathcal{X} \to \mathbb{R}$  are chosen from an orthogonal basis of  $L^2(\mathcal{X})$ , and satisfy  $\| \phi_i \|_{L_\mu^2(\mathcal{X})} / \text{Vol}(\mathcal{X}) \geq 1$ . Then there exist absolute constants  $C_1$  and  $C_2$  for which the exploration distribution  $\pi = \text{Unif}(\mathcal{X})$  satisfies
$$
\mathbb {P} \left(\forall t \geq 1: \kappa^ {2} (\Phi_ {t}, 2) \geq C _ {1} t ^ {- 1 / 4} - C _ {2} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \geq 1 - \delta .
$$
The  $d = 1$  condition is met without loss of generality, by splitting the higher dimensional feature maps and introducing more base features, which will increase  $M$ . Moreover, the orthonormality condition is met by orthogonalizing and re-scaling the feature maps. Basis functions such as Legendre polynomials and Fourier features [Rahimi et al., 2007] satisfy these conditions.
By invoking Proposition 11, instead of Lemma 8 in the proof of Theorem 1, we obtain the regret of ALEXP with uniform exploration.
Corollary 12 (ALEXP with Uniform Exploration). Let  $\delta \in (0,1]$ . Suppose  $\phi_j: \mathcal{X} \to \mathbb{R}$  are chosen from an orthogonal basis of  $L^2(\mathcal{X})$ , and satisfy  $\| \phi_i \|_{L_\mu^2(\mathcal{X})} / \text{Vol}(\mathcal{X}) \geq 1$ . Assume the oracle agent employs a UCB or a Greedy policy, as laid out in Section 5. Choose  $\eta_t = \mathcal{O}(1/\sqrt{t} C(M, \delta, d))$  and  $\gamma_t = \mathcal{O}(t^{-1/4})$  and  $\lambda_t = \mathcal{O}(C(M, \delta, d)/\sqrt{t})$ , then ALEXP with uniform exploration  $\pi = \text{Unif}(\mathcal{X})$  attains the regret
$$
\begin{array}{l} R (n) = \mathcal {O} \left(B n ^ {3 / 4} + \sqrt {n} C (M, \delta , d) \log M + B ^ {2} \sqrt {n} + B \sqrt {n \left((\log \log n B ^ {2}) _ {+} + \log (1 / \delta)\right)}\right) \\ \left. + \left(n ^ {3 / 4} + \log n\right) C (M, \delta , d) + n ^ {5 / 8} \sqrt {d \log n + \log (1 / \delta) + B ^ {2}}\right) \\ \end{array}
$$
with probability greater than  $1 - \delta$ , simultaneously for all  $n \geq 1$ . Here,
$$
C (M, \delta , d) = \mathcal {O} \left(\sqrt {1 + \sqrt {d (\log (M / \delta) + (\log \log d) _ {+})} + (\log (M / \delta) + (\log \log d) _ {+})}\right).
$$
C.2 Proof of Results on Exploration
As an intermediate step, we consider the restricted eigenvalue property of the empirical covariance matrix. Given  $t'$  samples, the empirical estimate of  $\Sigma$  is
$$
\hat {\Sigma} _ {t ^ {\prime}} := \frac {1}{t ^ {\prime}} \sum_ {s = 1} ^ {t ^ {\prime}} \phi \left(\boldsymbol {x} _ {s}\right) \phi^ {\top} \left(\boldsymbol {x} _ {s}\right) \tag {11}
$$
where  $\pmb{x}_s$  are sampled according to  $\pi$ . We show that every entry of  $\hat{\Sigma}_{t'}$  is close to the corresponding entry in  $\Sigma$ , and later use it in the proofs of eigenvalue lemmas.
Lemma 13 (Anytime Bound for The Entries of Empirical Covariance Matrix). Let  $\hat{\Sigma}_{t'}$  be the empirical covariance matrix corresponding to  $\Sigma(\pi, \phi)$  given  $t'$  samples. Then,
$$
\mathbb {P} \left(\exists t ^ {\prime}: d _ {\infty} (\Sigma , \hat {\Sigma} _ {t ^ {\prime}}) \geq \frac {5}{\sqrt {t ^ {\prime}}} \sqrt {((\log \log 4 t ^ {\prime}) _ {+} + \log (2 M d / \delta))}\right) \leq \delta
$$
where  $d_{\infty}(A,B)\coloneqq \max_{i,j}|A_{i,j} - B_{i,j}|$
Proof of Lemma 13. We show the element-wise convergence of  $\Sigma$  to  $\hat{\Sigma}_t$  for the  $(i,j)$  entry where  $i,j = 1,\ldots ,dM$ . Consider the random sequence  $X_{s}\coloneqq \Sigma_{i,j} - \phi_{i}(\pmb{x}_{s})\phi_{j}(\pmb{x}_{s})$ . We show that  $X_{1},\dots,X_{n}$  satisfies conditions of Lemma 26. We first observe that
$$
\mathbb {E} \left[ X _ {s} \mid X _ {1: s - 1} \right] = \mathbb {E} X _ {s} = \Sigma (i, j) - \mathbb {E} _ {\boldsymbol {x} \sim \pi} \phi_ {i} (\boldsymbol {x}) \phi_ {j} (\boldsymbol {x}) = 0
$$
since by definition  $\Sigma_{i,j} = \mathbb{E}_{\boldsymbol{x}\sim \pi}\phi_i(\boldsymbol{x})^\top \phi_j(\boldsymbol{x})$ . Moreover, we have normalized features  $\| \phi (\cdot)\| \leq 1$  therefore, each entry  $\phi_i(\cdot)\phi_j(\cdot)$  is also bounded, yielding  $|X_{s}|\leq 2$ . Then Lemma 26 implies that for all  $\tilde{\delta} >0$
$$
\mathbb {P} \left(\exists t ^ {\prime}: \frac {1}{t ^ {\prime}} \sum_ {s = 1} ^ {t ^ {\prime}} X _ {s} \geq \frac {5}{\sqrt {t ^ {\prime}}} \sqrt {\left(\left(\log \log 4 t ^ {\prime}\right) _ {+} + \log (2 / \tilde {\delta})\right)}\right) \leq \tilde {\delta}.
$$
Setting  $\tilde{\delta} = \delta /(dM)$  and taking a union bound over all indices concludes the proof.
We are now ready to present the proofs to the lemmas in Appendix C.
Proof of Lemma 7. By (11) we have  $\hat{\Sigma}_{t'} = \frac{\Phi_{\pi,t}^\top \Phi_{\pi,t}}{t'}$ , and thereby
$$
\kappa^ {2} (\Phi_ {\pi , t}, s) = \min _ {b \in \Xi_ {s}} b ^ {\top} \hat {\Sigma} _ {t ^ {\prime}} b = \tilde {\kappa} (\hat {\Sigma} _ {t ^ {\prime}}, s).
$$
Inspired by Lemma 10.1 in van de Geer and Buhlmann [2009], we show that element-wise closeness of matrices  $\Sigma$  and  $\hat{\Sigma}_{t'}$  (c.f. Lemma 13) implies closeness in  $\tilde{\kappa}$ :
$$
\begin{array}{l} \left| \kappa^ {2} \big (\Phi_ {\pi , t}, s \big) - \tilde {\kappa} (\Sigma , s) \right| = \left| \tilde {\kappa} \big (\hat {\Sigma} _ {t ^ {\prime}}, s \big) - \tilde {\kappa} (\Sigma , s) \right| \\ = \left| \tilde {\kappa} \left(\hat {\Sigma} _ {t ^ {\prime}} - \Sigma , s\right) \right| \\ \leq \min _ {b \in \Xi_ {s}} d _ {\infty} (\Sigma , \hat {\Sigma} _ {t ^ {\prime}}) \| b \| _ {1} ^ {2} \\ \end{array}
$$
where the last line holds due to Hölder's. Moreover, since  $b \in \Xi_s$ , for any  $J \subset [dM]$  where  $|J| \leq s$  it additionally holds that  $\| b_J \|_2 \leq 1$  and
$$
\left\| b \right\| _ {1} \leq (1 + 3) \left\| b _ {J} \right\| _ {1} \leq 4 \sqrt {s} \left\| b _ {J} \right\| _ {2} \leq 4 \sqrt {s}
$$
which gives,
$$
\kappa^ {2} \big (\Phi_ {\pi , t}, s \big) \geq \tilde {\kappa} \big (\Sigma , s \big) - 1 6 s d _ {\infty} \big (\Sigma , \hat {\Sigma} _ {t ^ {\prime}} \big).
$$
Therefore by Lemma 13,
$$
\kappa^ {2} \left(\Phi_ {\pi , t}, s\right) \geq \tilde {\kappa} (\Sigma , s) - \frac {8 0 s}{\sqrt {t ^ {\prime}}} \sqrt {\left(\left(\log \log 4 t ^ {\prime}\right) _ {+} + \log (2 M d / \delta)\right)} \tag {12}
$$
with probability greater than  $1 - \delta$ , simultaneously for all  $t' \geq 1$ .
Proof of Lemma 8. In Lemma 6 we showed that
where  $t'$  indicates the number of rows in the exploratory sub-matrix of  $\Phi_t$ . Recall that  $t' = \sum_{s=1}^{t} \alpha_s$  where  $\alpha_s$  are i.i.d Bernoulli random variables with success probability of  $\gamma_s$ . Due to Lemma 24,
$$
\mathbb {P} (\forall t \geq 1: | t ^ {\prime} - \Gamma_ {t} | \leq \Delta_ {t}) \geq 1 - \delta / 2 \tag {13}
$$
$$
\Delta_ {t} := \frac {5}{2} \sqrt {\frac {(\log \log t) _ {+} + \log (8 / \delta)}{t}}, \quad \Gamma_ {t} := \sum_ {s = 1} ^ {t} \gamma_ {s}
$$
Due to Lemma 7, with probability greater than  $1 - \delta /2$  the following holds for all  $t\geq 1$
$$
\begin{array}{l} \kappa^ {2} (\Phi_ {t}, 2) \geq \frac {t ^ {\prime}}{t} \tilde {\kappa} (\Sigma , 2) - \frac {1 6 0 \sqrt {t ^ {\prime}}}{t} \sqrt {(\log \log 4 t ^ {\prime}) _ {+} + \log (4 M d / \delta)} \\ \geq \frac {\Gamma_ {t} - \Delta_ {t}}{t} \tilde {\kappa} (\Sigma , 2) - 1 6 0 \sqrt {\frac {\Gamma_ {t} + \Delta_ {t}}{t ^ {2}}} \sqrt {(\log \log (4 \Gamma_ {t} + \Delta_ {t})) _ {+} + \log (4 M d / \delta)} \\ \end{array}
$$
where the second inequality holds with probability  $1 - \delta$ , by incorporating (13) and taking a union bound. For the rest of the proof and to keep the calculations simple, we ignore the values of the absolute constants. We use the notation  $g(t) = o(f(t))$  to show that  $f(t)$  grows much faster than  $g(t)$ . More formally, if for every constant  $c$  there exists  $t_0$ , where  $g(t) \leq c|f(t)|$  for all  $t \geq t_0$ . Since  $\gamma_s = \mathcal{O}(s^{-1/4})$  there exists  $C$  such that  $\Gamma_t = Ct^{3/4}$ , then it is straightforward to observe that there exists absolute constants  $\tilde{C}_i$  which satisfy,
$$
\begin{array}{l} \kappa^ {2} (\Phi_ {t}, 2) \geq \tilde {C} _ {1} t ^ {- 1 / 4} \tilde {\kappa} (\Sigma , 2) - \frac {5 t ^ {- 3 / 2} \tilde {\kappa} (\Sigma , 2)}{2} \sqrt {(\log \log t) _ {+} + \log (8 / \delta)} \\ - \tilde {C} _ {2} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}} - o \left(t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \\ \geq \tilde {C} _ {1} t ^ {- 1 / 4} \tilde {\kappa} (\Sigma , 2) - \tilde {C} _ {3} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}} \\ \end{array}
$$
The last inequality holds since  $t^{-3/2}\sqrt{\log\log t} = o(t^{-5/8}\sqrt{\log\log t})$ . The above chain of inequalities imply that there exist absolute constants  $C_1, C_2$ , for which
$$
\mathbb {P} \left(\forall t \geq 1: \kappa^ {2} (\Phi_ {t}, 2) \geq C _ {1} \tilde {\kappa} (\Sigma , 2) t ^ {- 1 / 4} - C _ {2} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \geq 1 - \delta .
$$
If  $\pi$  is chosen according to (2), then  $\tilde{\kappa} (\Sigma ,2)\geq C_{\mathrm{min}}$  yielding the lemma's second argument.
Proof of Lemma 9. Fix  $j \in \{1, \dots, M\}$ , and construct the set
$$
\Xi_ {1, j} = \left\{\boldsymbol {b} \in \mathbb {R} ^ {d} \backslash \{0 \} \middle | \boldsymbol {b} = (\boldsymbol {b} _ {1}, \dots , \boldsymbol {b} _ {M}), \text {s . t .} \boldsymbol {b} _ {j} \in \mathbb {R} ^ {d}, \| \boldsymbol {b} _ {j} \| _ {2} \leq 1 \text {a n d} \forall j ^ {\prime} \neq j: \boldsymbol {b} _ {j ^ {\prime}} = 0 \right\}.
$$
Note that  $\Xi_{1,j} \subset \Xi_s$ . Therefore,
$$
\inf  _ {b \in \Xi_ {1, j}} \| \Phi_ {t} \boldsymbol {b} \| _ {2} \geq \inf  _ {b \in \Xi_ {s}} \| \Phi_ {t} \boldsymbol {b} \| _ {2} = \sqrt {t} \kappa (\Phi_ {t}, s).
$$
Moreover, by construction of  $\Xi_{1,j}$  we have for all  $\pmb {b}\in \Xi_{1,j}$  that  $\Phi_t\pmb {b} = \Phi_{t,j}\pmb {b}_j$  , therefore,
$$
\inf_{b\in \Xi_{1,j}}\| \Phi_{t}\boldsymbol {b}\|_{2}^{2} = \inf_{\substack{\boldsymbol{b}_{j}\in \mathbb{R}^{d}\\ \| \boldsymbol{b}_{j}\|_{2}^{2}\leq 1}}\| \Phi_{t,j}\boldsymbol{b}_{j}\|_{2}^{2} = \lambda_{\min}(\Phi_{t,j}^{\top}\Phi_{t,j}).
$$
From the above equations we conclude that  $\lambda_{\min}(\Phi_{t,j}^{\top}\Phi_{t,j})\geq t\kappa^{2}(\Phi_{t},s)$ , for all  $j = 1,\ldots ,M$ . Therefore, using Lemma 8 we obtain that there exists  $C_1,C_2$  such that
$$
\mathbb {P} \left(\forall t \geq 1, j = 1, \dots , M: \lambda_ {\min } \left(\Phi_ {t, j} ^ {\top} \Phi_ {t, j}\right) \geq C _ {1} C _ {\min } t ^ {3 / 4} - C _ {2} t ^ {3 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \geq 1 - \delta
$$
Proof of Proposition 10. Since  $\mathcal{X}$  is a convex body, then there exists an invertible map  $T$ , such that  $T(\mathcal{X})$  is an isotropic body [e.g. Proposition 1.1.1., Giannopoulos, 2003]. Then by definition,  $\bar{X} \sim \mathrm{Unif}(T(\mathcal{X}))$  is an isotropic distribution and  $\operatorname{Cov}(\bar{X}) = I_d$  [e.g., c.f. Chapter 3.3.5 Vershynin, 2018]. Since  $\phi$  is linear and invertible, it may be written as  $\phi(\boldsymbol{x}) = A\boldsymbol{x}$ , where  $A$  is an invertible matrix. Therefore,
$$
\Sigma (\pi , \phi) = \operatorname {C o v} (\phi (X)) = A ^ {\top} \operatorname {C o v} (X) A = A ^ {\top} \operatorname {C o v} \left(T ^ {- 1} \bar {X}\right) A = A ^ {\top} \left(T ^ {- 1}\right) ^ {2} A.
$$
As for the minimum eigenvalue, suppose  $\pmb{v} \in \mathbb{R}^d$  and  $\| \pmb{v} \| = 1$ , then
$$
C _ {\min } \geq \lambda_ {\min } (\Sigma (\pi , \phi)) \geq \boldsymbol {v} ^ {\top} A ^ {\top} (T ^ {- 1}) ^ {2} A \boldsymbol {v} \geq \| A \boldsymbol {v} \| _ {2} \lambda_ {\min } (T ^ {- 2}) = \frac {\| A \boldsymbol {v} \| _ {2}}{\lambda_ {\max } ^ {2} (T)} \geq \frac {\lambda_ {\min } (A)}{\lambda_ {\max } ^ {2} (T)}.
$$
Proof of Proposition 11. By the assumption of the proposition, for all  $i \in M$
$$
[ \Sigma (\pi , \phi) ] _ {i, i} = \mathbb {E} _ {\boldsymbol {x} \sim \pi} \phi_ {i} ^ {2} (\boldsymbol {x}) = \frac {1}{\operatorname {V o l} (\mathcal {X})} \int_ {\mathcal {X}} \phi_ {i} ^ {2} (\boldsymbol {x}) \mathrm {d} \mu (\boldsymbol {x}) \geq 1
$$
and for all  $i \neq j$ ,
$$
\left[ \Sigma (\pi , \phi) \right] _ {i, j} = \mathbb {E} _ {\boldsymbol {x} \sim \pi} \phi_ {i} (\boldsymbol {x}) \phi_ {j} (\boldsymbol {x}) = \frac {1}{\operatorname {V o l} (\mathcal {X})} \int_ {\mathcal {X}} \phi_ {i} (\boldsymbol {x}) \phi_ {j} (\boldsymbol {x}) \mathrm {d} \mu (\boldsymbol {x}) = 0
$$
We use  $\Sigma = \Sigma (\pi ,\phi)$ . For any  $\pmb {b}\in \mathbb{R}^{Md}$  where  $\| \pmb {b}\| \leq 1$
$$
\begin{array}{l} \boldsymbol {b} ^ {\top} \Sigma \boldsymbol {b} = \sum_ {i, j \in [ M ]} \boldsymbol {b} _ {j} ^ {\top} \Sigma_ {i, j} \boldsymbol {b} _ {i} = \sum_ {i \in [ M ]} \boldsymbol {b} _ {i} ^ {\top} \Sigma_ {i, i} \boldsymbol {b} _ {i} + \sum_ {i, j \in [ M ], i \neq j} \boldsymbol {b} _ {j} ^ {\top} \Sigma_ {i, j} \boldsymbol {b} _ {i} \\ = \sum_ {i \in [ M ]} \boldsymbol {b} _ {i} ^ {\top} \Sigma_ {i, i} \boldsymbol {b} _ {i} \\ \geq 1 \sum_ {i \in [ M ]} \| \boldsymbol {b} _ {i} \| _ {2} ^ {2} \geq 1. \\ \end{array}
$$
Which implies,
$$
\tilde {\kappa} (\Sigma , s) = \min  _ {\boldsymbol {b} \in \Xi_ {s}} \boldsymbol {b} ^ {\top} \Sigma \boldsymbol {b} \geq 1.
$$
By Lemma 8, there exist absolute constants  $C_1$  and  $C_2$  for which,
$$
\mathbb {P} \left(\forall t \geq 1: \kappa^ {2} (\Phi_ {t}, 2) \geq \tilde {\kappa} (\Sigma , 2) C _ {1} t ^ {- 1 / 4} - C _ {2} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \geq 1 - \delta .
$$
D Proof of Regret Bound
Theorem 14 (Anytime Regret, Formal). Let  $\delta \in (0,1]$  and  $\pi$  be the maximizer of (2). Assume the oracle agent employs a UCB or a Greedy policy, as laid out in Section 5. Suppose  $\eta_t = \mathcal{O}(C_{\min}t^{-1/2}C(M,\delta,d))$  and  $\gamma_t = \mathcal{O}(t^{-1/4})$  and  $\lambda_t = \mathcal{O}(C(M,\delta,d)t^{-1/2})$ , then exists absolute constants  $C_1, \ldots, C_6$  for which ALEXP attains the regret
$$
\begin{array}{l} R (n) \leq C _ {1} B n ^ {3 / 4} + C _ {2} \sqrt {n} C _ {\min } ^ {- 1} C (M, \delta , d) \log M \\ + C _ {3} B ^ {2} C _ {\min } \sqrt {n} + C _ {4} B \sqrt {n \left(\left(\log \log n B ^ {2}\right) _ {+} + \log (1 / \delta)\right)} \\ + C _ {5} \left(1 + C _ {\min } ^ {- 1} n ^ {- 3 / 8} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right) \\ \times \left[ B n ^ {1 / 4} + \left(n ^ {3 / 4} + \frac {\log n}{C _ {\min }}\right) C (M, \delta , d) + \frac {n ^ {5 / 8}}{\sqrt {C _ {\min }}} \sqrt {d \log n + \log (1 / \delta) + B ^ {2}} \right] \\ \end{array}
$$
$$
C (M, \delta , d) = C _ {6} \sigma \sqrt {1 + \sqrt {d (\log (M / \delta) + (\log \log d) _ {+})} + (\log (M / \delta) + (\log \log d) _ {+})}.
$$
Our main regret bound is an immediate corollary of Lemma 15 and Lemma 16, considering the regret decomposition of (3).
Lemma 15 (Virtual Regret of the Oracle). Let  $\delta \in (0,1]$  and  $\tilde{\lambda} > 0$ . Assume the oracle agent employs a UCB or a Greedy policy, as laid out in Section 5. If  $\gamma_{t} = \mathcal{O}(t^{1/4})$ , there exists an absolute constant  $C_1$  for which with probability greater than  $1 - \delta$ , simultaneously for all  $n \geq 1$ ,
$$
\begin{array}{l} \tilde {R} _ {j ^ {\star}} (n) = \frac {C _ {1} n ^ {5 / 8}}{\sqrt {C _ {\operatorname* {m i n}}}} \left(1 + n ^ {- 3 / 8} C _ {\operatorname* {m i n}} ^ {- 1} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right) \\ \times \sqrt {\sigma^ {2} d \log \left(\frac {n}{\tilde {\lambda} d} + 1\right) + 2 \sigma^ {2} \log (1 / \delta) + \tilde {\lambda} B ^ {2}} \\ \end{array}
$$
Lemma 16 (Any-Time Model-Selection Regret, Formal). Let  $\delta \in (0,1]$  and  $\pi$  be the maximizer of (2). Suppose  $\eta_t = \mathcal{O}(C_{\min} / \sqrt{t} C(M,\delta,d))$  and  $\gamma_t = \mathcal{O}(t^{-1/4})$  and  $\lambda_t = \mathcal{O}(C(M,\delta,d) / \sqrt{t})$ , then exists absolute constants  $C_i$  for which ALEXP attains the model selection regret
$$
\begin{array}{l} R (n, j) \leq C _ {1} B n ^ {3 / 4} + C _ {2} \sqrt {n} C _ {\min } ^ {- 1} C (M, \delta , d) \log M \\ + C _ {3} B ^ {2} C _ {\min } \sqrt {n} + C _ {4} B \sqrt {n \left(\left(\log \log n B ^ {2}\right) _ {+} + \log (1 / \delta)\right)} \\ + C _ {5} \left(B n ^ {1 / 4} + \left(n ^ {3 / 4} + \frac {\log n}{C _ {\min}}\right) C (M, \delta , d)\right) \left(1 + C _ {\min } ^ {- 1} n ^ {- 3 / 8} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right) \\ \end{array}
$$
D.1 Proof of Model Selection Regret
Our technique for bounding the model selection regret relies on a classic horizon-independent analysis of the exponential weights algorithm, presented in Lemma 17.
Lemma 17 (Anytime Exponential Weights Guarantee). Assume  $\eta_t\hat{r}_{t,j}\leq 1$  for all  $1\le j\le M$  and  $t\geq 1$ . If the sequence  $(\eta_{t})_{t\geq 1}$  is non-increasing, then for all  $n\geq 1$
$$
\sum_ {t = 1} ^ {n} \hat {r} _ {t, k} - \sum_ {t = 1} ^ {n} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} \leq \frac {\log M}{\eta_ {n}} + \sum_ {t = 1} ^ {n} \eta_ {t} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} ^ {2}
$$
for any arm  $k\in [M]$
Proof of Lemma 17. Define  $\hat{R}_{t,i} \coloneqq \sum_{s=1}^{t} \hat{r}_{s,i}$  to be the expected cumulative reward of agent  $i$  after  $t$  steps. We rewrite for a fixed  $k$
$$
\sum_ {t = 1} ^ {n} \hat {r} _ {t, k} - \sum_ {t = 1} ^ {n} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} = \sum_ {t = 1} ^ {n} \hat {r} _ {t, k} - \sum_ {t = 1} ^ {n} \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ]. \tag {14}
$$
We focus on a single term in the second sum. For any  $t$ , we have
$$
\begin{array}{l} - \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ] = \log (\exp (- \mathbb {E} _ {j \sim q _ {t}} [ \frac {\eta_ {t}}{\eta_ {t}} \hat {r} _ {t, j} ])) = \log (\exp (- \mathbb {E} _ {j \sim q _ {t}} [ \eta_ {t} \hat {r} _ {t, j} ]) ^ {1 / \eta_ {t}}) \\ = \frac {1}{\eta_ {t}} \log \left(\exp \left(- \mathbb {E} _ {j \sim q _ {t}} \left[ \eta_ {t} \hat {r} _ {t, j} \right]\right)\right) \\ = \frac {1}{\eta_ {t}} \log \left(\mathbb {E} _ {i \sim q _ {t}} \exp \left(- \mathbb {E} _ {j \sim q _ {t}} \left[ \eta_ {t} \hat {r} _ {t, j} \right]\right)\right) \tag {15} \\ \end{array}
$$
The inner expectation is over  $j$ , while the outer one is over  $i$  and therefore has no effect. Moreover,
$$
\begin{array}{l} \frac {1}{\eta_ {t}} \log \mathbb {E} _ {i \sim q _ {t}} \exp (- \eta_ {t} \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ] + \eta_ {t} \hat {r} _ {t, i}) = \frac {1}{\eta_ {t}} \log (\exp (- \eta_ {t} \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ]) \mathbb {E} _ {i \sim q _ {t}} \exp (\eta_ {t} \hat {r} _ {t, i})) \\ = \frac {1}{\eta_ {t}} \log \mathbb {E} _ {i \sim q _ {t}} \exp \left(- \eta_ {t} \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ]\right) \\ + \frac {1}{\eta_ {t}} \log \mathbb {E} _ {i \sim q _ {t}} \exp \left(\eta_ {t} \hat {r} _ {t, i}\right) \tag {16} \\ \end{array}
$$
where again, the expectation can be reintroduced to get the last line. Combining (15) and (16),
$$
- \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ] = \frac {1}{\eta_ {t}} \log \mathbb {E} _ {i \sim q _ {t}} \exp \left(- \eta_ {t} \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ] + \eta_ {t} \hat {r} _ {t, i}\right) - \frac {1}{\eta_ {t}} \log \mathbb {E} _ {i \sim q _ {t}} \exp \left(\eta_ {t} \hat {r} _ {t, i}\right) \tag {17}
$$
This transformation is at the core of many exponential weight proofs [Bubeck et al., 2012, Lattimore and Szepesvári, 2020]. We first bound the first term in (17):
$$
\log \mathbb {E} _ {i \sim q _ {t}} \exp \left(- \eta_ {t} \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ] + \eta_ {t} \hat {r} _ {t, i}\right) = \log \mathbb {E} _ {i \sim q _ {t}} \exp \left(\eta_ {t} \hat {r} _ {t, i}\right) - \eta_ {t} \mathbb {E} _ {j \sim q _ {t}} \hat {r} _ {t, j}
$$
$$
\begin{array}{l} \stackrel {\text {(I)}} {\leq} \mathbb {E} _ {i \sim q _ {t}} \exp \left(\eta_ {t} \hat {r} _ {t, i}\right) - 1 - \eta_ {t} \mathbb {E} _ {j \sim q _ {t}} \hat {r} _ {t, j} \\ = \mathbb {E} _ {i \sim q _ {t}} \left[ \exp \left(\eta_ {t} \hat {r} _ {t, i}\right) - 1 - \eta_ {t} \hat {r} _ {t, i} \right] \\ \stackrel {(\mathrm {I I})} {\leq} \mathbb {E} _ {i \sim q _ {t}} \left[ \eta_ {t} ^ {2} \hat {r} _ {t, i} ^ {2} \right] \tag {18} \\ \end{array}
$$
where in (I) we use the fact that  $\log(z) \leq z - 1$  and in (II) we use the fact that for  $x \leq 1$ , we have  $\exp(x) \leq 1 + x + x^2$ , and hence  $\exp(x) - 1 - x \leq x^2$ . For the second term in (17), we will mirror the potential argument in Bubeck et al. [2012], but with a slightly different potential function. We expand the definition of  $q_t$ :
$$
\begin{array}{l} - \frac {1}{\eta_ {t}} \log \mathbb {E} _ {i \sim q _ {t}} \exp (\eta_ {t} \hat {r} _ {t, i}) = - \frac {1}{\eta_ {t}} \log \frac {\sum_ {i = 1} ^ {M} \exp (\eta_ {t} \hat {R} _ {t , i})}{\sum_ {i = 1} ^ {M} \exp (\eta_ {t} \hat {R} _ {t - 1 , i})} \\ = - \frac {1}{\eta_ {t}} \log \frac {1}{M} \sum_ {i = 1} ^ {M} \exp (\eta_ {t} \hat {R} _ {t, i}) + \frac {1}{\eta_ {t}} \log \frac {1}{M} \sum_ {i = 1} ^ {M} \exp (\eta_ {t} \hat {R} _ {t - 1, i}) \\ = J _ {t} \left(\eta_ {t}\right) - J _ {t - 1} \left(\eta_ {t}\right), \tag {19} \\ \end{array}
$$