SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where we define  $J_{t}(\eta) = -\frac{1}{\eta}\log \frac{1}{M}\sum_{i = 1}^{M}\exp (\eta \hat{R}_{t,i})$ . We also define  $F_{t}(\eta) = \frac{1}{\eta}\log \frac{1}{M}\sum_{i = 1}^{M}\exp (-\eta \hat{R}_{t,i})$ . We observe the relation  $J(\eta) = F(-\eta)$ . From this, it follows that for any  $\eta$ , we have  $J^{\prime}(\eta) = -F^{\prime}(-\eta)\leq 0$ , by the argument in Bubeck et al. [2012, Theorem 3.1] that shows  $F^{\prime}(\eta)\geq 0$  for any  $\eta$ .
Putting together the pieces Now, we can bound (17) by inputing (18) and (19):
$$
- \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ] \leq \mathbb {E} _ {i \sim q _ {t}} \left[ \eta_ {t} \hat {r} _ {t, i} ^ {2} \right] + J _ {t} (\eta_ {t}) - J _ {t - 1} (\eta_ {t})
$$
With this, we rewrite (14) as
$$
\sum_ {t = 1} ^ {n} \hat {r} _ {t, k} - \sum_ {t = 1} ^ {n} \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ] = \sum_ {t = 1} ^ {n} \hat {r} _ {t, k} + \sum_ {t = 1} ^ {n} \mathbb {E} _ {i \sim q _ {t}} \left[ \eta_ {t} \hat {r} _ {t, i} ^ {2} \right] + \sum_ {t = 1} ^ {n} J _ {t} (\eta_ {t}) - J _ {t - 1} (\eta_ {t}) \tag {20}
$$
Potential manipulation We can do an Abel transformation on the sum of potentials in (20), namely obtaining
$$
\sum_ {t = 1} ^ {n} J _ {t} (\eta_ {t}) - J _ {t - 1} (\eta_ {t}) = \sum_ {t = 1} ^ {n - 1} (J _ {t} (\eta_ {t}) - J _ {t} (\eta_ {t + 1})) + J _ {n} (\eta_ {n}),
$$
where we used that  $J_0(\eta) = 0$ . We know  $J'(\eta) \leq 0$  and so  $J$  is decreasing and since  $\eta_{t+1} \leq \eta_t$ , we have  $J(\eta_{t+1}) \geq J(\eta_t)$  or  $(J_t(\eta_t) - J_t(\eta_{t+1})) \leq 0$ , so that for any fixed  $k$
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} J _ {t} (\eta_ {t}) - J _ {t - 1} (\eta_ {t}) \leq J _ {n} (\eta_ {n}) \leq \frac {\log (M)}{\eta_ {n}} - \frac {1}{\eta_ {n}} \log \left(\sum_ {i = 1} ^ {M} \exp \left(\eta_ {n} \hat {R} _ {n, i}\right)\right) \\ \stackrel {(*)} {\leq} \frac {\log (M)}{\eta_ {n}} - \frac {1}{\eta_ {n}} \log \left(\exp \left(\eta_ {n} \hat {R} _ {n, k}\right)\right) \\ = \frac {\log (M)}{\eta_ {n}} - \sum_ {t = 1} ^ {n} \hat {r} _ {t, k} \tag {21} \\ \end{array}
$$
where  $(\ast)$  follows because  $\exp$  is positive and  $-\log$  is decreasing (notice that we drop  $M - 1$  terms from the sum). Plugging (21) into (20), we obtain
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \hat {r} _ {t, k} - \sum_ {t = 1} ^ {n} \mathbb {E} _ {j \sim q _ {t}} [ \hat {r} _ {t, j} ] \leq \sum_ {t = 1} ^ {n} \hat {r} _ {t, k} + \sum_ {t = 1} ^ {n} \mathbb {E} _ {i \sim q _ {t}} [ \eta_ {t} \hat {r} _ {t, i} ^ {2} ] + \sum_ {t = 1} ^ {n} J _ {t} (\eta_ {t}) - J _ {t - 1} (\eta_ {t}) \\ \leq \sum_ {t = 1} ^ {n} \hat {r} _ {t, k} + \sum_ {t = 1} ^ {n} \mathbb {E} _ {i \sim q _ {t}} \left[ \eta_ {t} \hat {r} _ {t, i} ^ {2} \right] + \frac {\log (M)}{\eta_ {n}} - \sum_ {t = 1} ^ {n} \hat {r} _ {t, k} \\ \leq \sum_ {t = 1} ^ {n} \mathbb {E} _ {i \sim q _ {t}} \left[ \eta_ {t} \hat {r} _ {t, i} ^ {2} \right] + \frac {\log (M)}{\eta_ {n}} \\ \end{array}
$$
$$
= \sum_ {t = 1} ^ {n} \eta_ {t} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} ^ {2} + \frac {\log (M)}{\eta_ {n}}.
$$
We expressed in Section 5.2, that the model selection regret of ALEXP, is closely tied to the bias and variance of the reward estimates  $\hat{r}_{t,j}$ . The following lemma formalizes this claim.
Lemma 18. (Anytime Generic regret bound) If  $\eta_t$  is picked such that  $\eta_t \hat{r}_{t,j} \leq 1$  for all  $1 \leq j \leq M$  and  $1 \leq t$  almost surely, then Algorithm 1 satisfies with probability greater than  $1 - 2\delta/3$ , that simultaneously for all  $n \geq 1$
$$
\begin{array}{l} R (n, i) \leq 2 B \sum_ {t = 1} ^ {n} \gamma_ {t} + \frac {\log M}{\eta_ {n}} + \sum_ {t = 1} ^ {n} \eta_ {t} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} ^ {2} + \sum_ {t = 1} ^ {n} \left(\omega_ {t, i} + \sum_ {j = 1} ^ {M} q _ {t, j} \omega_ {t, j}\right) \\ + 1 0 B \sqrt {n \left((\log \log n B ^ {2}) _ {+} + \log (1 2 / \delta)\right)} \\ \end{array}
$$
where  $\omega_{t,i} = |r_{t,i} - \hat{r}_{t,i}|$ .
Proof of Lemma 18. Let  $\alpha_{t}$  denote the Bernoulli random variable that is equal to 1 if at step  $t$  we select actions according to  $\pi$  and 0 otherwise. At each step  $t$  with  $\alpha_{t} = 1$  ALEXP accumulates a regret of at most  $2B$ , since  $\| \pmb{\theta}\|_{\infty} \leq B$  and  $\| \phi(\cdot)\| \leq 1$ . We can decompose the regret as,
$$
R (n, i) \leq \sum_ {t = 1} ^ {n} 2 B \alpha_ {t} + (r _ {t, i} - r _ {t}) (1 - \alpha_ {t})
$$
For the first term, by Lemma 24, we have
$$
2 B \sum_ {t = 1} ^ {n} \alpha_ {t} \leq 2 B \left(\sum_ {t = 1} ^ {n} \gamma_ {t} + \frac {5}{2} \sqrt {n \left((\log \log n) _ {+} + \log (4 / \delta_ {1})\right)}\right).
$$
simultaneously for all  $n \geq 1$ , with probability  $1 - \delta_1$ . Let  $\hat{r}_t \coloneqq \sum_{j=1}^{M} q_{t,j} \hat{r}_{t,j}$ . We may re-write the second term of the regret as follows,
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \left(1 - \alpha_ {t}\right) \left(r _ {t, i} - r _ {t}\right) \leq \sum_ {t = 1} ^ {n} \left(1 - \alpha_ {t}\right) \left[ \left(r _ {t, i} - \hat {r} _ {t, i}\right) + \left(\hat {r} _ {t, i} - \hat {r} _ {t}\right) + \left(\hat {r} _ {t} - r _ {t}\right) \right] \\ \leq \sum_ {t = 1} ^ {n} \omega_ {t, i} + (1 - \alpha_ {t}) \Big [ (\hat {r} _ {t, i} - \hat {r} _ {t}) + (\hat {r} _ {t} - r _ {t}) \Big ] \\ \end{array}
$$
We bound the second term on the right hand side, using Lemma 17
$$
\sum_ {t = 1} ^ {n} (1 - \alpha_ {t}) (\hat {r} _ {t, i} - \hat {r} _ {t}) \leq \sum_ {t = 1} ^ {n} (\hat {r} _ {t, i} - \hat {r} _ {t}) \leq \frac {\log M}{\eta_ {n}} + \sum_ {t = 1} ^ {n} \eta_ {t} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} ^ {2}.
$$
As for the third term,
$$
\begin{array}{l} (1 - \alpha_ {t}) \left(\sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} - r _ {t}\right) = (1 - \alpha_ {t}) \left[ \sum_ {j = 1} ^ {M} q _ {t, j} \left(\hat {r} _ {t, j} - r _ {t, j} + r _ {t, j}\right) - r _ {t} \right] \\ \leq \sum_ {j = 1} ^ {M} q _ {t, j} \omega_ {t, j} + (1 - \alpha_ {t}) \left(r _ {t} - \sum_ {j = 1} ^ {M} q _ {t, j} r _ {t, j}\right). \\ \end{array}
$$
It remains to bound the deviation term. For all  $t$  that satisfy  $\alpha_{t} = 0$ , the action/model is selected according to  $q_{t,j}$ , therefore the conditional expectation of  $r_t$  can be written as
$$
\mathbb {E} _ {t - 1} r _ {t} = \sum_ {j - 1} ^ {M} q _ {t, j} r _ {t, j}
$$
The sequence  $X_{t} \coloneqq r_{t} - \mathbb{E}_{t-1}r_{t}$  is a martingale difference sequence adapted to the history  $H_{t}$ , since for every  $t \geq 1$ ,
$$
\mathbb {E} _ {t - 1} X _ {t} = \mathbb {E} \left[ r _ {t} - \mathbb {E} _ {t - 1} r _ {t} | H _ {t - 1} \right] = 0.
$$
Since  $r_t \leq B$ , then  $X_t \leq 2B$  almost surely, which allows for an application of anytime Azuma-Hoeffding (Lemma 26):
$$
\mathbb {P} \left(\exists n: \sum_ {t = 1} ^ {n} \left(r _ {t} - \sum_ {j = 1} ^ {M} q _ {t, j} r _ {t, j}\right) \geq \frac {5 B}{2} \sqrt {n \left((\log \log n B ^ {2}) _ {+} + \log (2 / \delta_ {2})\right)}\right) \leq \delta_ {2}
$$
which, in turn, leads us to
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \left(1 - \alpha_ {t}\right) \left(r _ {t} - \sum_ {j = 1} ^ {M} q _ {t, j} r _ {t, j}\right) \leq \sum_ {t = 1} ^ {n} \left(r _ {t} - \sum_ {j = 1} ^ {M} q _ {t, j} r _ {t, j}\right) \\ \stackrel {\text {w . h . p .}} {\leq} \frac {5 B}{2} \sqrt {n \left(\left(\log \log n B ^ {2}\right) _ {+} + \log \left(2 / \delta_ {2}\right)\right)} \\ \end{array}
$$
simultaneously for all  $n \geq 1$ . We set  $\delta_1 = \delta_2 = \delta / 3$ , take a union bound and put the terms together obtaining,
$$
\begin{array}{l} R (n, i) \leq 2 B \sum_ {t = 1} ^ {n} \gamma_ {t} + \frac {\log M}{\eta_ {n}} + \eta \sum_ {t = 1} ^ {n} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} ^ {2} + \sum_ {t = 1} ^ {n} (\omega_ {t, i} + \sum_ {j = 1} ^ {M} q _ {t, j} \omega_ {t, j}) \\ + \frac {5 B}{2} \sqrt {n ((\log \log n B ^ {2}) _ {+} + \log (6 / \delta))} + 5 B \sqrt {n ((\log \log n) _ {+} + \log (1 2 / \delta))} \\ \end{array}
$$
We upper bound the sum of last two terms to conclude the proof.
The next two lemmas bound the bias and variance terms which appear in Lemma 18.
Lemma 19 (Anytime Bound on the Bias Term). If the regularization parameter of Lasso is chosen at every step as
$$
\lambda_ {t} = \frac {2 \sigma}{\sqrt {t}} \sqrt {1 + \frac {5}{\sqrt {2}} \sqrt {d (\log (2 M / \delta) + (\log \log d) _ {+})} + \frac {1 2}{\sqrt {2}} (\log (2 M / \delta) + (\log \log d) _ {+})}
$$
and  $\gamma_{t} = O(t^{-1 / 4})$ , then with probability greater than  $1 - \delta$ , simultaneously for all  $n \geq 1$ ,
$$
\sum_ {t = 1} ^ {n} \left| \hat {r} _ {t, i} - r _ {t, i} \right| \leq n ^ {3 / 4} C _ {\min } ^ {- 1} C (M, \delta , d) \left(1 + n ^ {- 3 / 8} C _ {\min } ^ {- 1} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right)
$$
$$
C (M, \delta , d) := C \sigma \sqrt {1 + \sqrt {d (\log (M / \delta) + (\log \log d) _ {+})} + (\log (M / \delta) + (\log \log d) _ {+})}
$$
and  $C$  is an absolute constant.
Proof of Lemma 19. By the definition of the expected reward and its estimate,
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \left| \hat {r} _ {t, i} - r _ {t, i} \right| = \sum_ {t = 1} ^ {n} \left| \int_ {\mathcal {X}} \left(r (\boldsymbol {x}) - \hat {r} _ {t} (\boldsymbol {x})\right) \mathrm {d} p _ {t + 1, i} (\boldsymbol {x}) \right| \\ \leq \sum_ {t = 1} ^ {n} \int_ {\mathcal {X}} | r (\boldsymbol {x}) - \hat {r} _ {t} (\boldsymbol {x}) | \mathrm {d} p _ {t + 1, i} (\boldsymbol {x}) \\ \stackrel {\mathrm {C . S .}} {\leq} \sum_ {t = 1} ^ {n} \int_ {\mathcal {X}} \left\| \boldsymbol {\theta} - \hat {\boldsymbol {\theta}} _ {t} \right\| _ {2} \| \phi (\boldsymbol {x}) \| _ {2} \mathrm {d} p _ {t + 1, i} (\boldsymbol {x}) \\ \stackrel {\text {b d d .}} {\leq} \phi \sum_ {t = 1} ^ {n} \left\| \boldsymbol {\theta} - \hat {\boldsymbol {\theta}} _ {t} \right\| _ {2} \int_ {\mathcal {X}} \mathrm {d} p _ {t + 1, i} (\boldsymbol {x}) = \sum_ {t = 1} ^ {n} \left\| \boldsymbol {\theta} - \hat {\boldsymbol {\theta}} _ {t} \right\| _ {2}. \\ \end{array}
$$
We highlight that the Cauchy-Schwarz step may be refined. By further assuming that  $\theta_{j^{\star}}$  is bounded away from zero (also called the beta-min condition [Buhlmann and Van De Geer, 2011]) one can show that  $\pmb {\theta} - \hat{\pmb{\theta}}_t$  is a 2-sparse vector. This will then allow one to only rely on boundedness of  $\| \phi_j\|$  rather than  $\| \phi \|$  to derive the last inequality, and relax our assumption of  $\| \phi (\cdot)\| \leq 1$  to  $\| \phi_j(\cdot)\| \leq 1$  for all  $j\in [M]$ . From Theorem 3, with probability greater than  $1 - \delta /2$  simultaneously for all  $n\geq 1$
$$
\sum_ {t = 1} ^ {n} \left| \hat {r} _ {t, i} - r _ {t, i} \right| \leq \sum_ {t = 1} ^ {n} \frac {4 \sqrt {1 0} \lambda_ {t}}{\kappa^ {2} (\Phi_ {t} , 2)} = \tilde {C} (M, \delta , d) \sum_ {t = 1} ^ {n} \frac {1}{\kappa^ {2} (\Phi_ {t} , 2) \sqrt {t}}
$$
$$
\tilde {C} (M, \delta , d) := 8 \sigma \sqrt {1 + \frac {5}{\sqrt {2}} \sqrt {d (\log (4 M / \delta) + (\log \log d) _ {+})} + \frac {1 2}{\sqrt {2}} (\log (4 M / \delta) + (\log \log d) _ {+})}.
$$
From Lemma 8, there exist absolute constants  $C_1, C_2$  for which,
$$
\mathbb {P} \left(\forall t \geq 1: \kappa^ {2} (\Phi_ {t}, 2) \geq C _ {1} C _ {\min } t ^ {- 1 / 4} - C _ {2} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \geq 1 - \delta .
$$
Using Taylor approximation we observe that,  $\frac{1}{1 - x^{-1}} = 1 + x^{-1} + o(x^{-1}) = \mathcal{O}(1 + x^{-1})$ . Therefore, these exist absolute constant  $C_3, C_4$ , for which with probability greater than  $1 - \delta$  for all  $t \geq 1$
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \frac {\tilde {C} (M , \delta , d)}{\kappa^ {2} (\Phi_ {t} , 2) \sqrt {t}} \leq \sum_ {t = 1} ^ {n} \frac {\tilde {C} (M , \delta , d)}{\sqrt {t}} \frac {1}{C _ {1} C _ {\min } t ^ {- 1 / 4} - C _ {2} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}} \\ \leq \sum_ {t = 1} ^ {n} \frac {\tilde {C} (M , \delta , d)}{\sqrt {t}} \frac {\tilde {C} _ {3}}{C _ {\min} t ^ {- 1 / 4}} \left(1 + \frac {C _ {2} t ^ {- 5 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}}{C _ {1} C _ {\min} t ^ {- 1 / 4}}\right) \\ \leq \sum_ {t = 1} ^ {n} \frac {\tilde {C} _ {3} \tilde {C} (M , \delta , d) t ^ {- 1 / 4}}{C _ {\min }} \left(1 + \tilde {C} _ {4} \frac {t ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \\ = \frac {C _ {3} \tilde {C} (M , \delta , d) n ^ {3 / 4}}{C _ {\mathrm {m i n}}} \left(1 + C _ {4} \frac {n ^ {- 3 / 8}}{C _ {\mathrm {m i n}}} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right). \\ \end{array}
$$
Lemma 20 (Anytime Bound on Variance Term). Suppose  $\lambda_{t}$  is chosen according to Lemma 19,  $\gamma_{t} = \mathcal{O}(t^{-1 / 4})$  and  $\eta_t = \mathcal{O}(C_{\mathrm{min}}t^{-1 / 2} / C(M,\delta ,d))$ . Then with probability greater than  $1 - \delta$ , the following holds simultaneously for all  $n\geq 1$  and  $t\geq 1$
$$
\hat {r} _ {t, j} \leq \frac {4 \sqrt {1 0} \lambda_ {t}}{\kappa^ {2} (\Phi_ {t} , 2)} + B, \quad \forall j \in [ M ]
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \eta_ {t} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} ^ {2} \leq C _ {1} B ^ {2} C _ {\min } \sqrt {n} + C _ {2} B n ^ {1 / 4} \left(1 + \frac {n ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right) \\ + C (M, \delta , d) \frac {\log n}{C _ {\min }} \left(1 + \frac {n ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right) \\ \end{array}
$$
where  $C_i$  are absolute constants, and  $C(M, \delta, d)$  is as defined in Lemma 19, up to constant factors.
Proof of Lemma 20. We start by upper bounding  $\hat{r}_{t,j}$ . For all  $j$  and  $t$  it holds that:
$$
\hat {r} _ {t, j} = \int_ {\mathcal {X}} \left\langle \hat {\boldsymbol {\theta}} _ {t}, \phi (\boldsymbol {x}) \right\rangle \mathrm {d} p _ {t + 1, j} (\boldsymbol {x}) \leq \left\| \hat {\boldsymbol {\theta}} _ {t} \right\| \int_ {\mathcal {X}} \| \phi (\boldsymbol {x}) \| \mathrm {d} p _ {t + 1, j} (\boldsymbol {x}) \leq \left\| \hat {\boldsymbol {\theta}} _ {t} \right\| _ {2} \leq B + \left\| (\hat {\boldsymbol {\theta}} _ {t} - \boldsymbol {\theta}) \right\| _ {2}
$$
since  $\| \pmb{\theta} \|_2 \leq B$ . To bound the last term, we only need to invoke Theorem 3, which, in turn, will simultaneously bound  $\hat{r}_{t,j}$  for all  $j = 1, \dots, M$ :
$$
\mathbb {P} \left(\forall t \geq 1, \forall j \in [ M ]: \hat {r} _ {t, j} \leq \frac {4 \sqrt {1 0} \lambda_ {t}}{c _ {\kappa , t} ^ {2}} + B\right) \geq 1 - \delta
$$
Which implies for all  $t\geq 1$
$$
\sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} ^ {2} \leq \left(\frac {4 \sqrt {1 0} \lambda_ {t}}{c _ {\kappa , t} ^ {2}} + B\right) ^ {2} \sum_ {j = 1} ^ {M} q _ {t, j} = \frac {1 6 0 \lambda_ {t} ^ {2}}{c _ {\kappa , t} ^ {4}} + B ^ {2} + \frac {8 B \sqrt {1 0} \lambda_ {t}}{c _ {\kappa , t} ^ {2}}.
$$
For the last term, similar to the proof of Lemma 19 we have,
$$
\sum_ {t = 1} ^ {n} \eta_ {t} \frac {8 B \sqrt {1 0} \lambda_ {t}}{c _ {\kappa , t} ^ {2}} \leq C _ {1} B n ^ {1 / 4} \left(1 + \frac {n ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right)
$$
for some absolute constant  $C_1$ . We treat the squared term similarly,
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \eta_ {t} \frac {1 6 0 \lambda_ {t} ^ {2}}{c _ {\kappa , t} ^ {4}} \leq C (M, \delta , d) \sum_ {t = 1} ^ {n} \frac {\bar {C} _ {3}}{t C _ {\min }} \left(1 + \bar {C} _ {4} \frac {t ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \\ \leq C (M, \delta , d) \frac {C _ {3} \log n}{C _ {\min }} \left(1 + C _ {4} \frac {n ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right). \\ \end{array}
$$
Note that the last inequality is not tight. This term will not be fastest growing term in the regret, so we have little motivation to bound it tightly. Therefore,
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \eta_ {t} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} ^ {2} \leq C _ {1} B ^ {2} C _ {\min } \sqrt {n} + C _ {2} B n ^ {1 / 4} \left(1 + \frac {n ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \\ + C (M, \delta , d) \frac {\log n}{C _ {\min }} \left(1 + C _ {4} \frac {n ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right) \\ \end{array}
$$
where  $C_i$  are absolute constants.
Proof of Lemma 16. We start by conditioning on the event  $E$  that  $\eta_t$  is picked such that  $\eta_t \hat{r}_{t,j} \leq 1$  for all  $t \geq 1$  and  $j = 1, \dots, M$ . Then by application of Lemma 18 we get with probability greater than  $1 - 2\delta / 3$ ,
$$
\begin{array}{l} R (n, i) \leq 2 B \sum_ {t = 1} ^ {n} \gamma_ {t} + \frac {\log M}{\eta_ {n}} + \sum_ {t = 1} ^ {n} \eta_ {t} \sum_ {j = 1} ^ {M} q _ {t, j} \hat {r} _ {t, j} ^ {2} + \sum_ {t = 1} ^ {n} \left(\omega_ {t, i} + \sum_ {j = 1} ^ {M} q _ {t, j} \omega_ {t, j}\right) \\ + 1 0 B \sqrt {n \left(\left(\log \log n B ^ {2}\right) _ {+} + \log (1 2 / \delta)\right)} \\ \end{array}
$$
We invoke Lemma 19 and Lemma 20 with  $\delta \rightarrow \delta /3$  take a union bound, to bound the variance and  $\omega_{t,i}$  terms as well. These lemmas require one application of Theorem 3 to hold simultaneously and no additional union bound is required between them, since the randomness comes only from the confidence interval over  $\hat{\pmb{\theta}}_t$ .
$$
\begin{array}{l} R (n, i) \leq C _ {1} B n ^ {3 / 4} + \frac {\log M}{\eta_ {n}} \\ + C _ {2} B ^ {2} C _ {\min } \sqrt {n} + C _ {3} B n ^ {1 / 4} \left(1 + \frac {n ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right) \\ + C (M, \delta , d) \frac {\log n}{C _ {\min }} \left(1 + \frac {n ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right) \\ + n ^ {3 / 4} C (M, \delta , d) \left(1 + \frac {n ^ {- 3 / 8}}{C _ {\min }} \sqrt {\log (M d / \delta) + (\log \log n) _ {+}}\right) \\ + 1 0 B \sqrt {n \left(\left(\log \log n B ^ {2}\right) _ {+} + \log (1 2 / \delta)\right)} \\ \end{array}
$$
with probability greater than  $1 - \delta$ , conditioned on event  $E$ . Assuming that event  $E$  happens with probability  $1 - 2\delta$ , let  $\mathcal{B} = \mathcal{B}(\tilde{\delta}, M, d, n, B, \sigma, C_{\min})$  denote the right-hand-side of the regret inequality above.
By the chain rule we may write,
$$
\begin{array}{l} \mathbb {P} \left(\operatorname {R e g} (n, i) \leq \mathcal {B}\right) \\ \geq \mathbb {P} \left(R (n, i) \leq \mathcal {B} \mid \forall t \in [ n ], j \in [ M ]: \eta_ {t} \hat {r} _ {t, j} \leq 1\right) \mathbb {P} (\forall t \in [ n ], j \in [ M ]: \eta_ {t} \hat {r} _ {t, j} \leq 1) \\ \geq \mathbb {P} \left(R (n, i) \leq \mathcal {B} \mid \forall t \in [ n ], j \in [ M ]: \eta_ {t} \hat {r} _ {t, j} \leq 1\right) (1 - 2 \delta) \\ \geq (1 - \delta) (1 - 2 \delta) \geq 1 - 3 \delta . \\ \end{array}
$$
It remains to verify that event  $E$  is met with probability  $1 - 2\delta$ . Recall that  $\eta_t = \mathcal{O}(C_{\min} / \sqrt{t} C(M, \delta, d))$ , and that from Lemma 8 with probability  $1 - \delta$ ,
$$
\frac {C _ {\operatorname* {m i n}}}{4 \sqrt {t} C (M , \delta , d)} \leq \frac {C _ {1} C _ {\operatorname* {m i n}} t ^ {1 / 4} - C _ {2} t ^ {- 1 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}}{4 \sqrt {1 0} C (M , \delta , d)} \leq \frac {\kappa^ {2} (\Phi_ {t} , 2)}{4 \sqrt {1 0} \lambda_ {t}}
$$
Therefore, from Lemma 20, there exists  $C_{\eta}$  such that  $\eta_t = C_{\eta}C_{\min} / B\sqrt{t} C(M,\delta,d)$  satisfying,
$$
\mathbb {P} \left(\forall t \geq 1, j \in [ M ]: \eta_ {t} \hat {r} _ {t, j} \leq 1\right) \geq 1 - 2 \delta
$$
The proof is then finished by setting  $\delta \gets 3\delta$  (and updating the absolute constants).
D.2 Proof of Virtual Regret
Proposition 21. For any fixed  $\tilde{\lambda} > 0$ , there exists an absolute constant  $C_1$  such that
$$
\mathbb {P} \left(\forall t \geq 1: \left\| \hat {\boldsymbol {\beta}} _ {t, j ^ {\star}} - \boldsymbol {\theta} _ {j ^ {\star}} \right\| _ {2} \leq \omega (t, \delta , d)\right) \geq 1 - \delta .
$$
$$
\omega (t, \delta , d) := C _ {1} \sqrt {\frac {\sigma^ {2} d \log \left(\frac {t}{\tilde {\lambda} d} + 1\right) + 2 \sigma^ {2} \log (1 / \delta) + \tilde {\lambda} B ^ {2}}{\tilde {\lambda} + C _ {\mathrm {m i n}} t ^ {3 / 4}}} \left(1 + C _ {\mathrm {m i n}} ^ {- 1} t ^ {- 3 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right).
$$
Moreover, for  $u_{t,j^{\star}}(\cdot)\coloneqq \hat{\beta}_{t,j^{\star}}^{\top}\phi_{j^{\star}}(\cdot) + \omega (t,\delta ,d),$
$$
\mathbb {P} \left(\forall t \geq 1, \boldsymbol {x} \in \mathcal {X}: r (\boldsymbol {x}) \leq u _ {t, j ^ {\star}} (\boldsymbol {x})\right) \geq 1 - \delta .
$$
Proof of Proposition 21. Define for convenience  $V_{t} = \Phi_{t,j^{\star}}^{\top}\Phi_{t,j^{\star}} + \bar{\lambda} I$ . We first observe that
$$
\hat {\boldsymbol {\beta}} _ {t, j ^ {\star}} = V _ {t} ^ {- 1} \left(\Phi_ {t, j ^ {\star}}\right) ^ {\top} \boldsymbol {y} _ {t}
$$
We can apply results from Abbasi-Yadkori et al. [2011] to get an anytime-valid confidence set. Their Theorem 2 asserts that with probability  $1 - \delta$ , for all  $t \geq 1$  we have
$$
\left\| \hat {\boldsymbol {\beta}} _ {t, j ^ {\star}} - \boldsymbol {\theta} _ {j ^ {\star}} \right\| _ {V _ {t}} ^ {2} \leq \beta_ {t}
$$
$$
\beta_ {t} = 2 \sigma^ {2} \log \left(\frac {\det  (V _ {t}) ^ {1 / 2}}{\det  (\tilde {\lambda} I) ^ {1 / 2} \delta}\right) + \tilde {\lambda} B ^ {2}
$$
Clearly,  $V_{t} \succeq \lambda_{\min}(V_{t})I$ , and therefore with high probability,
$$
\left\| \hat {\boldsymbol {\beta}} _ {t, j ^ {\star}} - \boldsymbol {\theta} _ {j ^ {\star}} \right\| _ {2} \leq \sqrt {\frac {\beta_ {t}}{\lambda_ {\operatorname* {m i n}} (V _ {t})}}
$$
uniformly over time. Our assumption is that  $\| \phi_j(\pmb {x})\| \leq 1$ , and hence, denoting by  $\nu_{i}$  the eigenvalues of  $V_{t}$ , the geometric-arithmetic mean inequality yields
$$
\det  (V _ {t}) \leq \prod_ {i = 1} ^ {d} \nu_ {i} \leq \left(\frac {1}{d} \operatorname {t r a c e} (V _ {t})\right) ^ {d}.
$$
2Their theorem statement is slightly different, but they prove the stronger version we state below.
Given that
$$
\operatorname {t r a c e} (V _ {t}) = \sum_ {i = 1} ^ {d} \sum_ {s = 1} ^ {t} \left(\phi_ {j ^ {\star}} (x)\right) _ {i} ^ {2} + \tilde {\lambda} d \leq t + \tilde {\lambda} d
$$
we can conclude that
$$
\beta_ {t} \leq 2 \sigma^ {2} \log \left(\frac {(t / d + \tilde {\lambda}) ^ {d / 2}}{\tilde {\lambda} ^ {d / 2} \delta}\right) + \tilde {\lambda} B ^ {2} = d \sigma^ {2} \log \left(\frac {t}{\tilde {\lambda} d} + 1\right) + 2 \sigma^ {2} \log (1 / \delta) + \tilde {\lambda} B ^ {2}
$$
$$
\lambda_ {\min } (V _ {t}) = \lambda_ {\min } (\Phi_ {t, j} ^ {\top} \Phi_ {t, j}) + \tilde {\lambda}.
$$
Then due to Lemma 9, there exist absolute constants  $C_1$  and  $C_2$  such that for all  $t \geq 1$ ,
$$
\lambda_ {\min } (V _ {t}) \geq \tilde {\lambda} + C _ {1} C _ {\min } t ^ {3 / 4} - C _ {2} t ^ {3 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}
$$
therefore, there exists  $C_3$  and  $C_4$  such that
$$
\begin{array}{l} \frac {1}{\sqrt {\lambda_ {\mathrm {m i n}} (V _ {t})}} \leq \frac {1}{\sqrt {\tilde {\lambda} + C _ {1} C _ {\mathrm {m i n}} t ^ {3 / 4} - C _ {2} t ^ {3 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}}} \\ \leq \frac {C _ {3}}{\sqrt {\tilde {\lambda} + C _ {1} C _ {\min} t ^ {3 / 4}}} \left(1 + \frac {t ^ {3 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}}{\tilde {\lambda} + C _ {1} C _ {\min} t ^ {3 / 4}}\right) \\ \leq \frac {C _ {4}}{\sqrt {\tilde {\lambda} + C _ {1} C _ {\min} t ^ {3 / 4}}} \left(1 + C _ {\min } ^ {- 1} t ^ {- 3 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right) \\ \end{array}
$$
with high probability for all  $t \geq 1$ . Setting
$$
\omega (t, \delta , d) = C _ {5} \sqrt {\frac {\sigma^ {2} d \log \left(\frac {t}{\tilde {\lambda} d} + 1\right) + 2 \sigma^ {2} \log (1 / \delta) + \tilde {\lambda} B ^ {2}}{\tilde {\lambda} + C _ {\min} t ^ {3 / 4}}} \left(1 + C _ {\min } ^ {- 1} t ^ {- 3 / 8} \sqrt {\log (M d / \delta) + (\log \log t) _ {+}}\right)
$$