SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we prove the cumulative regret bounds stated in Section 7. We prove the data-dependent regret bound in Thm. 7.5. We also prove the data-independent regret bound in Thm. 7.6 and another data-independent regret bound, which holds for more general choices of the mixture distributions and the  $\alpha$  parameter.
For convenience, we use some more compact notation in this section. For a symmetric positive semi-definite matrix  $\mathbf{A}$  and vector  $\mathbf{x}$ , let
$$
\left\| \boldsymbol {x} \right\| _ {\boldsymbol {A}} := \sqrt {\boldsymbol {x} ^ {\top} \boldsymbol {A} \boldsymbol {x}}.
$$
Before presenting the proof of the main results, we state some useful lemmas.
Lemma D.1 (Donsker-Varadhan Change of Measure (Donsker & Varadhan, 1976)). For any set  $\mathcal{X}$ , any measurable function  $h: \mathcal{X} \to \mathbb{R}$  and any probability distribution  $P \in \mathcal{P}(\mathcal{X})$  (i.e., any distribution on  $\mathcal{X}$ ), such that  $\mathbb{E}_{x \sim P}[e^{h(x)}] < \infty$ , we have
$$
\sup  _ {Q \in \mathcal {P} (\mathcal {X})} \left\{\underset {x \sim Q} {\mathbb {E}} [ h (x) ] - D _ {\mathrm {K L}} (Q | | P) \right\} = \ln \left(\underset {x \sim P} {\mathbb {E}} \left[ e ^ {h (x)} \right]\right). \tag {25}
$$
By rearranging (25), we have
$$
\inf  _ {Q \in \mathcal {P} (\mathcal {X})} \left\{\underset {x \sim Q} {\mathbb {E}} [ h (x) ] + D _ {\mathrm {K L}} (Q | | P) \right\} = - \ln \left(\underset {x \sim P} {\mathbb {E}} \left[ e ^ {- h (x)} \right]\right). \tag {26}
$$
Lemma D.2 (Determinant-Trace Inequality (Abbasi-Yadkori et al., 2011)). If assumption 7.3 holds (i.e.  $\| \phi (a)\| _2\leq L$ ), then for any  $\gamma >0$
$$
\ln \left(\det \left(\gamma \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) \leq d \ln \left(1 + \gamma t L ^ {2} / d\right). \tag {27}
$$
The Determinant-Trace Inequality in Lemma 10 of (Abbasi-Yadkori et al., 2011) is stated in the form
$$
\det  \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \frac {1}{\gamma} \mathbb {1}\right) \leq (1 / \gamma + t L ^ {2} / d) ^ {d}. \tag {28}
$$
Since  $\operatorname*{det}\left(\gamma \Phi_t^\top \Phi_t + \mathbb{1}\right) = \operatorname*{det}\left(\Phi_t^\top \Phi_t + (1 / \gamma)\mathbb{1}\right) / \operatorname*{det}\left((1 / \gamma)\mathbb{1}\right)$ , the statement in (27) follows from (28).
Lemma D.3. For any  $\sigma > 0$  and any  $\sigma_0 > 0$ , define  $\boldsymbol{\Sigma}_t = \left(\frac{1}{\sigma^2}\Phi_t^\top \Phi_t + \frac{1}{\sigma_0^2}\mathbb{1}\right)^{-1}$ . We have
$$
\operatorname {t r} \left(\Phi_ {t} ^ {\top} \Phi_ {t} \boldsymbol {\Sigma} _ {t}\right) = \sigma^ {2} d - \frac {\sigma^ {2}}{\sigma_ {0} ^ {2}} \operatorname {t r} (\boldsymbol {\Sigma} _ {t}) \leq \sigma^ {2} d.
$$
Proof. Since  $\pmb{\Sigma}_t$  is positive-definite, its trace is positive. Now
$$
\begin{array}{l} \operatorname {t r} \left(\Phi_ {t} ^ {\top} \Phi_ {t} \boldsymbol {\Sigma} _ {t}\right) = \sigma^ {2} \operatorname {t r} \left(\frac {1}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} \left(\frac {1}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \frac {1}{\sigma_ {0} ^ {2}} \mathbb {1}\right) ^ {- 1}\right) \\ = \sigma^ {2} \operatorname {t r} \left(\left(\frac {1}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \frac {1}{\sigma_ {0} ^ {2}} \mathbb {1}\right) \left(\frac {1}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \frac {1}{\sigma_ {0} ^ {2}} \mathbb {1}\right) ^ {- 1} - \frac {1}{\sigma_ {0} ^ {2}} \left(\frac {1}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \frac {1}{\sigma_ {0} ^ {2}} \mathbb {1}\right) ^ {- 1}\right) \\ = \sigma^ {2} d - \frac {\sigma^ {2}}{\sigma_ {0} ^ {2}} \mathrm {t r} (\pmb {\Sigma} _ {t}) \\ \leq \sigma^ {2} d. \\ \end{array}
$$
Lemma D.4. For any  $\sigma > 0$  and any  $\sigma_0 > 0$ , the matrix  $\boldsymbol{\Sigma}_t = \left(\frac{1}{\sigma^2}\Phi_t^\top \Phi_t + \frac{1}{\sigma_0^2}\mathbb{1}\right)^{-1}$  satisfies
$$
\operatorname {t r} \left(\boldsymbol {\Sigma} _ {t}\right) \leq \frac {d}{\sigma_ {0} ^ {2}}.
$$
Proof. Let  $\{\gamma_i\}_{i=1}^d$  denote the eigenvalues of  $\Phi_t^\top \Phi_t$ . Since  $\Phi_t^\top \Phi_t$  is positive semi-definite, its eigenvalues are real and non-negative. From the definition of eigenvalues, one can verify that the eigenvalues of  $\pmb{\Sigma}_t$  are  $\left\{\frac{\sigma^2}{\gamma_i + \sigma^2 / \sigma_0^2}\right\}_{i=1}^d$ . Using this, we have
$$
\mathrm {t r} (\pmb {\Sigma} _ {t}) = \sum_ {i = 1} ^ {d} \frac {\sigma^ {2}}{\gamma_ {i} + \sigma^ {2} / \sigma_ {0} ^ {2}} \leq \sum_ {i = 1} ^ {d} \frac {\sigma^ {2}}{\sigma^ {2} / \sigma_ {0} ^ {2}} = \frac {d}{\sigma_ {0} ^ {2}}.
$$
Lemma D.5. Let  $\epsilon_{t}$  denote the vector containing the first  $t$  noise variables (so  $\boldsymbol{r}_t = \Phi_t\pmb{\theta}^* + \epsilon_t$ ). For any  $\alpha > 0$ , we have
$$
\begin{array}{l} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} \leq \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {\left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1}} ^ {2} \\ + 2 \alpha \left\| \boldsymbol {\theta} ^ {*} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}} \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}}. \\ \end{array}
$$
Proof. Using  $r_t = \Phi_t\pmb{\theta}^* + \pmb{\epsilon}_t$ , we have
$$
\left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) = \boldsymbol {\epsilon} _ {t} ^ {\top} \boldsymbol {\epsilon} _ {t},
$$
$$
\begin{array}{l} - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} (\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} = - (\Phi_ {t} \boldsymbol {\theta} ^ {*} + \boldsymbol {\epsilon} _ {t}) ^ {\top} (\Phi_ {t} \boldsymbol {\theta} ^ {*} + \boldsymbol {\epsilon} _ {t}) \\ + \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} + \boldsymbol {\epsilon} _ {t}\right) ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} + \boldsymbol {\epsilon} _ {t}\right) \\ = - \boldsymbol {\epsilon} _ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} - 2 \boldsymbol {\theta} ^ {* \top} \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} - \boldsymbol {\theta} ^ {* \top} \Phi_ {t} ^ {\top} \Phi_ {t} \boldsymbol {\theta} ^ {*} + \boldsymbol {\theta} ^ {* \top} \Phi_ {t} ^ {\top} \Phi_ {t} (\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1} \Phi_ {t} ^ {\top} \Phi_ {t} \boldsymbol {\theta} ^ {*} \\ + 2 \pmb {\theta} ^ {* \top} \Phi_ {t} ^ {\top} \Phi_ {t} (\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1} \Phi_ {t} ^ {\top} \pmb {\epsilon} _ {t} + \pmb {\epsilon} _ {t} ^ {\top} \Phi_ {t} (\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1} \Phi_ {t} ^ {\top} \pmb {\epsilon} _ {t} \\ \leq - \boldsymbol {\epsilon} _ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} - 2 \boldsymbol {\theta} ^ {* \top} \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} + 2 \boldsymbol {\theta} ^ {* \top} \Phi_ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \\ + \boldsymbol {\epsilon} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \\ = - \boldsymbol {\epsilon} _ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} - 2 \alpha \boldsymbol {\theta} ^ {* \top} (\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} + \boldsymbol {\epsilon} _ {t} ^ {\top} \Phi_ {t} (\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t}. \\ \end{array}
$$
Using the Cauchy-Schwarz inequality, we have
$$
| \boldsymbol {\theta} ^ {* ^ {\top}} (\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} | \leq \| \boldsymbol {\theta} ^ {*} \| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}} \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}}.
$$
$$
- 2 \alpha \boldsymbol {\theta} ^ {*} ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \leq 2 \alpha \left\| \boldsymbol {\theta} ^ {*} \right\| _ {\left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1}} \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {\left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1}},
$$
$$
\begin{array}{l} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} \leq \boldsymbol {\epsilon} _ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} - \boldsymbol {\epsilon} _ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} + \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {\left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1}} ^ {2} \\ + 2 \alpha \left\| \boldsymbol {\theta} ^ {*} \right\| _ {\left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1}} \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {\left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1}} \\ = \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}} ^ {2} + 2 \alpha \left\| \boldsymbol {\theta} ^ {*} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}} \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}}. \\ \end{array}
$$
Theorem D.6 (Self-Normalised Bound for Vector-Valued Martingales (Theorem 1 of (Abbasi-Yadkori et al., 2011))). Let  $(\mathcal{H}_t|t\geq 0)$  be a filtration. Let  $(\epsilon_t|t\geq 1)$  be a real-valued stochastic process such that  $\epsilon_{t}$  is  $\mathcal{H}_t$ -measurable and  $\epsilon_{t}$  is conditionally  $\sigma$ -sub-Gaussian for some  $\sigma >0$ . Let  $(\phi (a_{t})|t\geq 1)$  be an  $\mathbb{R}^d$ -valued stochastic process such that  $\phi (a_{t})$  is  $\mathcal{H}_{t - 1}$ -measurable. For any  $\delta \in (0,1]$  and any  $\alpha >0$ , with probability at least  $1 - \delta$
$$
\forall t \geq 0, \quad \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}} ^ {2} \leq \sigma^ {2} \ln \left(\det  \left(\frac {1}{\alpha} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) + 2 \sigma^ {2} \ln (1 / \delta).
$$
Lemma D.7. For any symmetric positive semi-definite matrix  $A$  with largest eigenvalue  $\gamma_{\mathrm{max}}$ , we have
$$
\left\| \boldsymbol {x} \right\| _ {\boldsymbol {A}} ^ {2} \leq \gamma_ {\max } \left\| \boldsymbol {x} \right\| _ {2} ^ {2}.
$$
Proof. Let  $\{\gamma_i\}_{i=1}^d$  and  $\{\pmb{v}_i\}_{i=1}^d$  be the eigenvalues and eigenvectors of  $\pmb{A}$ . Since  $\{\pmb{v}_i\}_{i=1}^d$  form a basis, there are constants  $\{c_i\}_{i=1}^d$  such that  $\pmb{x} = \sum_{i=1}^d c_i \pmb{v}_i$ . We have
$$
\| \boldsymbol {x} \| _ {\boldsymbol {A}} ^ {2} = \sum_ {i = 1, j = 1} ^ {d} c _ {i} c _ {j} \boldsymbol {v} _ {i} ^ {\top} \boldsymbol {A} \boldsymbol {v} _ {j} = \sum_ {i = 1, j = 1} ^ {d} \gamma_ {j} c _ {i} c _ {j} \boldsymbol {v} _ {i} ^ {\top} \boldsymbol {v} _ {j} \leq \gamma_ {\max } \sum_ {i = 1, j = 1} ^ {d} c _ {i} c _ {j} \boldsymbol {v} _ {i} ^ {\top} \boldsymbol {v} _ {j} = \gamma_ {\max } \| \boldsymbol {x} \| _ {2} ^ {2}.
$$
Lemma D.8. For all  $x\geq 0$
$$
\min (1, x) \leq \frac {1}{\ln (2)} \ln (1 + x).
$$
Proof. Since  $\ln (1 + x) / \ln (2)$  is monotonically increasing in  $x$ , we only need to prove that  $x \leq \ln (1 + x) / \ln (2)$  for all  $x \in [0, 1]$ . For any positive constant  $a$ , the function  $a \ln (1 + x)$  is concave on the domain  $[0, 1]$ . Therefore, if  $x \leq a \ln (1 + x)$  at the end points  $x = 0$  and  $x = 1$ , then  $x \leq a \ln (1 + x)$  for every  $x \in [0, 1]$ . At  $x = 0$ , we have  $a \ln (1 + x) = 0$  for any  $a$ , which means we can choose the smallest  $a$  such that  $1 \leq a \ln (1 + 1)$ . By rearranging this inequality, we obtain  $a \geq 1 / \ln (2)$ .
D.1 Data-Dependent Regret Bound
First, we show that the cumulative regret of both of our algorithms can be upper bounded by the sum of the widths of the UCB/LCBs that they use. Let
$$
\mathrm {U C B} _ {\Theta_ {t}} (a) = \max  _ {\boldsymbol {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\}, \quad \text {a n d} \quad \mathrm {L C B} _ {\Theta_ {t}} (a) = \min  _ {\boldsymbol {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\}.
$$
In words,  $\mathrm{UCB}_{\Theta_t}(a)$  and  $\mathrm{LCB}_{\Theta_t}(a)$  are the upper and lower confidence bounds used by CMM-UCB (evaluated at  $a$ ). Similarly, let
$$
\mathrm {A U C B} _ {\Theta_ {t}} (a) = \phi (a) ^ {\top} \widehat {\boldsymbol {\theta}} _ {\alpha , t} + R _ {\mathrm {A M M}, t} \| \phi (a) \| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}},
$$
$$
\mathrm {A L C B} _ {\Theta_ {t}} (a) = \phi (a) ^ {\top} \widehat {\boldsymbol {\theta}} _ {\alpha , t} - R _ {\mathrm {A M M}, t} \| \phi (a) \| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}}.
$$
$\mathrm{AUCB}_{\Theta_t}(a)$  and  $\mathrm{ALCB}_{\Theta_t}(a)$  are the analytic upper and lower confidence bounds used by AMM-UCB. Lemma D.9 shows that the cumulative regret of CMM-UCB and AMM-UCB can be upper bounded by the sum of the widths (UCB minus LCB) of the confidence bounds that they use.
Lemma D.9. Suppose the actions  $a_1, a_2, \ldots$  are selected by the CMM-UCB algorithm. For any adaptive sequence of mixture distributions  $P_t = \mathcal{N}(\boldsymbol{\mu}_t, \boldsymbol{T}_t)$  and any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$
$$
\forall T \geq 1, \quad \sum_ {t = 1} ^ {T} \Delta \left(a _ {t}\right) \leq \sum_ {t = 1} ^ {T} \mathrm {U C B} _ {\Theta_ {t - 1}} \left(a _ {t}\right) - \mathrm {L C B} _ {\Theta_ {t - 1}} \left(a _ {t}\right). \tag {29}
$$
Suppose the actions  $a_1, a_2, \ldots$  are selected by the AMM-UCB algorithm. For any adaptive sequence of mixture distributions  $P_t = \mathcal{N}(\pmb{\mu}_t, \pmb{T}_t)$  and any  $\delta \in (0, 1]$ , with probability at least  $1 - \delta$
$$
\forall \alpha > 0, T \geq 1, \quad \sum_ {t = 1} ^ {T} \Delta \left(a _ {t}\right) \leq \sum_ {t = 1} ^ {T} \mathrm {A U C B} _ {\Theta_ {t - 1}} \left(a _ {t}\right) - \mathrm {A L C B} _ {\Theta_ {t - 1}} \left(a _ {t}\right). \tag {30}
$$
Proof. Using Cor. 5.2 (i.e. the fact that  $\Theta_1, \Theta_2, \ldots$  is a confidence sequence), for any adaptive sequence of mixture distributions  $P_t = \mathcal{N}(\pmb{\mu}_t, \pmb{T}_t)$  and any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$
$$
\forall a \in \mathcal {A}, t \geq 1, \quad \mathrm {L C B} _ {\Theta_ {t - 1}} (a) \leq \phi (a) ^ {\top} \boldsymbol {\theta} ^ {*} \leq \mathrm {U C B} _ {\Theta_ {t - 1}} (a).
$$
Using Thm. 6.1, this implies
$$
\forall \alpha > 0, a \in \mathcal {A}, t \geq 1, \quad \mathrm {A L C B} _ {\Theta_ {t - 1}} (a) \leq \phi (a) ^ {\top} \boldsymbol {\theta} ^ {*} \leq \mathrm {A U C B} _ {\Theta_ {t - 1}} (a).
$$
Let  $a_1, a_2, \ldots$  be the actions selected by CMM-UCB, i.e.  $a_t = \operatorname{argmax}_{a \in \mathcal{A}_t} \left\{ \mathrm{UCB}_{\Theta_{t-1}}(a) \right\}$ . Then, with probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \Delta (a _ {t}) = \sum_ {t = 1} ^ {T} \phi \left(a _ {t} ^ {*}\right) ^ {\top} \boldsymbol {\theta} ^ {*} - \phi \left(a _ {t}\right) ^ {\top} \boldsymbol {\theta} ^ {*} \\ \leq \sum_ {t = 1} ^ {T} \mathrm {U C B} _ {\Theta_ {t - 1}} \left(a _ {t} ^ {*}\right) - \mathrm {L C B} _ {\Theta_ {t - 1}} \left(a _ {t}\right) \\ \leq \sum_ {t = 1} ^ {T} \mathrm {U C B} _ {\Theta_ {t - 1}} (a _ {t}) - \mathrm {L C B} _ {\Theta_ {t - 1}} (a _ {t}). \\ \end{array}
$$
Now, let  $a_1, a_2, \ldots$  be the actions selected by AMM-UCB, i.e.  $a_t = \operatorname{argmax}_{a \in \mathcal{A}_t} \left\{ \mathrm{AUCB}_{\Theta_{t-1}}(a) \right\}$ . Then, with probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \Delta (a _ {t}) = \sum_ {t = 1} ^ {T} \phi (a _ {t} ^ {*}) ^ {\top} \boldsymbol {\theta} ^ {*} - \phi (a _ {t}) ^ {\top} \boldsymbol {\theta} ^ {*} \\ \leq \sum_ {t = 1} ^ {T} \mathrm {A U C B} _ {\Theta_ {t - 1}} \left(a _ {t} ^ {*}\right) - \mathrm {A L C B} _ {\Theta_ {t - 1}} \left(a _ {t}\right) \\ \leq \sum_ {t = 1} ^ {T} \mathrm {A U C B} _ {\Theta_ {t - 1}} (a _ {t}) - \mathrm {A L C B} _ {\Theta_ {t - 1}} (a _ {t}). \\ \end{array}
$$
Since  $\forall \alpha > 0, a \in \mathcal{A}$  and  $t \geq 1$ ,  $\mathrm{AUCB}_{\Theta_{t-1}}(a) \geq \mathrm{UCB}_{\Theta_{t-1}}(a)$  and  $\mathrm{ALCB}_{\Theta_{t-1}}(a) \leq \mathrm{LCB}_{\Theta_{t-1}}(a)$ , (29) implies that (30) also holds when  $a_1, a_2, \ldots$  are the actions selected by CMM-UCB.
Proof of Theorem 7.5. We start by using Lemma D.9. Suppose  $a_1, a_2, \ldots$  are the actions selected by CMM-UCB or AMM-UCB. For any adaptive sequence of mixture distributions  $P_t = \mathcal{N}(\pmb{\mu}_t, \pmb{T}_t)$  and any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$
$$
\forall \alpha > 0, T \geq 1, \quad \sum_ {t = 1} ^ {T} \Delta (a _ {t}) \leq \sum_ {t = 1} ^ {T} \mathrm {A U C B} _ {\Theta_ {t - 1}} (a _ {t}) - \mathrm {A L C B} _ {\Theta_ {t - 1}} (a _ {t}).
$$
Using the definitions of  $\mathrm{AUCB}_{\Theta_{t-1}}(a_t)$  and  $\mathrm{ALCB}_{\Theta_{t-1}}(a_t)$ , we have
$$
\forall \alpha > 0, T \geq 1, \quad \sum_ {t = 1} ^ {T} \Delta (a _ {t}) \leq \sum_ {t = 1} ^ {T} 2 R _ {\mathrm {A M M}, t - 1} \| \phi (a _ {t}) \| _ {(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \alpha \mathbb {1}) ^ {- 1}}.
$$
D.2 Data-Independent Regret Bound
To establish data-independent regret bounds, we first prove data-independent upper bounds on the radius  $R_{\mathrm{AMM},t}$  and the norms  $\| \phi (a_t)\|_{(\Phi_{t - 1}^\top \Phi_{t - 1} + \alpha \mathbb{1})^{-1}}$ . Then, we take the data-dependent regret bound in Lemma D.9 and substitute in these bounds on the radius and the norms.
D.2.1 Bounding the Radius
Lemma D.10. If, for any  $c > 0$ , the sequence of mixture distributions is  $P_{t} = \mathcal{N}(\mathbf{0}, c\Phi_{t}\Phi_{t}^{\top})$  and  $\alpha = \sigma^{2} / c$ , then
$$
R _ {\mathrm {A M M}, t} ^ {2} \leq \sigma^ {2} d \ln \left(1 + \frac {c t L ^ {2}}{\sigma^ {2} d}\right) + \frac {\sigma^ {2} B ^ {2}}{c} + 2 \sigma^ {2} \ln (1 / \delta). \tag {31}
$$
Proof. In Equation (24), we already saw that for this choice of  $\alpha$  and the mixture distributions, we have
$$
R _ {\mathrm {A M M}, t} ^ {2} = \sigma^ {2} \ln \left(\det \left(\frac {c}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) + \frac {\sigma^ {2} B ^ {2}}{c} + 2 \sigma^ {2} \ln (1 / \delta).
$$
To obtain a data-independent upper bound on the radius, all that remains is to upper bound  $\ln \left(\operatorname*{det}\left(\frac{c}{\sigma^2}\Phi_t^\top \Phi_t + \mathbb{1}\right)\right)$  by a data-independent quantity. Using Lemma D.2, we have
$$
\left. \right. \ln \left(\det \left(\frac {c}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) \leq d \ln \left(1 + \frac {c t L ^ {2}}{\sigma^ {2} d}\right).
$$
$$
R _ {\mathrm {A M M}, t} ^ {2} \leq \sigma^ {2} d \ln \left(1 + \frac {c t L ^ {2}}{\sigma^ {2} d}\right) + \frac {\sigma^ {2} B ^ {2}}{c} + 2 \sigma^ {2} \ln (1 / \delta).
$$
Lemma D.11. If, for any  $\theta_0\in \mathbb{R}^d$  and any  $\sigma_0 > 0$ , the sequence of mixture distributions is  $P_{t} = \mathcal{N}(\Phi_{t}\pmb{\theta}_{0},\sigma_{0}^{2}\Phi_{t}\Phi_{t}^{\top})$ , then for any  $\delta \in (0,1]$  and any  $\alpha >0$ , with probability at least  $1 - \delta$ , for all  $t\geq 1$
$$
\begin{array}{l} R _ {\mathrm {A M M}, t} ^ {2} \leq \sigma^ {2} d + \frac {\sigma^ {2}}{\sigma_ {0} ^ {2}} \| \boldsymbol {\theta} ^ {*} - \boldsymbol {\theta} _ {0} \| _ {2} ^ {2} + \sigma^ {2} d \ln \left(1 + \frac {t \sigma_ {0} ^ {2} L ^ {2}}{\sigma^ {2} d}\right) + \alpha B ^ {2} + 4 \sigma^ {2} \ln (1 / \delta) \\ + \sigma^ {2} d \ln (1 + t L ^ {2} / (\alpha d)) + 2 \sqrt {\alpha} \| \boldsymbol {\theta} ^ {*} \| _ {2} \sqrt {\sigma^ {2} d \ln (1 + t L ^ {2} / (\alpha d)) + 2 \sigma^ {2} \ln (1 / \delta)}. \\ \end{array}
$$
Proof. In App. B.2 (see Equation 16), we saw that the squared radius  $R_{\mathrm{MM},t}^2$  can be written as
$$
R _ {\mathrm {M M}, t} ^ {2} = - 2 \sigma^ {2} \ln \left(\underset {\boldsymbol {f} _ {t} \sim \mathcal {N} \left(\Phi_ {t} \boldsymbol {\theta} _ {0}, \sigma_ {0} ^ {2} \Phi_ {t} \Phi_ {t} ^ {\top}\right)} {\mathbb {E}} \left[ \exp \left(- \frac {1}{2 \sigma^ {2}} \left(\boldsymbol {f} _ {t} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\boldsymbol {f} _ {t} - \boldsymbol {r} _ {t}\right)\right) \right]\right) + 2 \sigma^ {2} \ln (1 / \delta). \tag {32}
$$
Using the substitution  $\Phi_t\pmb {\theta} = \pmb {f}_t$  (32) is equivalent to
$$
R _ {\mathrm {M M}, t} ^ {2} = - 2 \sigma^ {2} \ln \left(\underset {\boldsymbol {\theta} \sim \mathcal {N} \left(\boldsymbol {\theta} _ {0}, \sigma_ {0} ^ {2} \mathbb {1}\right)} {\mathbb {E}} \left[ \exp \left(- \frac {1}{2 \sigma^ {2}} \left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right)\right) \right]\right) + 2 \sigma^ {2} \ln (1 / \delta). \tag {33}
$$
Using the Donsker-Varadhan change of measure inequality (specifically (26)), the first term on the right-hand-side of (33) is equal to
$$
\inf _ {Q \in \mathcal {P} (\mathbb {R} ^ {d})} \left\{\underset {\pmb {\theta} \sim Q} {\mathbb {E}} \left[ (\Phi_ {t} \pmb {\theta} - \pmb {r} _ {t}) ^ {\top} (\Phi_ {t} \pmb {\theta} - \pmb {r} _ {t}) \right] + 2 \sigma^ {2} D _ {\mathrm {K L}} (Q | | \mathcal {N} (\pmb {\theta} _ {0}, \sigma_ {0} ^ {2} \mathbb {1})) \right\}.
$$
If we evaluate this at any specific distribution  $Q$ , we obtain an upper bound on the infimum over  $Q$ . We choose  $Q = \mathcal{N}(\pmb{\theta}^{*}, \pmb{\Sigma}_{t})$ , where  $\pmb{\Sigma}_{t} = (\frac{1}{\sigma^{2}} \Phi_{t}^{\top} \Phi_{t} + \frac{1}{\sigma_{0}^{2}} \mathbb{1})^{-1}$ . Combining everything so far, we have