SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} R _ {\mathrm {M M}, t} ^ {2} \leq \underset {\boldsymbol {\theta} \sim \mathcal {N} (\boldsymbol {\theta} ^ {*}, \boldsymbol {\Sigma} _ {t})} {\mathbb {E}} \left[ \left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) \right] + 2 \sigma^ {2} D _ {\mathrm {K L}} \left(\mathcal {N} \left(\boldsymbol {\theta} ^ {*}, \boldsymbol {\Sigma} _ {t}\right) | | \mathcal {N} \left(\boldsymbol {\theta} _ {0}, \sigma_ {0} ^ {2} \mathbb {1}\right)\right) + 2 \sigma^ {2} \ln (1 / \delta) \\ = \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) + \operatorname {t r} \left(\Phi_ {t} ^ {\top} \Phi_ {t} \boldsymbol {\Sigma} _ {t}\right) + \frac {\sigma^ {2}}{\sigma_ {0} ^ {2}} \operatorname {t r} \left(\boldsymbol {\Sigma} _ {t}\right) \\ - \sigma^ {2} d + \frac {\sigma^ {2}}{\sigma_ {0} ^ {2}} \| \boldsymbol {\theta} ^ {*} - \boldsymbol {\theta} _ {0} \| _ {2} ^ {2} + \sigma^ {2} \ln \left(\frac {\det \left(\boldsymbol {\Sigma} _ {t} ^ {- 1}\right)}{\det \left((1 / \sigma_ {0} ^ {2}) \mathbb {1}\right)}\right) + 2 \sigma^ {2} \ln (1 / \delta). \tag {34} \\ \end{array}
$$
Using Lemma D.3, we have
$$
\operatorname {t r} \left(\Phi_ {t} ^ {\top} \Phi_ {t} \boldsymbol {\Sigma} _ {t}\right) \leq \sigma^ {2} d.
$$
Using Lemma D.4, we have
$$
\frac {\sigma^ {2}}{\sigma_ {0} ^ {2}} \operatorname {t r} (\boldsymbol {\Sigma} _ {t}) \leq \sigma^ {2} d.
$$
Using Lemma D.2, we have
$$
\ln \left(\frac {\mathrm {d e t} (\pmb {\Sigma} _ {t} ^ {- 1})}{\mathrm {d e t} ((1 / \sigma_ {0} ^ {2}) \mathbb {1})}\right) = \ln \left(\mathrm {d e t} \left(\frac {\sigma_ {0} ^ {2}}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) \leq d \ln \left(1 + \frac {\sigma_ {0} ^ {2} t L ^ {2}}{\sigma^ {2} d}\right).
$$
The bound on  $R_{\mathrm{MM},t}^2$  in (34) becomes
$$
R _ {\mathrm {M M}, t} ^ {2} \leq (\Phi_ {t} \pmb {\theta} ^ {*} - \pmb {r} _ {t}) ^ {\top} (\Phi_ {t} \pmb {\theta} ^ {*} - \pmb {r} _ {t}) + \frac {\sigma^ {2}}{\sigma_ {0} ^ {2}} \| \pmb {\theta} ^ {*} - \pmb {\theta} _ {0} \| _ {2} ^ {2} + \sigma^ {2} d + \sigma^ {2} d \ln \left(1 + \frac {\sigma_ {0} ^ {2} t L ^ {2}}{\sigma^ {2} d}\right) + 2 \sigma^ {2} \ln (1 / \delta).
$$
$$
\begin{array}{l} R _ {\mathrm {A M M}, t} ^ {2} \leq \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) + \frac {\sigma^ {2}}{\sigma_ {0} ^ {2}} \| \boldsymbol {\theta} ^ {*} - \boldsymbol {\theta} _ {0} \| _ {2} ^ {2} + \sigma^ {2} d + \sigma^ {2} d \ln \left(1 + \frac {\sigma_ {0} ^ {2} t L ^ {2}}{\sigma^ {2} d}\right) + 2 \sigma^ {2} \ln (1 / \delta) \\ + \alpha B ^ {2} - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t}. \tag {35} \\ \end{array}
$$
Finally, using Lemma D.5, then Theorem D.6 and Lemma D.7, and then Lemma D.2, for any  $\delta \in (0,1]$  and any  $\alpha > 0$ , with probability at least  $1 - \delta$ , for all  $t \geq 0$  simultaneously
$$
\begin{array}{l} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t}\right) - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} \\ \leq \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}} ^ {2} + 2 \alpha \left\| \boldsymbol {\theta} ^ {*} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}} \left\| \Phi_ {t} ^ {\top} \boldsymbol {\epsilon} _ {t} \right\| _ {(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}) ^ {- 1}} \\ \leq \sigma^ {2} \ln \left(\det  \left(\frac {1}{\alpha} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) + 2 \sigma^ {2} \ln (1 / \delta) + 2 \sqrt {\alpha} \| \boldsymbol {\theta} ^ {*} \| _ {2} \sigma \sqrt {\ln \left(\det  \left(\frac {1}{\alpha} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) + 2 \ln (1 / \delta)} \\ \leq \sigma^ {2} d \ln \left(\det \left(1 + \frac {t L ^ {2}}{\alpha d}\right)\right) + 2 \sigma^ {2} \ln (1 / \delta) + 2 \sqrt {\alpha} \| \boldsymbol {\theta} ^ {*} \| _ {2} \sigma \sqrt {d \ln \left(\det \left(1 + \frac {t L ^ {2}}{\alpha d}\right)\right) + 2 \ln (1 / \delta)}. \\ \end{array}
$$
Substituting this into (35), we have
D.2.2 Bounding the Sum of Norms
We use the following upper bound on the sum of the squared norms.
Lemma D.12 (Lemma 11 of (Abbasi-Yadkori et al., 2011)). For any  $\alpha > 0$ , we have
$$
\sum_ {t = 1} ^ {T} \min  \left(1, \| \phi (a _ {t}) \| _ {\left(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \alpha \mathbb {1}\right) ^ {- 1}} ^ {2}\right) \leq \frac {1}{\ln (2)} d \ln \left(1 + \frac {T L ^ {2}}{\alpha d}\right).
$$
In Lemma 11 of (Abbasi-Yadkori et al., 2011),  $1 / \ln(2) \approx 1.44$  is replaced with 2. We achieve an improved constant by using Lemma D.8 instead of the looser bound  $\min(1, x) \leq 2\ln(1 + x)$ , for  $x \geq 0$ .
D.2.3 Regret Bounds
We are now ready to prove our data-independent regret bounds.
Proof of Theorem 7.6. Following the same steps as in the proof of Lemma D.9, we can also obtain the following data-dependent bound on the per-round regret for actions selected by CMM-UCB or AMM-UCB. For the mixture distributions  $P_{t} = \mathcal{N}(\mathbf{0}_{t},c\Phi_{t}\Phi_{t}^{\top})$ $\alpha = \sigma^2 /c$  and any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$
$$
\forall t \geq 1, \quad \Delta \left(a _ {t}\right) \leq 2 R _ {\mathrm {A M M}, t - 1} \| \phi \left(a _ {t}\right) \| _ {\left(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \frac {\sigma^ {2}}{c} \mathbb {1}\right) ^ {- 1}}. \tag {36}
$$
From Assumption 7.4  $(\phi(a)^\top \pmb{\theta}^* \in [-C, C])$ , we have another bound on the per-round regret
$$
\Delta \left(a _ {t}\right) \leq 2 C. \tag {37}
$$
The combination of (36) and (37) yields
$$
\begin{array}{l} \Delta (a _ {t}) \leq \min  (2 C, 2 R _ {\mathrm {A M M}, t - 1} \| \phi (a _ {t}) \| _ {\left(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \frac {\sigma^ {2}}{c} \mathbb {1}\right) ^ {- 1}}) \\ \leq 2 \max  (C, R _ {\mathrm {A M M}, t - 1}) \min  (1, \| \phi (a _ {t}) \| _ {\left(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \frac {\sigma^ {2}}{c} \mathbb {1}\right) ^ {- 1}}). \\ \end{array}
$$
Starting with the Cauchy-Schwarz inequality, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \Delta \left(a _ {t}\right) \leq \sqrt {T \sum_ {t = 1} ^ {T} \Delta \left(a _ {t}\right) ^ {2}} \tag {38} \\ \leq \sqrt {T \sum_ {t = 1} ^ {T} 4 \max  \left(C ^ {2} , R _ {\mathrm {A M M} , t - 1} ^ {2}\right) \min  \left(1 , \| \phi (a _ {t}) \| _ {\left(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \frac {\sigma^ {2}}{c} \mathbb {1}\right) ^ {- 1}} ^ {2}\right)}. \\ \end{array}
$$
We will now use the upper bound on  $R_{\mathrm{AMM},t - 1}^2$  from Lemma D.10. Let  $U_{\mathrm{AMM},t - 1}^2$  denote this upper bound (i.e. the right-hand-side of (31)). We have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \Delta (a _ {t}) \leq \sqrt {T \sum_ {t = 1} ^ {T} 4 \max  \left(C ^ {2} , U _ {\mathrm {A M M} , t - 1} ^ {2}\right) \min  \left(1 , \| \phi (a _ {t}) \| _ {\left(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \frac {\sigma^ {2}}{c} \mathbb {1}\right) ^ {- 1}} ^ {2}\right)} \\ \leq 2 \max  \left(C, U _ {\mathrm {A M M}, T - 1}\right) \sqrt {T \sum_ {t = 1} ^ {T} \min  \left(1 , \| \phi (a _ {t}) \| _ {\left(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \frac {\sigma^ {2}}{c} \mathbb {1}\right) ^ {- 1}} ^ {2}\right)} \\ \end{array}
$$
Finally, using the bound on the sum of norms in Lemma D.12, we have
$$
\sum_ {t = 1} ^ {T} \Delta (a _ {t}) \leq \frac {2}{\sqrt {\ln (2)}} \max \left(C, \sigma \sqrt {d \ln \left(1 + \frac {c (T - 1) L ^ {2}}{\sigma^ {2} d}\right) + \frac {B ^ {2}}{c} + 2 \ln \left(\frac {1}{\delta}\right)}\right) \sqrt {d T \ln \left(1 + \frac {c T L ^ {2}}{\sigma^ {2} d}\right)}.
$$
Now, we state and prove a cumulative regret bound that holds for more general choices of the mixture distributions and the parameter  $\alpha$ .
Theorem D.13. Suppose that assumptions 7.1-7.4 hold. If, for any  $\pmb{\theta}_0\in \mathbb{R}^d$  and any  $\sigma_0 > 0$ , the sequence of mixture distributions is  $P_{t} = \mathcal{N}(\Phi_{t}\pmb{\theta}_{0},\sigma_{0}^{2}\Phi_{t}\Phi_{t}^{\top})$ , then for any  $\delta \in (0,1 / 2]$  and any  $\alpha >0$ , with probability at least  $1 - 2\delta$ , for all  $T\geq 1$  simultaneously, the cumulative regret of CMM-UCB and AMM-UCB is bounded by
$$
\Delta_ {1: T} \leq \frac {2}{\sqrt {\ln 2}} \max  \left\{C, U _ {\mathrm {A M M}, T - 1} \right\} \sqrt {d T \ln \left(1 + \frac {L ^ {2} T}{\alpha d}\right)} = \mathcal {O} (d \sqrt {T} \ln (T)),
$$
$$
\begin{array}{l} U _ {\mathrm {A M M}, T - 1} ^ {2} \leq \sigma^ {2} d + \frac {\sigma^ {2}}{\sigma_ {0} ^ {2}} \| \boldsymbol {\theta} ^ {*} - \boldsymbol {\theta} _ {0} \| _ {2} ^ {2} + \sigma^ {2} d \ln \left(1 + \frac {(T - 1) \sigma_ {0} ^ {2} L ^ {2}}{\sigma^ {2} d}\right) + \alpha B ^ {2} + 4 \sigma^ {2} \ln (1 / \delta) \tag {39} \\ + \sigma^ {2} d \ln \left(1 + \frac {(T - 1) L ^ {2}}{\alpha d}\right) + 2 \sqrt {\alpha} \| \boldsymbol {\theta} ^ {*} \| _ {2} \sqrt {\sigma^ {2} d \ln \left(1 + \frac {(T - 1) L ^ {2}}{\alpha d}\right) + 2 \sigma^ {2} \ln (1 / \delta)}. \\ \end{array}
$$
Proof. Following the proof of Theorem 7.6, we obtain (with high probability)
$$
\sum_ {t = 1} ^ {T} \Delta (a _ {t}) \leq \sqrt {T \sum_ {t = 1} ^ {T} 4 \max  \left(C ^ {2} , R _ {\mathrm {A M M} , t - 1} ^ {2}\right) \min  \left(1 , \| \phi (a _ {t}) \| _ {(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \alpha \mathbb {1}) ^ {- 1}} ^ {2}\right)}.
$$
This time, we use the bound on the radius from Lemma D.11. Let  $U_{\mathrm{AMM},T - 1}$  denote this bound on the radius (i.e. the square root of the right-hand-side of (39)). Also, note that this bound on the radius holds with probability at  $1 - \delta$ . Since  $U_{\mathrm{AMM},T - 1}$  is monotonically increasing with  $T$ , we have
$$
\sum_ {t = 1} ^ {T} \Delta (a _ {t}) \leq 2 \max  \left(C, U _ {\mathrm {A M M}, T - 1}\right) \sqrt {T \sum_ {t = 1} ^ {T} \min  \left(1 , \| \phi (a _ {t}) \| _ {\left(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \alpha \mathbb {1}\right) ^ {- 1}} ^ {2}\right)}.
$$
Finally, we use Lemma D.12 to obtain
$$
\sum_ {t = 1} ^ {T} \Delta (a _ {t}) \leq \frac {2}{\sqrt {\ln 2}} \max  \left\{C, U _ {\mathrm {A M M}, T - 1} \right\} \sqrt {d T \ln \left(1 + \frac {L ^ {2} T}{\alpha d}\right)}.
$$
We used two inequalities that each hold with probability at least  $1 - \delta$ . By a union bound argument, the cumulative regret bound holds with probability at least  $1 - 2\delta$ .
In this section, we present the results of some additional experiments in which we investigate the effect of the mixture distributions (or priors) on our upper and lower confidence bounds.
E.1 The Effect of The Mixture Distributions
We investigate how our CMM-UCB upper and lower confidence bounds behave when we provide an uninformative but well-specified mixture distribution/prior, an informative and well-specified mixture distribution/prior, and a misspecified mixture distribution/prior. Here misspecification refers to prior misspecification in a Bayesian sense. For reference, we compare the behaviour of our upper and lower confidence bounds with the upper and lower limits of a Bayesian credible interval.
For a fair comparison with CMM-UCB, we attempt to construct a Bayesian credible interval that holds with high probability for all rounds  $t \geq 0$ . However, whilst the CMM-UCB confidence set holds with high probability over the random draw of the data  $a_1, r_1, a_2, r_2, \ldots$ , the Bayesian credible interval holds with high probability over the random draw of  $\theta^*$  from a prior (for fixed data  $a_1, r_1, a_2, r_2, \ldots$ ).
For the Bayesian credible interval, we use a Gaussian prior and assume  $\pmb{\theta}^{*}\sim \mathcal{N}(\pmb{\mu}_{0},\pmb{\Sigma}_{0})$ . We assume a Gaussian likelihood function, i.e. rewards are of the form  $r_t = \phi (a_t)^\top \pmb {\theta} + \epsilon_t$ , where  $\epsilon_{t}\sim \mathcal{N}(0,\sigma^{2})$ . The Bayesian posterior for  $\pmb{\theta}^{*}$  is another Gaussian  $\mathcal{N}(\pmb{\mu}_t,\pmb{\Sigma}_t)$ , where
$$
\pmb {\mu} _ {t} = \pmb {\Sigma} _ {t} \left(\pmb {\Sigma} _ {0} ^ {- 1} \pmb {\mu} _ {0} + \frac {1}{\sigma^ {2}} \Phi_ {t} ^ {\top} \pmb {r} _ {t}\right), \qquad \pmb {\Sigma} _ {t} = \left(\frac {1}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \pmb {\Sigma} _ {0} ^ {- 1}\right) ^ {- 1}.
$$
Using Bayes' rule, at any round  $t$ , we have  $\theta^{*} \sim \mathcal{N}(\pmb{\mu}_{t}, \pmb{\Sigma}_{t})$ . Therefore
$$
\left(\boldsymbol {\mu} _ {t} - \boldsymbol {\theta} ^ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {t} ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {\theta} ^ {*}\right) \sim \chi^ {2} (d),
$$
where  $\chi^2 (d)$  is a chi-squared distribution with  $d$  degrees of freedom. Let  $Q_{d}(\cdot)$  be the quantile function of the chi-squared distribution with  $d$  degrees of freedom. With probability at least  $1 - \delta_t$  (over the random draw of  $\theta^{*}$  from  $\mathcal{N}(\pmb {\mu}_t,\pmb {\Sigma}_t)$ )
$$
\left(\boldsymbol {\mu} _ {t} - \boldsymbol {\theta} ^ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {t} ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {\theta} ^ {*}\right) \leq Q _ {d} \left(1 - \delta_ {t}\right). \tag {40}
$$
Using a union bound argument, if  $\delta_t = \frac{6\delta}{(t + 1)^2\pi^2}$ , then (40) holds with probability at least  $1 - \delta_t$  for all  $t \geq 0$ . Therefore, if  $\pmb{\theta}^{*} \sim \mathcal{N}(\pmb{\mu}_0, \pmb{\Sigma}_0)$ , then with high probability the following credible sets contain  $\pmb{\theta}^{*}$  for all  $t \geq 0$  simultaneously:
$$
\Theta_ {t} = \left\{\boldsymbol {\theta} \in \mathbb {R} ^ {d} \Big | (\boldsymbol {\mu} _ {t} - \boldsymbol {\theta} ^ {*}) ^ {\top} \boldsymbol {\Sigma} _ {t} ^ {- 1} (\boldsymbol {\mu} _ {t} - \boldsymbol {\theta} ^ {*}) \leq Q _ {d} \left(1 - \frac {6 \delta}{(t + 1) ^ {2} \pi^ {2}}\right) \right\}.
$$
The upper limit of the credible interval for this credible set (and the one we use in Figure 4) is
$$
\sup _ {\pmb {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {T} \pmb {\theta} \right\} = \phi (a) ^ {T} \pmb {\mu} _ {t} + \sqrt {Q _ {d} \left(1 - \frac {6 \delta}{(t + 1) ^ {2} \pi^ {2}}\right)} \sqrt {\phi (a) ^ {\top} \pmb {\Sigma} _ {t} \phi (a)}.
$$
We compute confidence bounds/credible intervals for a randomly generated linear function of the form  $f(x) = \phi (x)^{\top}\pmb{\theta}^{*}$ , with inputs  $x\in \mathbb{R}$  and  $\pmb{\theta}^{*}\in \mathbb{R}^{20}$ , the latter drawn from a standard Gaussian distribution and if necessary scaled down to  $\| \pmb{\theta}^*\| _2\leq 10\eqqcolon B$ . For the feature map  $\phi$ , we use Random Fourier Features. We generate random data  $\{(x_k,y_k)\}_{k = 1}^t$ , where  $y_{k} = \phi (x_{k})^{\top}\pmb{\theta}^{*} + \eta_{k}$ ,  $\eta_{k}\sim \mathcal{N}(0,\sigma^{2})$  and  $\sigma = 0.1$  (so the Gaussian likelihood is well-specified).
Figure 4: The upper and lower confidence bounds of our CMM-UCB method (left) and Bayesian posterior credible intervals (right) with different choices of the prior. The top row uses the prior  $f_{t} \sim \mathcal{N}(\mathbf{0}, \Phi_{t}\Phi_{t}^{\top})$  for CMM-UCB and  $\pmb{\theta}^{*} \sim \mathcal{N}(\mathbf{0}, \mathbb{1})$  for Bayes. The middle row uses an informative prior:  $f_{t} \sim \mathcal{N}(\Phi_{t}\pmb{\theta}^{*}, 0.1\Phi_{t}\Phi_{t}^{\top})$  for CMM-UCB and  $\pmb{\theta}^{*} \sim \mathcal{N}(\pmb{\theta}^{*}, 0.1\mathbb{1})$  for Bayes. The bottom row uses a misspecified prior:  $f_{t} \sim \mathcal{N}(-\Phi_{t}\pmb{\theta}^{*}, 0.1\Phi_{t}\Phi_{t}^{\top})$  for CMM-UCB and  $\pmb{\theta}^{*} \sim \mathcal{N}(-\pmb{\theta}^{*}, 0.1\mathbb{1})$  for Bayes.
Figure 4 shows the CMM-UCB upper and lower confidence bounds (left) and the Bayesian credible intervals (right) with different choices of the prior. We use roughly equivalent priors for both methods. If the Bayesian credible interval uses the prior  $\pmb{\theta}^{*}\sim \mathcal{N}(\pmb{\mu}_{0},\pmb{\Sigma}_{0})$ , then CMM-UCB uses the induced distribution over the function values  $\Phi_t\pmb{\theta}^*$ , i.e.  $\pmb{f}_t\sim \mathcal{N}(\Phi_t\pmb {\mu}_0,\Phi_t\pmb {\Sigma}_0\Phi_t^\top)$ .
In the top and middle rows of Figure 4, where the prior is well-specified, we observe that the CMM-UCB upper and lower confidence bounds are slightly looser than the Bayesian credible intervals. In the bottom row of Figure 4, when the prior is misspecified, the CMM-UCB interval gets looser whereas the Bayesian credible interval becomes wrong. In summary, the Bayesian credible interval appears to be slightly tighter when the prior is well-specified and at least somewhat informative, but CMM-UCB is robust to "misspecified" mixture distributions.
E.2 Benefits of Adaptive Mixture Distributions
In this section, we investigate a method for refining  $\pmb{\mu}_t$  and  $\pmb{T}_{t}$  based on previously observed actions and rewards. Recall that  $\pmb{\mu}_t$  and  $\pmb{T}_{t}$  must be chosen such that: (a)  $\pmb{\mu}_t$  and  $\pmb{T}_{t}$  can only depend on  $a_1,\ldots ,a_t$  and  $r_1,\dots ,r_{t - 1}$ ; (b) the first  $t - 1$  elements of  $\pmb{\mu}_t$  must be equal to  $\pmb{\mu}_{t - 1}$ ; (c) the upper left  $t - 1\times t - 1$  block of  $\pmb{T}_{t}$  must be  $\pmb{T}_{t - 1}$ ; (d)  $\pmb{T}_{t}$  must be positive (semi-)definite.
As in Sec. 6.4,  $m$  and  $k$  are any fixed mean and kernel functions. Each new row and column of  $\pmb{T}_t$  is set using an adaptive kernel function  $k_{t-1}$ . For  $\beta > 0$ , define
$$
k _ {t} (a, a ^ {\prime}) := k (a, a ^ {\prime}) - \boldsymbol {k} _ {t} (a) ^ {\top} \left(\boldsymbol {K} _ {t} + \beta \mathbb {1}\right) ^ {- 1} \boldsymbol {k} _ {t} (a ^ {\prime}),
$$
where  $\pmb{k}_t(a) = [k(a, a_1), \dots, k(a, a_t)]^\top$  and  $\pmb{K}_t$  is the kernel matrix whose  $(i,j)$ th element is  $k(a_i, a_j)$ . For  $\pmb{T}_t$ , we choose
$$
\boldsymbol {T} _ {t} = \left[ \begin{array}{c c c c} k _ {0} \left(a _ {1}, a _ {1}\right) & k _ {1} \left(a _ {1}, a _ {2}\right) & \dots & k _ {t - 1} \left(a _ {1}, a _ {t}\right) \\ k _ {1} \left(a _ {2}, a _ {1}\right) & k _ {1} \left(a _ {2}, a _ {2}\right) & \dots & k _ {t - 1} \left(a _ {2}, a _ {t}\right) \\ \vdots & \vdots & \ddots & \vdots \\ k _ {t - 1} \left(a _ {t}, a _ {1}\right) & k _ {t - 1} \left(a _ {t}, a _ {2}\right) & \dots & k _ {t - 1} \left(a _ {t}, a _ {t}\right) \end{array} \right]. \tag {41}
$$
The  $i^{\mathrm{th}}$  column and  $i^{\mathrm{th}}$  row of this matrix depend only on only  $a_1, r_1, \ldots, a_i$ . Our motivation for this kernel function is: (a) generalising the usual Bayesian Gaussian process (GP) posterior covariance, one can show that if the kernel function  $k$  is positive definite, then  $\pmb{T}_t$  is positive semi-definite; (b)  $k_t$  is the Bayesian GP posterior covariance function (with a Gaussian likelihood with variance  $\beta$ ). Each new element of  $\pmb{\mu}_t$  is set by evaluating an adaptive mean function  $m_{t-1}$  at the latest action  $a_t$ . Define
$$
m _ {t} (a) := m (a) - \boldsymbol {k} _ {t} (a) ^ {\top} \left(\boldsymbol {K} _ {t} + \beta \mathbb {1}\right) ^ {- 1} \left(\boldsymbol {m} _ {t} - \boldsymbol {r} _ {t}\right).
$$
For  $\pmb{\mu}_t$ , we choose
$$
\boldsymbol {\mu} _ {t} = \left[ m _ {0} \left(a _ {1}\right), m _ {1} \left(a _ {2}\right), \dots , m _ {t - 1} \left(a _ {t}\right) \right] ^ {\top}. \tag {42}
$$
The  $i^{\mathrm{th}}$  element,  $m_{i - 1}(a_i)$ , depends on only  $a_1, r_1, \ldots, a_i$ , so this is a valid choice for  $\pmb{\mu}_t$ . Note that  $m_t$  is the Bayesian GP posterior mean function (again with a Gaussian likelihood with variance  $\beta$ ).
We now compare the standard "non-adaptive" mixture distributions (with  $\pmb{\mu}_t$  and  $T_{t}$  as in (9)) and an adaptive sequence of Gaussian mixture distributions (with  $\pmb{\mu}_t$  as in (42) and  $T_{t}$  as in (41)). With both sequences of mixture distributions, we use  $m(a) = 0$  and  $k(a,a^{\prime}) = \phi (a)^{\top}\phi (a^{\prime})$ . For the adaptive sequence of mixture distributions, we set  $\beta = 4\sigma^{2}$ .
Figure 5: The upper and lower confidence bounds of CMM-UCB with the standard (non-adaptive) sequence of Gaussian mixture distributions (purple) and the adaptive sequence of Gaussian mixture distributions (red).
Figure 5 shows upper and lower confidence bounds for a randomly generated linear function  $f(x) = \phi (x)^{\top}\pmb{\theta}^{*}$ , where  $x\in \mathbb{R}$ ,  $\pmb{\theta}^{*}\in \mathbb{R}^{20}$ , and  $\phi$  is a random Fourier feature map. In this example, the adaptive sequence of mixture distributions leads to tighter upper and lower confidence bounds.
Figure 6: The radius  $R_{\mathrm{AMM},t}$  (left) and test accuracy (right) with the standard sequence of Gaussian mixture distributions (blue) and the adaptive sequence of Gaussian mixture distributions (orange). On the left, we plot the mean value of the radius  $R_{\mathrm{AMM},t}$  over 10 runs and show the mean ± one standard deviation in the shaded regions. On the right, we plot the mean reward over 10 runs and after Gaussian kernel smoothing.
Figure 6 shows the radius  $R_{\mathrm{AMM},t}$  (left) and test accuracy (right) for AMM-UCB in the SVM hyperparameter tuning problem with the Raisin data set (described in Sec. 8.2). We observe that AMM-UCB with the adaptive mixture distributions achieves slightly higher test accuracy.
In Sec. D.2.1, we saw that for a standard sequence of mixture distributions,  $R_{\mathrm{AMM},T}$  could be upper bounded by a data-independent quantity of order  $\mathcal{O}(\sqrt{d\ln(T)})$ . In Figure 6,  $R_{\mathrm{AMM},T}$  does appear to grow roughly logarithmically with  $T$  when the mixture distributions are the standard choice. However,  $R_{\mathrm{AMM},T}$  appears to be bounded by a constant when the mixture distributions are adaptive. If, when using adaptive mixture distributions, we could prove a data-independent bound on  $R_{\mathrm{AMM},T}$  of order  $\mathcal{O}(\sqrt{d})$ , then we would be able to improve our data-independent cumulative regret bounds to  $\mathcal{O}(d\sqrt{T\ln(T)})$  (rather than  $\mathcal{O}(d\sqrt{T}\ln (T))$ ). This would be within a  $\sqrt{\ln(T)}$  factor of the lower bound  $\Omega (d\sqrt{T})$ .
F Efficient Radius Computation
If we compute the squared radius  $R_{\mathrm{MM},t}^2$  using the expression in (5), then we have to compute the inverse and determinant of the  $t\times t$  matrix  $\mathbb{1} + T_t / \sigma^2$ . We will now show that for any mixture distribution of the form  $P_{t} = \mathcal{N}(\pmb{\mu}_{t},\Phi_{t}\pmb{\Sigma}_{0}\Phi_{t}^{\top})$ , where  $\pmb{\Sigma}_{0}$  is symmetric and positive-definite, we can re-write the expression for  $R_{\mathrm{MM},t}^2$  such that we instead need to compute the inverse and determinant of a  $d\times d$  matrix. When  $P_{t} = \mathcal{N}(\pmb{\mu}_{t},\Phi_{t}\pmb{\Sigma}_{0}\Phi_{t}^{\top})$ , the squared radius  $R_{\mathrm{MM},t}^2$  is equal to
$$
R _ {\mathrm {M M}, t} ^ {2} = \left(\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\mathbb {1} + \frac {\Phi_ {t} \boldsymbol {\Sigma} _ {0} \Phi_ {t} ^ {\top}}{\sigma^ {2}}\right) ^ {- 1} (\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}) + \sigma^ {2} \ln \left(\det \left(\mathbb {1} + \frac {\Phi_ {t} \boldsymbol {\Sigma} _ {0} \Phi_ {t} ^ {\top}}{\sigma^ {2}}\right)\right) + 2 \sigma^ {2} \ln (1 / \delta).
$$
By using the Weinstein-Aronszajn identity, and then doing some algebra, we have