SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| \Phi_ {t} \boldsymbol {\theta} ^ {*} - \boldsymbol {r} _ {t} \| _ {2} ^ {2} \leq \sigma^ {2} (\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}) ^ {\top} (\boldsymbol {T} _ {t} + \sigma^ {2} \mathbb {1}) ^ {- 1} (\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}) - 2 \sigma^ {2} \ln \left(\sqrt {\frac {\det  (\sigma^ {2} \mathbb {1})}{\det  (\boldsymbol {T} _ {t} + \sigma^ {2} \mathbb {1})}}\right) + 2 \sigma^ {2} \ln \left(\frac {1}{\delta}\right). \\ = \left(\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\mathbb {1} + \frac {\boldsymbol {T} _ {t}}{\sigma^ {2}}\right) ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}\right) + \sigma^ {2} \ln \det  \left(\mathbb {1} + \frac {\boldsymbol {T} _ {t}}{\sigma^ {2}}\right) + 2 \sigma^ {2} \ln \left(\frac {1}{\delta}\right). \\ \end{array}
$$
C Computing Upper Confidence Bounds
First, we state and prove some useful lemmas.
Lemma C.1. For any  $\alpha > 0$
$$
\left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) + \alpha \boldsymbol {\theta} ^ {\top} \boldsymbol {\theta} - R _ {\mathrm {M M}, t} ^ {2} - \alpha B ^ {2} = \left(\boldsymbol {\theta} - \widehat {\boldsymbol {\theta}} _ {\alpha , t}\right) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) \left(\boldsymbol {\theta} - \widehat {\boldsymbol {\theta}} _ {\alpha , t}\right) - R _ {\mathrm {A M M}, t} ^ {2},
$$
where  $R_{\mathrm{MM},t}^2$  is the squared radius quantity from Cor. 5.2 and
$$
\begin{array}{l} \widehat {\boldsymbol {\theta}} _ {\alpha , t} = \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t}, \\ R _ {\mathrm {A M M}, t} ^ {2} = R _ {\mathrm {M M}, t} ^ {2} + \alpha B ^ {2} - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t}. \\ \end{array}
$$
Proof. For any symmetric matrix  $A$ , we have
$$
\left(\boldsymbol {\theta} - \boldsymbol {b}\right) ^ {\top} \boldsymbol {A} \left(\boldsymbol {\theta} - \boldsymbol {b}\right) + c = \boldsymbol {\theta} ^ {\top} \boldsymbol {A} \boldsymbol {\theta} - 2 \boldsymbol {b} ^ {\top} \boldsymbol {A} \boldsymbol {\theta} + \boldsymbol {b} ^ {\top} \boldsymbol {A} \boldsymbol {b} + c.
$$
$$
\left(\Phi_ {t} \pmb {\theta} - \pmb {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \pmb {\theta} - \pmb {r} _ {t}\right) + \alpha \pmb {\theta} ^ {\top} \pmb {\theta} - R _ {\mathrm {M M}, t} ^ {2} - \alpha B ^ {2} = \pmb {\theta} ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) \pmb {\theta} - 2 \pmb {r} _ {t} ^ {\top} \Phi_ {t} \pmb {\theta} + \pmb {r} _ {t} ^ {\top} \pmb {r} _ {t} - R _ {\mathrm {M M}, t} ^ {2} - \alpha B ^ {2}.
$$
We can now find  $A$ ,  $b$  and  $c$  by equating coefficients. We find that
$$
\boldsymbol {A} = \Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1},
$$
which is a symmetric matrix. We have
$$
\begin{array}{l} \boldsymbol {b} ^ {\top} \boldsymbol {A} = \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \\ \Longrightarrow \boldsymbol {A} \boldsymbol {b} = \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} \\ \Rightarrow \boldsymbol {b} = \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} = \widehat {\boldsymbol {\theta}} _ {\alpha , t}. \\ \end{array}
$$
$$
\begin{array}{l} c = - R _ {\mathrm {M M}, t} ^ {2} - \alpha B ^ {2} + \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} - \boldsymbol {b} ^ {\top} \boldsymbol {A} \boldsymbol {b} \\ = - R _ {\mathrm {M M}, t} ^ {2} - \alpha B ^ {2} + \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} - \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} \\ = - R _ {\mathrm {A M M}, t} ^ {2} \\ \end{array}
$$
Therefore, we have shown that
$$
\left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) + \alpha \boldsymbol {\theta} ^ {\top} \boldsymbol {\theta} - R _ {\mathrm {M M}, t} ^ {2} - \alpha B ^ {2} = \left(\boldsymbol {\theta} - \widehat {\boldsymbol {\theta}} _ {\alpha , t}\right) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) \left(\boldsymbol {\theta} - \widehat {\boldsymbol {\theta}} _ {\alpha , t}\right) - R _ {\mathrm {A M M}, t} ^ {2}.
$$
Lemma C.2. For any symmetric, positive-definite matrix  $\mathbf{A} \in \mathbb{R}^{d \times d}$ , any vectors  $\mathbf{a}, \mathbf{b} \in \mathbb{R}^d$ , any  $R > 0$ , and any  $\eta < 0$
$$
\max  _ {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} \left\{\boldsymbol {a} ^ {\top} \boldsymbol {\theta} + \eta \left((\boldsymbol {\theta} - \boldsymbol {b}) ^ {\top} \boldsymbol {A} (\boldsymbol {\theta} - \boldsymbol {b}) - R ^ {2}\right) \right\} = \boldsymbol {a} ^ {\top} \boldsymbol {b} - \frac {1}{4 \eta} \boldsymbol {a} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {a} - \eta R ^ {2}.
$$
$$
f (\boldsymbol {\theta}) = \boldsymbol {a} ^ {\top} \boldsymbol {\theta} + \eta \left(\left(\boldsymbol {\theta} - \boldsymbol {b}\right) ^ {\top} \boldsymbol {A} (\boldsymbol {\theta} - \boldsymbol {b}) - R ^ {2}\right)
$$
The gradient and Hessian of  $f$  are
$$
\frac {\partial}{\partial \boldsymbol {\theta}} f (\boldsymbol {\theta}) = \boldsymbol {a} + 2 \eta \boldsymbol {A} (\boldsymbol {\theta} - \boldsymbol {b}), \quad \frac {\partial^ {2}}{\partial \boldsymbol {\theta} ^ {2}} f (\boldsymbol {\theta}) = 2 \eta \boldsymbol {A}.
$$
Since  $\mathbf{A}$  is positive-definite and  $\eta < 0$ ,  $\frac{\partial^2}{\partial\theta^2} f(\theta)$  is negative-definite for all  $\pmb {\theta}\in \mathbb{R}^d$ . Therefore, any solution  $\pmb{\theta}^{*}$  of  $\frac{\partial}{\partial\theta} f(\pmb {\theta}) = 0$  must be a maximiser of  $f(\pmb {\theta})$ . There is a unique solution, which is
$$
\boldsymbol {\theta} ^ {*} = \boldsymbol {b} - \frac {1}{2 \eta} \boldsymbol {A} ^ {- 1} \boldsymbol {a}.
$$
The maximum is
$$
f (\boldsymbol {\theta} ^ {*}) = \boldsymbol {a} ^ {\top} \boldsymbol {b} - \frac {1}{4 \eta} \boldsymbol {a} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {a} - \eta R ^ {2}.
$$
Lemma C.3. For any symmetric, positive-definite matrix  $\mathbf{A} \in \mathbb{R}^{d \times d}$ , any vectors  $\mathbf{a}, \mathbf{b} \in \mathbb{R}^d$ , and any  $R > 0$
$$
\min  _ {\eta <   0} \left\{\boldsymbol {a} ^ {\top} \boldsymbol {b} - \frac {1}{4 \eta} \boldsymbol {a} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {a} - \eta R ^ {2} \right\} = \boldsymbol {a} ^ {\top} \boldsymbol {b} + R \sqrt {\boldsymbol {a} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {a}}.
$$
$$
g (\eta) = \boldsymbol {a} ^ {\top} \boldsymbol {b} - \frac {1}{4 \eta} \boldsymbol {a} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {a} - \eta R ^ {2}.
$$
The first and second derivatives of  $g$  are
$$
\frac {\mathrm {d}}{\mathrm {d} \eta} g (\eta) = \frac {1}{4 \eta^ {2}} \boldsymbol {a} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {a} - R ^ {2}, \quad \frac {\mathrm {d} ^ {2}}{\mathrm {d} \eta^ {2}} g (\eta) = - \frac {1}{2 \eta^ {3}} \boldsymbol {a} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {a}.
$$
Since  $\mathbf{A}$  is positive-definite,  $\frac{\mathrm{d}^2}{\mathrm{d}\eta^2} g(\eta)$  is positive for all  $\eta < 0$ . Therefore, any negative solution  $\eta^*$  of  $\frac{\mathrm{d}}{\mathrm{d}\eta} g(\eta) = 0$  must be a minimiser of  $g(\eta)$ . There is a unique (negative) solution, which is
$$
\eta^ {*} = - \frac {1}{2 R} \sqrt {\boldsymbol {a} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {a}}.
$$
The minimum is
$$
g \left(\eta^ {*}\right) = \boldsymbol {a} ^ {\top} \boldsymbol {b} + R \sqrt {\boldsymbol {a} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {a}}.
$$
C.1 Analytic UCBs
Here, we prove Theorem 6.1, which states that for all  $\alpha >0$ :
$$
\max  _ {\boldsymbol {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\} \leq \phi (a) ^ {\top} \widehat {\boldsymbol {\theta}} _ {\alpha , t} + R _ {\mathrm {A M M}, t} \sqrt {\phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a)}, \tag {18}
$$
$\Theta_{t}$  is the confidence set at time  $t$  in our confidence sequence from Cor. 5.2 and  $R_{\mathrm{MM},t}$  is the radius from Cor. 5.2 and Eq. (5). As well as proving this statement, we will also show that if  $\Theta_{t}$  has an interior point, then when the right-hand-side of (18) is optimised with respect to  $\alpha >0$ , the inequality in (18) becomes an equality, i.e.
$$
\left. \max  _ {\boldsymbol {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\} = \min  _ {\alpha > 0} \left\{\phi (a) ^ {\top} \widehat {\boldsymbol {\theta}} _ {\alpha , t} + R _ {\mathrm {A M M}, t} \sqrt {\phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a)} \right\}. \right. \tag {19}
$$
Proof of Thm. 6.1. We use weak Lagrangian duality to prove the upper bound and strong Lagrangian duality to prove the second part. The convex optimisation problem  $\max_{\theta \in \Theta_t} \{\phi(a)^\top \theta\}$  can be stated as
$$
\max  _ {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} \phi (a) ^ {\top} \boldsymbol {\theta} \quad \text {s . t .} (\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}) ^ {\top} (\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}) \leq R _ {\mathrm {M M}, t} ^ {2} \quad \text {a n d} \quad \boldsymbol {\theta} ^ {\top} \boldsymbol {\theta} \leq B ^ {2}. \tag {20}
$$
Rewriting both constraints in the form  $f(\pmb{\theta}) \leq 0$ , we can see that the Lagrangian for this problem is
$$
L (\boldsymbol {\theta}, \eta_ {1}, \eta_ {2}) = \phi (a) ^ {\top} \boldsymbol {\theta} + \eta_ {1} \left(\left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) - R _ {\mathrm {M M}, t} ^ {2}\right) + \eta_ {2} \left(\boldsymbol {\theta} ^ {\top} \boldsymbol {\theta} - B ^ {2}\right).
$$
$\eta_{1}$  and  $\eta_{2}$  are called the Lagrange multipliers. The Lagrange dual function (or just dual function) is
$$
g (\eta_ {1}, \eta_ {2}) = \max  _ {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} \left\{L (\boldsymbol {\theta}, \eta_ {1}, \eta_ {2} \right\}.
$$
By weak duality, for any  $\eta_1,\eta_2\leq 0$  , the dual function is an upper bound on the solution of the primal problem in (20), i.e. for any  $\eta_{1},\eta_{2}\leq 0$
$$
\max  _ {\boldsymbol {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\} \leq g \left(\eta_ {1}, \eta_ {2}\right). \tag {21}
$$
Alternatively, (21) can be verified by starting from the inequality  $\phi(a)^{\top} \pmb{\theta} \leq L(\pmb{\theta}, \eta_1, \eta_2)$  for all  $\pmb{\theta} \in \Theta_t, \eta_1 \leq 0$ , and  $\eta_2 \leq 0$ . The challenge is to set the Lagrange multipliers such that the dual function has a closed-form expression while being as close as possible to its minimum value  $\min_{\eta_1, \eta_2 \leq 0} \{g(\eta_1, \eta_2)\}$ . We will now show that for any  $\alpha > 0$ ,  $\min_{\eta \leq 0} \{g(\eta, \alpha \eta)\}$  has a closed-form solution, which is the right-hand-side of (18). The Lagrangian, evaluated with the Lagrange multipliers  $\eta$  and  $\alpha \eta$ , is
$$
L (\boldsymbol {\theta}, \eta , \alpha \eta) = \phi (a) ^ {\top} \boldsymbol {\theta} + \eta \left(\left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\Phi_ {t} \boldsymbol {\theta} - \boldsymbol {r} _ {t}\right) + \alpha \boldsymbol {\theta} ^ {\top} \boldsymbol {\theta} - R _ {\mathrm {M M}, t} ^ {2} - \alpha B ^ {2}\right).
$$
Using Lemma C.1, the Lagrangian can be rewritten as
$$
L (\boldsymbol {\theta}, \eta , \alpha \eta) = \phi (a) ^ {\top} \boldsymbol {\theta} + \eta \left((\boldsymbol {\theta} - \widehat {\boldsymbol {\theta}} _ {\alpha , t}) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) (\boldsymbol {\theta} - \widehat {\boldsymbol {\theta}} _ {\alpha , t}) - R _ {\mathrm {A M M}, t} ^ {2}\right).
$$
Using Lemma C.2, the dual function evaluated at  $\eta$  and  $\alpha \eta$  is
$$
\begin{array}{l} g (\eta , \alpha \eta) = \max  _ {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} + \eta \left(\left(\boldsymbol {\theta} - \widehat {\boldsymbol {\theta}} _ {\alpha , t}\right) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) \left(\boldsymbol {\theta} - \widehat {\boldsymbol {\theta}} _ {\alpha , t}\right) - R _ {\mathrm {A M M}, t} ^ {2}\right) \right\} \\ = \phi (a) ^ {\top} \widehat {\pmb {\theta}} _ {\alpha , t} - \frac {1}{4 \eta} \phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a) - \eta R _ {\mathrm {A M M}, t} ^ {2}. \\ \end{array}
$$
Using Lemma C.3, we have
$$
\begin{array}{l} \min _ {\eta \leq 0} \left\{g (\eta , \alpha \eta) \right\} = \min _ {\eta \leq 0} \left\{\phi (a) ^ {\top} \widehat {\boldsymbol {\theta}} _ {\alpha , t} - \frac {1}{4 \eta} \phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a) - \eta R _ {\mathrm {A M M}, t} ^ {2} \right\} \\ = \phi (a) ^ {\top} \widehat {\pmb {\theta}} _ {\alpha , t} + R _ {\mathrm {A M M}, t} \sqrt {\phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a)}. \\ \end{array}
$$
This concludes the proof of Theorem 6.1. To prove (19), we use strong duality. Clearly  $\min_{\alpha >0}\min_{\eta \leq 0}\{g(\eta ,\alpha \eta)\} = \min_{\eta_1,\eta_2\leq 0}\{g(\eta_1,\eta_2)\}$ , so if we optimise the upper bound in (18) with respect to  $\alpha$ , then we will recover the minimum of the dual function. If strong duality holds, then the minimum of the dual function is equal to  $\max_{\theta \in \Theta_t}\{\phi (a)^\top \pmb {\theta}\}$ . Since  $\max_{\pmb {\theta}\in \Theta_t}\{\phi (a)^\top \pmb {\theta}\}$  is a convex optimisation problem, we can use Slater's condition to obtain a sufficient condition for strong duality to hold. In particular, if  $\Theta_t$  has an interior point, then strong duality holds, which means
$$
\begin{array}{l} \max  _ {\boldsymbol {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\} = \min  _ {\alpha > 0} \min  _ {\eta \leq 0} \left\{g (\eta , \alpha \eta) \right\} \\ = \min _ {\alpha > 0} \left\{\phi (a) ^ {\top} \widehat {\pmb {\theta}} _ {\alpha , t} + R _ {\mathrm {A M M}, t} \sqrt {\phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a)} \right\}. \\ \end{array}
$$
One can follow the same steps, with a few minor modifications, to prove a similar statement for lower confidence bounds. For all  $\alpha > 0$
$$
\min _ {\boldsymbol {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\} \geq \phi (a) ^ {\top} \widehat {\boldsymbol {\theta}} _ {\alpha , t} - R _ {\mathrm {A M M}, t} \sqrt {\phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a)}.
$$
If  $\Theta_t$  has an interior point, then
$$
\min _ {\boldsymbol {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\} = \max _ {\alpha > 0} \left\{\phi (a) ^ {\top} \widehat {\boldsymbol {\theta}} _ {\alpha , t} - R _ {\mathrm {A M M}, t} \sqrt {\phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a)} \right\}.
$$
C.2 OFUL vs AMM-UCB (and CMM-UCB)
We will now show that for any value of the parameter  $\alpha$ , we can choose a sequence of Gaussian mixture distributions, such that the confidence bounds of AMM-UCB (and therefore also CMM-UCB) are always tighter than the confidence bounds of OFUL (Abbasi-Yadkori et al., 2011).
To do this, we will use the following lemma.
Lemma C.4. For any  $\gamma > 0$ ,  $\pmb{v} \in \mathbb{R}^t$  and  $\pmb{M} \in \mathbb{R}^{t \times d}$ , we have
$$
\boldsymbol {v} ^ {\top} \boldsymbol {v} - \boldsymbol {v} ^ {\top} M \left(M ^ {\top} M + \gamma \mathbb {1}\right) ^ {- 1} M ^ {\top} \boldsymbol {v} = \boldsymbol {v} ^ {\top} \left(\frac {1}{\gamma} M M ^ {\top} + \mathbb {1}\right) ^ {- 1} \boldsymbol {v}.
$$
Proof. We start with the identity
$$
M \left(M ^ {\top} M + \gamma \mathbb {1}\right) = \left(M M ^ {\top} + \gamma \mathbb {1}\right) M.
$$
By post-multiplying both sides with  $\left(M^{\top}M + \gamma \mathbb{1}\right)^{-1}$  and pre-multiplying both sides with  $\left(MM^{\top} + \gamma \mathbb{1}\right)^{-1}$ , we obtain
$$
\left(M M ^ {\top} + \gamma \mathbb {1}\right) ^ {- 1} M = M \left(M ^ {\top} M + \gamma \mathbb {1}\right) ^ {- 1}. \tag {22}
$$
Now, using (22), we have
$$
\begin{array}{l} \boldsymbol {v} ^ {\top} \boldsymbol {v} - \boldsymbol {v} ^ {\top} M \left(M ^ {\top} M + \gamma \mathbb {1}\right) ^ {- 1} M ^ {\top} \boldsymbol {v} = \boldsymbol {v} ^ {\top} \boldsymbol {v} - \boldsymbol {v} ^ {\top} \left(M M ^ {\top} + \gamma \mathbb {1}\right) ^ {- 1} M M ^ {\top} \boldsymbol {v} \\ = \boldsymbol {v} ^ {\top} \boldsymbol {v} - \boldsymbol {v} ^ {\top} \left(M M ^ {\top} + \gamma \mathbb {1}\right) ^ {- 1} \left(M M ^ {\top} + \gamma \mathbb {1} - \gamma \mathbb {1}\right) \boldsymbol {v} \\ = \boldsymbol {v} ^ {\top} \boldsymbol {v} - \boldsymbol {v} ^ {\top} \boldsymbol {v} + \gamma \boldsymbol {v} ^ {\top} \left(\boldsymbol {M} \boldsymbol {M} ^ {\top} + \gamma \mathbb {1}\right) ^ {- 1} \boldsymbol {v} \\ = \boldsymbol {v} ^ {\top} \left(\frac {1}{\gamma} M M ^ {\top} + \mathbb {1}\right) ^ {- 1} \boldsymbol {v}. \\ \end{array}
$$
With  $\pmb{v} = \pmb{r}_t$  and  $M = \Phi_t$ , we obtain
$$
\boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} - \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \gamma \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} = \boldsymbol {r} _ {t} ^ {\top} \left(\frac {1}{\gamma} \Phi_ {t} \Phi_ {t} ^ {\top} + \mathbb {1}\right) ^ {- 1} \boldsymbol {r} _ {t}.
$$
We will also use the fact that, due to the Weinstein-Aronszajn identity, for any  $\gamma > 0$
$$
\det \left(\gamma \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right) = \det \left(\gamma \Phi_ {t} \Phi_ {t} ^ {\top} + \mathbb {1}\right). \tag {23}
$$
For any  $\alpha > 0$  (in (Abbasi-Yadkori et al., 2011), what we call  $\alpha$  is called  $\lambda$ ), the OFUL UCB states that
$$
\phi (a) ^ {\top} \boldsymbol {\theta} ^ {*} \leq \phi (a) ^ {\top} \widehat {\boldsymbol {\theta}} _ {\alpha , t} + R _ {\mathrm {O F U L}, t} \sqrt {\phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a)},
$$
$$
\text {w h e r e} R _ {\mathrm {O F U L}, t} = \sigma \sqrt {\ln \left(\det \left(\frac {1}{\alpha} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) + 2 \ln (1 / \delta)} + \sqrt {\alpha} B.
$$
For any  $\alpha > 0$  and any  $\delta \in (0,1]$ , this statement holds with probability at least  $1 - \delta$  for all  $t \geq 0$  and all  $a \in \mathcal{A}$ . By comparison, our AMM-UCB holds uniformly over all  $t \geq 0$ , all  $a \in \mathcal{A}$  and all  $\alpha > 0$  (i.e. we could optimise the AMM-UCB with respect to  $\alpha$  in a data-dependent manner, which would yield our CMM-UCB).
Notice that for any history  $a_1, r_1, a_2, r_2, \ldots$  and any  $\alpha > 0$ , the OFUL UCB is the same as our AMM-UCB, except that our AMM radius quantity  $R_{\mathrm{AMM},t}$  is replaced with  $R_{\mathrm{OFUL},t}$ . The same is true for the LCBs of OFUL and AMM-UCB (with the same  $R_{\mathrm{OFUL},t}$ ), so we will only focus on the UCBs. We will now show that for any history  $a_1, r_1, a_2, r_2, \ldots$  and any  $\alpha > 0$ , we can choose a sequence of Gaussian mixture distributions such that  $R_{\mathrm{AMM},t} \leq R_{\mathrm{OFUL},t}$ . This means that the UCBs of our CMM-UCB and AMM-UCB algorithms are never worse than the OFUL UCB.
Without loss of generality, suppose we choose  $\alpha = \sigma^2 /c$ , for some  $c > 0$ . With this choice, the OFUL radius is
$$
R _ {\mathrm {O F U L}, t} = \sigma \left(\sqrt {\ln \left(\det \left(\frac {c}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) + 2 \ln (1 / \delta)} + \frac {B}{\sqrt {c}}\right).
$$
For any  $\alpha > 0$  and a Gaussian mixture distribution  $P_{t} = \mathcal{N}(\pmb{\mu}_{t},\pmb{T}_{t})$ , the squared AMM-UCB radius is
$$
\begin{array}{l} R _ {\mathrm {A M M}, t} ^ {2} = R _ {\mathrm {M M}, t} ^ {2} + \alpha B ^ {2} - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} \\ = \left(\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}\right) ^ {\top} \left(\mathbb {1} + \frac {\boldsymbol {T} _ {t}}{\sigma^ {2}}\right) ^ {- 1} \left(\boldsymbol {\mu} _ {t} - \boldsymbol {r} _ {t}\right) + \sigma^ {2} \ln \left(\det  \left(\mathbb {1} + \frac {\boldsymbol {T} _ {t}}{\sigma^ {2}}\right)\right) + 2 \sigma^ {2} \ln \left(\frac {1}{\delta}\right) \\ + \alpha B ^ {2} - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t}. \\ \end{array}
$$
For AMM-UCB, we will use  $\alpha = \sigma^2 /c$  and the scaled standard mixture distributions  $P_{t} = \mathcal{N}(\mathbf{0},c\Phi_{t}\Phi_{t}^{\top})$  for each  $t$ . With these choices, and using Lemma C.4 and (23), the squared AMM-UCB radius is
$$
\begin{array}{l} R _ {\mathrm {A M M}, t} ^ {2} = \boldsymbol {r} _ {t} ^ {\top} \left(\frac {c}{\sigma^ {2}} \Phi_ {t} \Phi_ {t} ^ {\top} + \mathbb {1}\right) ^ {- 1} \boldsymbol {r} _ {t} - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \frac {\sigma^ {2}}{c} \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t} \tag {24} \\ + \sigma^ {2} \ln \left(\det \left(\frac {c}{\sigma^ {2}} \Phi_ {t} \Phi_ {t} ^ {\top} + \mathbb {1}\right)\right) + 2 \sigma^ {2} \ln \left(\frac {1}{\delta}\right) + \frac {\sigma^ {2} B ^ {2}}{c} \\ = \sigma^ {2} \left(\ln \left(\det \left(\frac {c}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) + 2 \ln \left(\frac {1}{\delta}\right) + \frac {B ^ {2}}{c}\right). \\ \end{array}
$$
Using the basic inequality  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$  for  $a, b \geq 0$ , we have
$$
\begin{array}{l} R _ {\mathrm {A M M}, t} = \sigma \sqrt {\ln \left(\det \left(\frac {c}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) + 2 \ln \left(\frac {1}{\delta}\right) + \frac {B ^ {2}}{c}} \\ \leq \sigma \left(\sqrt {\ln \left(\det  \left(\frac {c}{\sigma^ {2}} \Phi_ {t} ^ {\top} \Phi_ {t} + \mathbb {1}\right)\right) + 2 \ln \left(\frac {1}{\delta}\right)} + \frac {B}{\sqrt {c}}\right) \\ = R _ {\text {O F U L}, t}. \\ \end{array}
$$
Therefore, the confidence bounds of AMM-UCB, with  $\alpha = \sigma^2 /c$  and  $P_{t} = \mathcal{N}(\mathbf{0},c\Phi_{t}\Phi_{t}^{\top})$  are never looser than the confidence bounds of OFUL with an arbitrary  $\alpha = \sigma^2 /c$ . Since  $\ln \left(\operatorname *{det}\left(\frac{c}{\sigma^2}\Phi_t^\top \Phi_t + \mathbb{1}\right)\right) + 2\ln \left(\frac{1}{\delta}\right)$  and  $B^{2} / c$  are strictly positive, there is actually a strict inequality. This means that the AMM-UCB (and CMM-UCB) confidence bounds are always strictly tighter than the OFUL confidence bounds.
Note that  $P_{t} = \mathcal{N}(\mathbf{0}, c\Phi_{t}\Phi_{t}^{\top})$  is not necessarily the best choice for the mixture distribution. With a better choice of the mixture distribution, e.g. a mixture distribution that is chosen using some prior knowledge about the expected reward function and/or refined using previously observed rewards,  $R_{\mathrm{AMM},t}$  will be smaller and the gap between AMM-UCB and OFUL will be greater.
D Cumulative Regret Bounds