SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Delta_ {A _ {t}} = A _ {*} ^ {\top} \theta - A _ {t} ^ {\top} \theta \leq A _ {*} ^ {\top} \theta - U _ {t, A _ {*}} + U _ {t, A _ {t}} - A _ {t} ^ {\top} \theta \leq U _ {t, A _ {t}} - A _ {t} ^ {\top} \theta \leq 2 C _ {t, A _ {t}}.
$$
The first inequality holds because  $U_{t,A_t} - U_{t,A_*} \geq 0$  by the design of BayesUCB. The second one uses that  $A_*^\top \theta - U_{t,A_*} \leq 0$ . Specifically, for any action  $a \in \mathcal{A}$  on event  $E_t$ ,
$$
a ^ {\top} \theta - U _ {t, a} = a ^ {\top} \left(\theta - \hat {\theta} _ {t}\right) - C _ {t, a} \leq C _ {t, a} - C _ {t, a} = 0.
$$
Cases 2 and 3 are bounded as in Appendix A.6. Now we chain all inequalities, add them over all rounds, and get
$$
\begin{array}{l} R (n) \leq 2 \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \| A _ {t} \| _ {\hat {\Sigma} _ {t}} \right] \sqrt {2 \log (1 / \delta)} + \varepsilon n + 4 L L _ {*} K n \delta \\ \leq 2 \sqrt {\mathbb {E} \left[ \sum_ {t = 1} ^ {n} \| A _ {t} \| _ {\hat {\Sigma} _ {t}} ^ {2} \right]} \sqrt {2 n \log (1 / \delta)} + \varepsilon n + 4 L L _ {*} K n \delta , \\ \end{array}
$$
where the last inequality uses the Cauchy-Schwarz inequality and the concavity of the square root. Finally, the sum  $\sum_{t=1}^{n} \|A_t\|_{\Sigma_t}^2$  is bounded using Lemma 10. This completes the proof.
Figure 3: The difference in regret of UCB1 and BayesUCB on 81 Bayesian bandit instances, sorted by the difference. In plot (a), the noise is Gaussian  $\mathcal{N}(0,\sigma^2)$ . In plot (b), the noise is  $\sigma$  with probability 0.5 and  $-\sigma$  otherwise.
D Comparison of BayesUCB and UCB1
We report the difference in regret of UCB1 and BayesUCB on 81 Bayesian bandit instances. These instances are obtained by all combinations of  $K \in \{5, 10, 20\}$  actions, reward noise  $\sigma \in \{0.5, 1, 2\}$ , prior gap  $\Delta_0 \in \{0.5, 1, 2\}$ , and prior width  $\sigma_0 \in \{0.5, 1, 2\}$ . The horizon is  $n = 1000$  rounds and all results are averaged over 1000 random runs.
Our results are reported in Figure 3. In Figure 3a, the noise is Gaussian  $\mathcal{N}(0,\sigma^2)$ . In Figure 3b, the noise is  $\sigma$  with probability 0.5 and  $-\sigma$  otherwise. Therefore, this noise is  $\sigma^2$ -sub-Gaussian, of the same magnitude as  $\mathcal{N}(0,\sigma^2)$  but far from it in terms of the distribution. This tests the robustness of BayesUCB to Gaussian posterior updates. UCB1 only needs  $\sigma^2$ -sub-Gaussian noise. In both plots, and in all 81 Bayesian bandit instances, BayesUCB has a lower regret than UCB1. It is also remarkably robust to noise misspecification, although we cannot prove it.
E Note on Information-Theory Bounds
Our approach could be used to derive Bayesian information-theory bounds [Russo and Van Roy, 2016]. The key step in these bounds, where the information-theory term  $I_{t,a}$  for action  $a$  at round  $t$  arises, is  $\Delta_{A_t} \leq \Gamma \sqrt{I_{t,A_t}}$ , where  $\Gamma$  is the highest possible ratio of regret to information gain. As in Case 1 in Appendix A.6, the  $n$ -round regret can be bounded as
$$
\sum_ {t = 1} ^ {n} \Delta_ {A _ {t}} = \sum_ {t = 1} ^ {n} \frac {1}{\Delta_ {A _ {t}}} \Delta_ {A _ {t}} ^ {2} \leq \frac {1}{\Delta_ {\min} ^ {\varepsilon}} \sum_ {t = 1} ^ {n} \Delta_ {A _ {t}} ^ {2} \leq \frac {\Gamma^ {2}}{\Delta_ {\min} ^ {\varepsilon}} \sum_ {t = 1} ^ {n} I _ {t, A _ {t}}.
$$
The term  $\sum_{t=1}^{n} I_{t,A_t}$  can be bounded using a worst-case argument by a  $O(\log n)$  bound.