SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \Delta_ {A _ {t}} \mathbb {1} \left\{\bar {E} _ {t} \right\} \right] \tag {8} \\ = \mathbb {E} \left[ \mathbb {E} \left[ \left(\theta_ {A _ {*}} - \theta_ {A _ {t}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid H _ {t} \right] \right] \\ \leq \mathbb {E} \left[ \mathbb {E} \left[ \left(\theta_ {A _ {*}} - U _ {t, A _ {*}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid H _ {t} \right] + \mathbb {E} \left[ \left(U _ {t, A _ {t}} - \theta_ {A _ {t}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid H _ {t} \right] \right] \\ \leq \mathbb {E} \left[ \mathbb {E} \left[ \left(\theta_ {A _ {*}} - \hat {\theta} _ {t, A _ {*}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid H _ {t} \right] + \mathbb {E} \left[ \left(\hat {\theta} _ {t, A _ {t}} - \theta_ {A _ {t}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid H _ {t} \right] + \mathbb {E} \left[ C _ {t, A _ {t}} \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid H _ {t} \right] \right]. \\ \end{array}
$$
To bound the resulting terms, we use that  $\theta_{a} - \hat{\theta}_{t,a}\mid H_{t}\sim \mathcal{N}(0,\hat{\sigma}_{t,a}^{2})$
Lemma 7. For any action  $a \in \mathcal{A}$ , round  $t \in [n]$ , and history  $H_t$ ,
$$
\mathbb {E} \left[ \left(\theta_ {a} - \hat {\theta} _ {t, a}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid H _ {t} \right] \leq 2 \sigma_ {0} K \delta .
$$
Proof. Let  $E_{t,a} = \left\{|\theta_a - \hat{\theta}_{t,a}| \leq C_{t,a}\right\}$ . We start with decomposing  $\bar{E}_t$  into individual  $\bar{E}_{t,a}$  as
$$
\mathbb {E} \left[ (\theta_ {a} - \hat {\theta} _ {t, a}) \mathbb {1} \left\{\bar {E} _ {t} \right\} \Big | H _ {t} \right] \leq \mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mathbb {1} \left\{\bar {E} _ {t, a} \right\} \Big | H _ {t} \right] + \sum_ {a ^ {\prime} \neq a} \mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mathbb {1} \left\{\bar {E} _ {t, a ^ {\prime}} \right\} \Big | H _ {t} \right].
$$
To bound the first term, we use that  $\theta_{a} - \hat{\theta}_{t,a}\mid H_{t}\sim \mathcal{N}(0,\hat{\sigma}_{t,a}^{2})$ . Thus
$$
\begin{array}{l} \mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mathbb {1} \left\{\bar {E} _ {t, a} \right\} \mid H _ {t} \right] \leq \frac {2}{\sqrt {2 \pi \hat {\sigma} _ {t , a} ^ {2}}} \int_ {x = C _ {t, a}} ^ {\infty} x \exp \left[ - \frac {x ^ {2}}{2 \hat {\sigma} _ {t , a} ^ {2}} \right] \mathrm {d} x \\ = - \sqrt {\frac {2 \hat {\sigma} _ {t , a} ^ {2}}{\pi}} \int_ {x = C _ {t, a}} ^ {\infty} \frac {\partial}{\partial x} \left(\exp \left[ - \frac {x ^ {2}}{2 \hat {\sigma} _ {t , a} ^ {2}} \right]\right) d x \\ = \sqrt {\frac {2 \hat {\sigma} _ {t , a} ^ {2}}{\pi}} \delta \leq \sigma_ {0} \delta . \\ \end{array}
$$
To bound the second term, we use the independence of the distributions for  $a$  and  $a'$ ,
$$
\mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mathbb {1} \left\{\bar {E} _ {t, a ^ {\prime}} \right\} \mid H _ {t} \right] = \mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mid H _ {t} \right] \mathbb {P} \left(\bar {E} _ {t, a ^ {\prime}} \mid H _ {t}\right).
$$
The probability is at most  $2\delta$  and the expectation can be bounded as
$$
\mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \Big | H _ {t} \right] = \mathbb {E} \left[ \sqrt {(\theta_ {a} - \hat {\theta} _ {t , a}) ^ {2}} \Big | H _ {t} \right] \leq \sqrt {\mathbb {E} \left[ (\theta_ {a} - \hat {\theta} _ {t , a}) ^ {2} \Big | H _ {t} \right]} = \hat {\sigma} _ {t, a} \leq \sigma_ {0}.
$$
The first two terms in (8) can be bounded using a union bound over  $a \in \mathcal{A}$  and Lemma 7. For the last term, we use that  $C_{t,a} \leq \sqrt{2\sigma_0^2\log(1 / \delta)}$  and a union bound in  $\mathbb{P}\left(\bar{E}_t \mid H_t\right)$  to get
$$
\mathbb {E} \left[ C _ {t, A _ {t}} \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid H _ {t} \right] \leq \sqrt {2 \sigma_ {0} ^ {2} \log (1 / \delta)} \mathbb {P} \left(\bar {E} _ {t} \mid H _ {t}\right) \leq 2 \sqrt {2 \log (1 / \delta)} \sigma_ {0} K \delta .
$$
Finally, we sum up the upper bounds on (8) over all rounds  $t \in [n]$ .
A.2 Proof of Corollary 2
The key idea in the proof is to integrate out the random gap in (7). Fix action  $a \in \mathcal{A}$  and thresholds  $\varepsilon_2 > \varepsilon > 0$ . We consider two cases.
Case 1: Diminishing gaps  $\varepsilon < \Delta_{a} \leq \varepsilon_{2}$ . Let
$$
\xi_ {a} (\theta_ {a} ^ {*}) = \underset {x \in [ \varepsilon , \varepsilon_ {2} ]} {\arg \max } h _ {a} (\theta_ {a} ^ {*} - x)
$$
and  $N_{a}$  be defined as in Appendix A.1. Then
$$
\begin{array}{l} \mathbb {E} \left[ \Delta_ {a} N _ {a} \mathbb {1} \left\{\varepsilon <   \Delta_ {a} <   \varepsilon_ {2} \right\} \right] = \int_ {\theta_ {- a}} \int_ {\theta_ {a} = \theta_ {a} ^ {*} - \varepsilon_ {2}} ^ {\theta_ {a} ^ {*} - \varepsilon} \Delta_ {a} N _ {a} h _ {a} \left(\theta_ {a}\right) d \theta_ {a} h _ {- a} \left(\theta_ {- a}\right) d \theta_ {- a} \\ \leq \int_ {\theta_ {- a}} \left(\int_ {\theta_ {a} = \theta_ {a} ^ {*} - \varepsilon_ {2}} ^ {\theta_ {a} ^ {*} - \varepsilon} \Delta_ {a} N _ {a}   d \theta_ {a}\right) h _ {a} \big (\theta_ {a} ^ {*} - \xi_ {a} \big (\theta_ {a} ^ {*} \big) \big) h _ {- a} \big (\theta_ {- a} \big)   d \theta_ {- a}, \\ \end{array}
$$
where the inequality is by the definition of  $\xi_{a}$ . Now the inner integral is independent of  $h_a$  and thus can be easily bounded. Specifically, the upper bound in (6) and simple integration yield
$$
\begin{array}{l} \int_ {\theta_ {a} = \theta_ {a} ^ {*} - \varepsilon_ {2}} ^ {\theta_ {a} ^ {*} - \varepsilon} \Delta_ {a} N _ {a} d \theta_ {a} \leq \int_ {\theta_ {a} = \theta_ {a} ^ {*} - \varepsilon_ {2}} ^ {\theta_ {a} ^ {*} - \varepsilon} \left(\frac {8 \sigma^ {2} \log (1 / \delta)}{\theta_ {a} ^ {*} - \theta_ {a}} - \sigma^ {2} \sigma_ {0} ^ {- 2} (\theta_ {a} ^ {*} - \theta_ {a})\right) d \theta_ {a} \\ = 8 \sigma^ {2} \log (1 / \delta) (\log \varepsilon_ {2} - \log \varepsilon) - \frac {\sigma^ {2} (\varepsilon_ {2} ^ {2} - \varepsilon^ {2})}{2 \sigma_ {0} ^ {2}}. \\ \end{array}
$$
For  $\varepsilon = 1 / n$  and  $\varepsilon_{2} = 1 / \sqrt{\log n}$ , we get
$$
\begin{array}{l} \int_ {\theta_ {a} = \theta_ {a} ^ {*} - \varepsilon_ {2}} ^ {\theta_ {a} ^ {*} - \varepsilon} \Delta_ {a} N _ {a} d \theta_ {a} \leq 8 \sigma^ {2} \log (1 / \delta) \log n - \frac {\sigma^ {2}}{2 \sigma_ {0} ^ {2} \log n} + \frac {\sigma^ {2}}{2 \sigma_ {0} ^ {2} n ^ {2}} - 4 \sigma^ {2} \log (1 / \delta) \log \log n \tag {9} \\ \leq 8 \sigma^ {2} \log (1 / \delta) \log n - \frac {\sigma^ {2}}{2 \sigma_ {0} ^ {2} \log n}. \\ \end{array}
$$
The last inequality holds for  $\sigma_0^2 \geq \frac{1}{8\log(1 / \delta)n^2\log\log n}$ .
Case 2: Large gaps  $\Delta_{a} > \varepsilon_{2}$ . Here we use (6) together with  $\varepsilon_{2} = 1 / \sqrt{\log n}$  to get
$$
\mathbb {E} \left[ \Delta_ {a} N _ {a} \mathbb {1} \left\{\Delta_ {a} > \varepsilon_ {2} \right\} \right] \leq \mathbb {E} \left[ \frac {8 \sigma^ {2} \log (1 / \delta)}{\Delta_ {a}} \mathbb {1} \left\{\Delta_ {a} > \varepsilon_ {2} \right\} \right] <   8 \sigma^ {2} \log (1 / \delta) \sqrt {\log n}. \tag {10}
$$
Finally, we chain all inequalities.
$$
\begin{array}{l} \sum_ {a \in \mathcal {A}} \int_ {\theta_ {- a}} h _ {a} \left(\theta_ {a} ^ {*}\right) h _ {- a} \left(\theta_ {- a}\right) d \theta_ {- a} \\ \leq \sum_ {a \in \mathcal {A}} \int_ {\theta_ {- a}} \left(\sum_ {a ^ {\prime} \neq a} h _ {a} \left(\theta_ {a ^ {\prime}}\right)\right) \left(\prod_ {a ^ {\prime} \neq a} h _ {a ^ {\prime}} \left(\theta_ {a ^ {\prime}}\right)\right) d \theta_ {- a} \\ = \sum_ {a \in \mathcal {A}} \sum_ {a ^ {\prime} \neq a} \int_ {\theta_ {a ^ {\prime}}} h _ {a} (\theta_ {a ^ {\prime}}) h _ {a ^ {\prime}} (\theta_ {a ^ {\prime}}) \mathrm {d} \theta_ {a ^ {\prime}} \\ = \frac {1}{2 \pi \sigma_ {0} ^ {2}} \sum_ {a \in \mathcal {A}} \sum_ {a ^ {\prime} \neq a} \int_ {\theta_ {a ^ {\prime}}} \exp \left[ - \frac {(\theta_ {a ^ {\prime}} - \mu_ {0 , a}) ^ {2}}{2 \sigma_ {0} ^ {2}} - \frac {(\theta_ {a ^ {\prime}} - \mu_ {0 , a ^ {\prime}}) ^ {2}}{2 \sigma_ {0} ^ {2}} \right] \mathrm {d} \theta_ {a ^ {\prime}} \\ = \frac {1}{2 \sqrt {\pi \sigma_ {0} ^ {2}}} \sum_ {a \in \mathcal {A}} \sum_ {a ^ {\prime} \neq a} \exp \left[ - \frac {(\mu_ {0 , a} - \mu_ {0 , a ^ {\prime}}) ^ {2}}{4 \sigma_ {0} ^ {2}} \right], \\ \end{array}
$$
where the last step is by completing the square and integrating out  $\theta_{a'}$ .
The regret bound of UCB1 is proved similarly to Theorem 1 and Corollary 2. This is because UCB1 can be viewed as BayesUCB where  $\sigma_0 = \infty$  and each action  $a\in \mathcal{A}$  is initially taken once at round  $t = a$ . Since  $\sigma_0 = \infty$ , the confidence interval becomes
$$
C _ {t, a} = \sqrt {\frac {2 \sigma^ {2} \log (1 / \delta)}{N _ {t , a}}}.
$$
The proof differs in two steps. First, the regret in the first  $K$  rounds is bounded by  $\sum_{a\in \mathcal{A}}\mathbb{E}[\Delta_a]$ . Second, the concentration argument (Case 3 in Appendix A.1) changes because the bandit instance  $\theta$  is fixed and the estimated model parameter  $\hat{\theta}_t$  is random. We detail it below.
$$
\begin{array}{l} \mathbb {E} \left[ \Delta_ {A _ {t}} \mathbb {1} \left\{\bar {E} _ {t} \right\} \right] \tag {11} \\ = \mathbb {E} \left[ \mathbb {E} \left[ \left(\theta_ {A _ {*}} - \theta_ {A _ {t}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid \theta \right] \right] \\ \leq \mathbb {E} \left[ \mathbb {E} \left[ \left(\theta_ {A _ {*}} - U _ {t, A _ {*}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid \theta \right] + \mathbb {E} \left[ \left(U _ {t, A _ {t}} - \theta_ {A _ {t}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid \theta \right] \right] \\ \leq \mathbb {E} \left[ \mathbb {E} \left[ \left(\theta_ {A _ {*}} - \hat {\theta} _ {t, A _ {*}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid \theta \right] + \mathbb {E} \left[ \left(\hat {\theta} _ {t, A _ {t}} - \theta_ {A _ {t}}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid \theta \right] + \mathbb {E} \left[ C _ {t, A _ {t}} \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid \theta \right] \right]. \\ \end{array}
$$
To bound the resulting terms, we use that  $\theta_{a} - \hat{\theta}_{t,a}\mid \theta \sim \mathcal{N}(0,\sigma^{2} / N_{t,a})$
Lemma 8. For any action  $a \in \mathcal{A}$ , round  $t > K$ , and  $N_{t,a} \geq 1$
$$
\mathbb {E} \left[ \left(\theta_ {a} - \hat {\theta} _ {t, a}\right) \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid \theta \right] \leq 2 \sigma K \delta .
$$
$$
\mathbb {E} \left[ (\theta_ {a} - \hat {\theta} _ {t, a}) \mathbb {1} \left\{\bar {E} _ {t} \right\} \Big | \theta \right] \leq \mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mathbb {1} \left\{\bar {E} _ {t, a} \right\} \Big | \theta \right] + \sum_ {a ^ {\prime} \neq a} \mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mathbb {1} \left\{\bar {E} _ {t, a ^ {\prime}} \right\} \Big | \theta \right].
$$
To bound the first term, we use that  $\theta_{a} - \hat{\theta}_{t,a} \mid \theta \sim \mathcal{N}(0, \sigma^{2} / N_{t,a})$ . Thus
$$
\begin{array}{l} \mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid \theta \right] \leq \frac {2}{\sqrt {2 \pi \sigma^ {2} / N _ {t , a}}} \int_ {x = C _ {t, a}} ^ {\infty} x \exp \left[ - \frac {x ^ {2}}{2 \sigma^ {2} / N _ {t , a}} \right] d x \\ = - \sqrt {\frac {2 \sigma^ {2}}{\pi N _ {t , a}}} \int_ {x = C _ {t, a}} ^ {\infty} \frac {\partial}{\partial x} \left(\exp \left[ - \frac {x ^ {2}}{2 \sigma^ {2} / N _ {t , a}} \right]\right) d x \\ = \sqrt {\frac {2 \sigma^ {2}}{\pi N _ {t , a}}} \delta \leq \sigma \delta . \\ \end{array}
$$
$$
\mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mathbb {1} \left\{\bar {E} _ {t, a ^ {\prime}} \right\} \mid \theta \right] = \mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mid \theta \right] \mathbb {P} \left(\bar {E} _ {t, a ^ {\prime}} \mid \theta\right).
$$
$$
\mathbb {E} \left[ | \theta_ {a} - \hat {\theta} _ {t, a} | \mid \theta \right] = \mathbb {E} \left[ \sqrt {(\theta_ {a} - \hat {\theta} _ {t , a}) ^ {2}} \mid \theta \right] \leq \sqrt {\mathbb {E} \left[ (\theta_ {a} - \hat {\theta} _ {t , a}) ^ {2} \mid \theta \right]} = \sqrt {\frac {\sigma^ {2}}{N _ {t , a}}} \leq \sigma .
$$
The first two terms in (11) can be bounded using a union bound over  $a \in \mathcal{A}$  and Lemma 8. For the last term, we use that  $C_{t,a} \leq \sqrt{2\sigma^2\log(1 / \delta)}$  and a union bound in  $\mathbb{P}\left(\bar{E}_t \mid \theta\right)$  to get
$$
\mathbb {E} \left[ C _ {t, A _ {t}} \mathbb {1} \left\{\bar {E} _ {t} \right\} \mid \theta \right] \leq \sqrt {2 \sigma^ {2} \log (1 / \delta)} \mathbb {P} \left(\bar {E} _ {t} \mid \theta\right) \leq 2 \sqrt {2 \log (1 / \delta)} \sigma K \delta .
$$
Finally, we sum up the upper bounds on (11) over all rounds  $t \in [n]$ .
Let  $E_{t} = \left\{\forall a\in \mathcal{A}:|\theta_{a} - \hat{\theta}_{t,a}|\leq C_{t,a}\right\}$  be the event that all confidence intervals at round  $t$  hold. Fix  $\varepsilon >0$ . We decompose the  $n$ -round regret as in (5) and then bound each resulting term next.
Case 1: Event  $E_{t}$  occurs and the gap is large,  $\Delta_{A_t} \geq \varepsilon$ . As in Appendix A.1,
$$
N _ {t, a} \leq \frac {2 \log (1 / \delta)}{\Delta_ {a} ^ {2}} - (\alpha_ {a} + \beta_ {a} + 1).
$$
$$
\sum_ {t = 1} ^ {n} \mathbb {E} \left[ \Delta_ {A _ {t}} \mathbb {1} \left\{\Delta_ {A _ {t}} \geq \varepsilon , E _ {t} \right\} \right] \leq \mathbb {E} \left[ \sum_ {a \neq A _ {*}} \left(\frac {2 \log (1 / \delta)}{\Delta_ {a}} - (\alpha_ {a} + \beta_ {a} + 1) \Delta_ {a}\right) \mathbb {1} \left\{\Delta_ {a} \geq \varepsilon \right\} \right].
$$
Case 3: Event  $E_{t}$  does not occur. Since  $\theta_{a} \in [0,1]$ , the last term in (5) can be bounded as
$$
\mathbb {E} \left[ \Delta_ {A _ {t}} \mathbb {1} \left\{\bar {E} _ {t} \right\} \right] \leq \mathbb {E} \left[ \mathbb {P} \left(\bar {E} _ {t} \mid H _ {t}\right) \right] \leq 2 K \delta .
$$
This completes the first part of the proof.
The second claim is proved as in Appendix A.2 and we only comment on what differs. For  $\varepsilon = 1 / n$  and  $\varepsilon_{2} = 1 / \sqrt{\log n}$ , (9) becomes
$$
\begin{array}{l} \int_ {\theta_ {a} = \theta_ {a} ^ {*} - \varepsilon_ {2}} ^ {\theta_ {a} ^ {*} - \varepsilon} \Delta_ {a} N _ {a} d \theta_ {a} \leq \int_ {\theta_ {a} = \theta_ {a} ^ {*} - \varepsilon_ {2}} ^ {\theta_ {a} ^ {*} - \varepsilon} \frac {2 \log (1 / \delta)}{\theta_ {a} ^ {*} - \theta_ {a}} - \lambda (\theta_ {a} ^ {*} - \theta_ {a}) d \theta_ {a} \\ = 2 \log (1 / \delta) (\log \varepsilon_ {2} - \log \varepsilon) - \frac {\lambda (\varepsilon_ {2} ^ {2} - \varepsilon^ {2})}{2} \\ = 2 \log (1 / \delta) \log n - \frac {\lambda}{2 \log n} + \frac {\lambda}{2 n ^ {2}} - \log (1 / \delta) \log \log n \\ \leq 2 \log (1 / \delta) \log n - \frac {\lambda}{2 \log n}. \\ \end{array}
$$
The last inequality holds for  $\lambda \leq 2\log (1 / \delta)n^2\log \log n$ . Moreover, (10) becomes
$$
\mathbb {E} \left[ \Delta_ {a} N _ {a} \mathbb {1} \{\Delta_ {a} > \varepsilon_ {2} \} \right] \leq \mathbb {E} \left[ \frac {2 \log (1 / \delta)}{\Delta_ {a}} \mathbb {1} \{\Delta_ {a} > \varepsilon_ {2} \} \right] <   2 \log (1 / \delta) \sqrt {\log n}.
$$
This completes the second part of the proof.
$$
E _ {t} = \left\{\forall a \in \mathcal {A}: | a ^ {\top} (\theta - \hat {\theta} _ {t}) | \leq \sqrt {2 \log (1 / \delta)} \| a \| _ {\hat {\Sigma} _ {t}} \right\} \tag {12}
$$
be an event that high-probability confidence intervals for mean rewards at round  $t$  hold. Our proof has three parts.
Case 1: Event  $E_{t}$  occurs and the gap is large,  $\Delta_{A_t} \geq \varepsilon$ . Then
$$
\begin{array}{l} \Delta_ {A _ {t}} = \frac {1}{\Delta_ {A _ {t}}} \Delta_ {A _ {t}} ^ {2} \leq \frac {1}{\Delta_ {\min} ^ {\varepsilon}} (A _ {*} ^ {\top} \theta - A _ {t} ^ {\top} \theta) ^ {2} \leq \frac {1}{\Delta_ {\min} ^ {\varepsilon}} (A _ {*} ^ {\top} \theta - U _ {t, A _ {*}} + U _ {t, A _ {t}} - A _ {t} ^ {\top} \theta) ^ {2} \\ \leq \frac {1}{\Delta_ {\mathrm {m i n}} ^ {\varepsilon}} (U _ {t, A _ {t}} - A _ {t} ^ {\top} \theta) ^ {2} \leq \frac {4}{\Delta_ {\mathrm {m i n}} ^ {\varepsilon}} C _ {t, A _ {t}} ^ {2} = \frac {8 \log (1 / \delta)}{\Delta_ {\mathrm {m i n}} ^ {\varepsilon}} \| A _ {t} \| _ {\dot {\Sigma} _ {t}} ^ {2}. \\ \end{array}
$$
The first inequality follows from definitions of  $\Delta_{A_t}$  and  $\Delta_{\min}^{\varepsilon}$ ; and that the gap is large,  $\Delta_{A_t} \geq \varepsilon$ . The second inequality holds because  $A_*^\top \theta - A_t^\top \theta \geq 0$  by definition and  $U_{t,A_t} - U_{t,A_*} \geq 0$  by the design of BayesUCB. The third inequality holds because  $A_*^\top \theta - U_{t,A_*} \leq 0$  on event  $E_t$ . Specifically, for any action  $a \in \mathcal{A}$  on event  $E_t$ ,
$$
a ^ {\top} \theta - U _ {t, a} = a ^ {\top} (\theta - \hat {\theta} _ {t}) - C _ {t, a} \leq C _ {t, a} - C _ {t, a} = 0.
$$
The last inequality follows from the definition of event  $E_{t}$ . Specifically, for any action  $a \in \mathcal{A}$  on event  $E_{t}$ ,
$$
U _ {t, a} - a ^ {\top} \theta = a ^ {\top} \left(\hat {\theta} _ {t} - \theta\right) + C _ {t, a} \leq C _ {t, a} + C _ {t, a} = 2 C _ {t, a}.
$$
Case 2: The gap is small,  $\Delta_{A_t} \leq \varepsilon$ . Then naively  $\Delta_{A_t} \leq \varepsilon$ .
Case 3: Event  $E_{t}$  does not occur. Then  $\Delta_{A_t}\mathbb{1}\{\bar{E}_t\} \leq 2\| A_t\|_2\|\theta\|_2\mathbb{1}\{\bar{E}_t\} \leq 2LL_*\mathbb{1}\{\bar{E}_t\}$ , where  $2LL_*$  is a trivial upper bound on  $\Delta_{A_t}$ . We bound the event in expectation as follows.
Lemma 9. For any round  $t \in [n]$  and history  $H_t$ , we have that  $\mathbb{P}\left(\bar{E}_t \mid H_t\right) \leq 2K\delta$ .
Proof. First, note that for any history  $H_{t}$
$$
\mathbb {P} \left(\bar {E} _ {t} \mid H _ {t}\right) \leq \sum_ {a \in \mathcal {A}} \mathbb {P} \left(| a ^ {\top} (\theta - \hat {\theta} _ {t}) | \geq \sqrt {2 \log (1 / \delta)} \| a \| _ {\hat {\Sigma} _ {t}} \mid H _ {t}\right).
$$
By definition,  $\theta -\hat{\theta}_t\mid H_t\sim \mathcal{N}(\mathbf{0}_d,\hat{\Sigma}_t)$ , and therefore  $a^\top (\theta -\hat{\theta}_t) / \| a\|_{\hat{\Sigma}_t}\mid H_t\sim \mathcal{N}(0,1)$  for any action  $a\in \mathcal{A}$ . It immediately follows that
$$
\mathbb {P} \left( \right.\left| a ^ {\top} \left(\theta - \hat {\theta} _ {t}\right)\right| \geq \sqrt {2 \log (1 / \delta)} \| a \| _ {\hat {\Sigma} _ {t}} \left. \right\rvert   H _ {t}\left. \right) \leq 2 \delta .
$$
Finally, we chain all inequalities, add them over all rounds, and get
$$
R (n) \leq 8 \mathbb {E} \left[ \frac {1}{\Delta_ {\min } ^ {\varepsilon}} \sum_ {t = 1} ^ {n} \| A _ {t} \| _ {\hat {\Sigma} _ {t}} ^ {2} \right] \log (1 / \delta) + \varepsilon n + 4 L L _ {*} K n \delta .
$$
The sum can be bounded using a worst-case argument below, which yields our claim.
Lemma 10. The sum of posterior variances is bounded as
$$
\sum_ {t = 1} ^ {n} \| A _ {t} \| _ {\hat {\Sigma} _ {t}} ^ {2} \leq \frac {\sigma_ {0 , \mathrm {m a x}} ^ {2} d}{\log \left(1 + \frac {\sigma_ {0 , \mathrm {m a x}} ^ {2}}{\sigma^ {2}}\right)} \log \left(1 + \frac {\sigma_ {0 , \mathrm {m a x}} ^ {2} n}{\sigma^ {2} d}\right).
$$
Proof. We start with an upper bound on the posterior variance of the mean reward estimate of any action. In any round  $t \in [n]$ , by Weyl's inequalities, we have
$$
\lambda_ {1} (\hat {\Sigma} _ {t}) = \lambda_ {1} \big ((\Sigma_ {0} ^ {- 1} + G _ {t}) ^ {- 1} \big) = \lambda_ {d} ^ {- 1} (\Sigma_ {0} ^ {- 1} + G _ {t}) \leq \lambda_ {d} ^ {- 1} (\Sigma_ {0} ^ {- 1}) = \lambda_ {1} (\Sigma_ {0}).
$$
Thus, when  $\|a\|_2 \leq L$  for any action  $a \in \mathcal{A}$ , we have  $\max_{a \in \mathcal{A}} \|a\|_{\hat{\Sigma}_t} \leq \sqrt{\lambda_1(\Sigma_0)} L = \sigma_{0,\max}$ .
Now we bound the sum of posterior variances  $\sum_{t=1}^{n}\|A_t\|_{\hat{\Sigma}_t}^2$ . Fix round  $t$  and note that
$$
\left\| A _ {t} \right\| _ {\hat {\Sigma} _ {t}} ^ {2} = \sigma^ {2} \frac {A _ {t} ^ {\top} \hat {\Sigma} _ {t} A _ {t}}{\sigma^ {2}} \leq c _ {1} \log \left(1 + \sigma^ {- 2} A _ {t} ^ {\top} \hat {\Sigma} _ {t} A _ {t}\right) = c _ {1} \log \det  \left(I _ {d} + \sigma^ {- 2} \hat {\Sigma} _ {t} ^ {\frac {1}{2}} A _ {t} A _ {t} ^ {\top} \hat {\Sigma} _ {t} ^ {\frac {1}{2}}\right) \tag {13}
$$
$$
c _ {1} = \frac {\sigma_ {0 , \mathrm {m a x}} ^ {2}}{\log (1 + \sigma^ {- 2} \sigma_ {0 , \mathrm {m a x}} ^ {2})}.
$$
This upper bound is derived as follows. For any  $x \in [0, u]$ ,
$$
x = \frac {x}{\log (1 + x)} \log (1 + x) \leq \left(\max  _ {x \in [ 0, u ]} \frac {x}{\log (1 + x)}\right) \log (1 + x) = \frac {u}{\log (1 + u)} \log (1 + x).
$$
Then we set  $x = \sigma^{-2}A_t^\top \hat{\Sigma}_tA_t$  and use the definition of  $\sigma_{0,\max}$ .
The next step is bounding the logarithmic term in (13), which can be rewritten as
$$
\log \det  (\boldsymbol {I} _ {d} + \sigma^ {- 2} \hat {\Sigma} _ {t} ^ {\frac {1}{2}} A _ {t} A _ {t} ^ {\top} \hat {\Sigma} _ {t} ^ {\frac {1}{2}}) = \log \det  (\hat {\Sigma} _ {t} ^ {- 1} + \sigma^ {- 2} A _ {t} A _ {t} ^ {\top}) - \log \det  (\hat {\Sigma} _ {t} ^ {- 1}).
$$
Because of that, when we sum over all rounds, we get telescoping and the total contribution of all terms is at most
$$
\begin{array}{l} \sum_ {t = 1} ^ {n} \log \det  \left(I _ {d} + \sigma^ {- 2} \hat {\Sigma} _ {t} ^ {\frac {1}{2}} A _ {t} A _ {t} ^ {\top} \hat {\Sigma} _ {t} ^ {\frac {1}{2}}\right) = \log \det  \left(\hat {\Sigma} _ {n + 1} ^ {- 1}\right) - \log \det  \left(\hat {\Sigma} _ {1} ^ {- 1}\right) \\ = \log \det \left(\Sigma_ {0} ^ {\frac {1}{2}} \hat {\Sigma} _ {n + 1} ^ {- 1} \Sigma_ {0} ^ {\frac {1}{2}}\right) \\ \leq d \log \left(\frac {1}{d} \operatorname {t r} \left(\Sigma_ {0} ^ {\frac {1}{2}} \hat {\Sigma} _ {n + 1} ^ {- 1} \Sigma_ {0} ^ {\frac {1}{2}}\right)\right) \\ = d \log \left(1 + \frac {1}{\sigma^ {2} d} \sum_ {t = 1} ^ {n} \operatorname {t r} \left(\Sigma_ {0} ^ {\frac {1}{2}} A _ {t} A _ {t} ^ {\top} \Sigma_ {0} ^ {\frac {1}{2}}\right)\right) \\ = d \log \left(1 + \frac {1}{\sigma^ {2} d} \sum_ {t = 1} ^ {n} A _ {t} ^ {\top} \Sigma_ {0} A _ {t}\right) \\ \leq d \log \left(1 + \frac {\sigma_ {0 , \operatorname* {m a x}} ^ {2} n}{\sigma^ {2} d}\right). \\ \end{array}
$$
B Complete Statement of Corollary 2
Theorem 11. Let  $\sigma_0^2 \geq \frac{1}{8\log(1 / \delta)n^2\log\log n}$ . Then there exist functions  $\xi_{a}:\mathbb{R}\to \left[\frac{1}{n},\frac{1}{\sqrt{\log n}}\right]$  such that the  $n$ -round Bayes regret of BayesUCB in a  $K$ -armed Gaussian bandit is bounded as
where  $C = 8\sigma^2 K\log (1 / \delta)\sqrt{\log n} +(2\sqrt{2\log(1 / \delta)} +1)\sigma_0Kn\delta +1$  is a low-order term.
Moreover, when  $\sigma_0^2 < \frac{1}{8\log(1 / \delta)n^2\log\log n}$ , the regret is bounded as
$$
R (n) \leq \frac {2 \sqrt {2 \log (1 / \delta)} + 1}{\sqrt {8 \log (1 / \delta) \log \log n}} K \delta + 1.
$$
Proof. The first claim is proved in Appendix A.2. The second claim can be proved as follows. Take Theorem 1, set  $\varepsilon = 0$ , and consider the three cases in Appendix A.1.
Case 1: Event  $E_{t}$  occurs and the gap is large,  $\Delta_{A_t} \geq \varepsilon$ . On event  $E_{t}$ , action  $a$  can be taken only if
$$
\Delta_ {a} \leq 2 \sqrt {\frac {2 \log (1 / \delta)}{\sigma_ {0} ^ {- 2} + \sigma^ {- 2} N _ {t , a}}} \leq 2 \sqrt {2 \sigma_ {0} ^ {2} \log (1 / \delta)} \leq 2 \sqrt {\frac {1}{4 n ^ {2} \log \log n}} <   \frac {1}{n}.
$$
Therefore, the corresponding  $n$ -round regret is bounded by 1.
Case 2: The gap is small,  $\Delta_{A_t} < \varepsilon$ . This case cannot happen because  $\varepsilon = 0$ .
Case 3: Event  $E_{t}$  does not occur. The  $n$ -round regret is bounded by
$$
(2 \sqrt {2 \log (1 / \delta)} + 1) \sigma_ {0} K n \delta \leq \frac {2 \sqrt {2 \log (1 / \delta)} + 1}{\sqrt {8 \log (1 / \delta) \log \log n}} K \delta .
$$
C Gap-Free Regret Bound of BayesUCB in Linear Bandit
Let  $E_{t}$  be the event in (12). Our proof has three parts.