SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finite-Time Logarithmic Bayes Regret Upper Bounds
Alexia Atsidakou  
University of Texas, Austin
Branislav Kveton  
AWS AI Labs*
Sumeet Katariya  
Amazon
Constantine Caramanis  
University of Texas, Austin
Sujay Sanghavi  
University of Texas, Austin / Amazon
We derive the first finite-time logarithmic Bayes regret upper bounds for Bayesian bandits. In a multi-armed bandit, we obtain  $O(c_{\Delta} \log n)$  and  $O(c_h \log^2 n)$  upper bounds for an upper confidence bound algorithm, where  $c_h$  and  $c_{\Delta}$  are constants depending on the prior distribution and the gaps of bandit instances sampled from it, respectively. The latter bound asymptotically matches the lower bound of Lai (1987). Our proofs are a major technical departure from prior works, while being simple and general. To show the generality of our techniques, we apply them to linear bandits. Our results provide insights on the value of prior in the Bayesian setting, both in the objective and as a side information given to the learner. They significantly improve upon existing  $\tilde{O}(\sqrt{n})$  bounds, which have become standard in the literature despite the logarithmic lower bound of Lai (1987).
A stochastic multi-armed bandit [Lai and Robbins, 1985, Auer et al., 2002, Lattimore and Szepesvari, 2019] is an online learning problem where a learner sequentially interacts with an environment over  $n$  rounds. In each round, the learner takes an action and receives its stochastic reward. The goal of the learner is to maximize its expected cumulative reward over  $n$  rounds. The mean rewards of the actions are unknown a priori but can be learned by taking the actions. Therefore, the learner faces the exploration-exploitation dilemma: explore, and learn more about the actions; or exploit, and take the action with the highest estimated reward. Bandits have been successfully applied to problems where uncertainty modeling and subsequent adaptation are beneficial. One example is recommender systems [Li et al., 2010, Zhao et al., 2013, Kawale et al., 2015, Li et al., 2016], where the actions are recommended items and their rewards are clicks. Another example is hyper-parameter optimization [Li et al., 2018], where the actions are values of the optimized parameters and their reward is the optimized metric.
Cumulative regret minimization in stochastic bandits has been traditionally studied in two settings: frequentist [Lai and Robbins, 1985, Auer et al., 2002, Abbasi-Yadkori et al., 2011] and Bayesian [Gittins, 1979, Tsitsiklis, 1994, Lai, 1987, Russo and Van Roy, 2014, Russo et al., 2018]. In the frequentist setting, the learner minimizes the regret with respect to a fixed unknown bandit instance. In the Bayesian setting, the learner minimizes the average regret with respect to bandit instances drawn from a prior distribution. The instance is unknown but the learner knows its prior distribution. The Bayesian setting allows surprisingly simple and insightful analyses of Thompson sampling. One fundamental result in this setting is that linear Thompson sampling [Russo and Van Roy, 2014] has a comparable regret bound to LinUCB in the frequentist setting [Abbasi-Yadkori et al., 2011, Agrawal and Goyal, 2013, Abeille and Lazaric, 2017]. Moreover, many recent meta- and multi-task bandit works [Bastani et al., 2019, Kveton et al., 2021, Basu et al., 2021, Simchowitz et al., 2021, Wang et al., 2021, Hong et al., 2022, Aouali et al., 2023] adopt the Bayes regret to analyze the stochastic
structure of their problems, that the bandit tasks are similar because their parameters are sampled i.i.d. from a task distribution.
Many bandit algorithms have frequentist regret bounds that match a lower bound. As an example, in a  $K$ -armed bandit with the minimum gap  $\Delta$  and horizon  $n$ , the gap-dependent  $O(K\Delta^{-1}\log n)$  regret bound of UCB1 [Auer et al., 2002] matches the gap-dependent  $\Omega(K\Delta^{-1}\log n)$  lower bound of Lai and Robbins [1985]. Moreover, the gap-free  $\tilde{O}(\sqrt{Kn})$  regret bound of UCB1 matches, up to logarithmic factors, the gap-free  $\Omega(\sqrt{Kn})$  lower bound of Auer et al. [1995]. The extra logarithmic factor in the  $\tilde{O}(\sqrt{Kn})$  bound can be eliminated by modifying UCB1 [Audibert and Bubeck, 2009]. In contrast, and despite the popularity of the model, matching upper and lower bounds mostly do not exist in the Bayesian setting. Specifically, Lai [1987] proved asymptotic  $c_h\log^2 n$  upper and lower bounds, where  $c_h$  is a prior-dependent constant. However, all recent Bayes regret bounds are  $\tilde{O}(\sqrt{n})$  [Russo and Van Roy, 2014, 2016, Lu and Van Roy, 2019, Hong et al., 2020, Kveton et al., 2021]. This leaves open the question of finite-time logarithmic regret bounds in the Bayesian setting.
In this work, we answer this question positively and make the following contributions:
1. We derive the first finite-time logarithmic Bayes regret upper bounds for a Bayesian upper confidence bound (UCB) algorithm. The bounds are  $O(c_{\Delta} \log n)$  and  $O(c_h \log^2 n)$ , where  $c_h$  and  $c_{\Delta}$  are constants depending on the prior distribution  $h$  and the gaps of random bandit instances sampled from  $h$ , respectively. The latter matches the lower bound of Lai [1987] asymptotically. When compared to prior  $\tilde{O}(\sqrt{n})$  bounds, we better characterize low-regret regimes, where the random gaps are large.
2. To show the value of prior as a side information, we also derive a finite-time logarithmic Bayes regret upper bound for a frequentist UCB algorithm. The bound changes only little as the prior becomes more informative, while the regret bound for the Bayesian algorithm eventually goes to zero. The bounds match asymptotically when  $n \to \infty$  and the prior is overtaken by data.
3. To show the generality of our approach, we prove a  $O(d c_{\Delta} \log^2 n)$  Bayes regret bound for a Bayesian linear bandit algorithm, where  $d$  denotes the number of dimensions and  $c_{\Delta}$  is a constant depending on random gaps. This bound also improves with a better prior.
4. Our analyses are a major departure from all recent Bayesian bandit analyses, starting with Russo and Van Roy [2014]. Roughly speaking, we first bound the regret in a fixed bandit instance, similarly to frequentist analyses, and then integrate out the random gap.
5. We show the tightness of our bounds empirically and compare them to prior bounds.
This paper is organized as follows. In Section 2, we introduce the setting of Bayesian bandits. In Section 3, we present a Bayesian upper confidence bound algorithm called BayesUCB [Kaufmann et al., 2012]. In Section 4, we derive finite-time logarithmic Bayes regret bounds for BayesUCB, in both multi-armed and linear bandits. These are the first such bounds ever derived. In Section 5, we compare our bounds to prior works and show that one matches an existing lower bound [Lai, 1987] asymptotically. In Section 6, we evaluate the bounds empirically. We conclude in Section 7.
We start with introducing our notation. Random variables are capitalized, except for Greek letters like  $\theta$ . For any positive integer  $n$ , we define  $[n] = \{1, \dots, n\}$ . The indicator function is  $\mathbb{1}\{\cdot\}$ . The  $i$ -th entry of vector  $v$  is  $v_i$ . If the vector is already indexed, such as  $v_j$ , we write  $v_{j,i}$ . We denote the maximum and minimum eigenvalues of matrix  $M \in \mathbb{R}^{d \times d}$  by  $\lambda_1(M)$  and  $\lambda_d(M)$ , respectively.
Our setting is defined as follows. We have a multi-armed bandit [Lai and Robbins, 1985, Lai, 1987, Auer et al., 2002, Abbasi-Yadkori et al., 2011] with an action set  $\mathcal{A}$ . Each action  $a \in \mathcal{A}$  is associated with a reward distribution  $p_a(\cdot; \theta)$ , which is parameterized by an unknown model parameter  $\theta$  shared by all actions. The learner interacts with the bandit instance for  $n$  rounds indexed by  $t \in [n]$ . In each round  $t$ , it takes an action  $A_t \in \mathcal{A}$  and observes its stochastic reward  $Y_t \sim p_{A_t}(\cdot; \theta)$ . The rewards are sampled independently across the rounds. We denote the mean of  $p_a(\cdot; \theta)$  by  $\mu_a(\theta)$  and call it
*The work started at Amazon Search.
Algorithm 1 BayesUCB
1: for  $t = 1,\dots ,n$  do   
2: Compute the posterior distribution of  $\theta$  using prior  $h$  and history  $H_{t}$    
3: for each action  $a\in \mathcal{A}$  do   
4: Compute  $U_{t,a}$  according to (1)   
5: Take action  $A_{t}\gets \arg \max_{a\in \mathcal{A}}U_{t,a}$  and observe its reward  $Y_{t}$
the mean reward of action  $a$ . The optimal action is  $A_{*} = \arg \max_{a\in \mathcal{A}}\mu_{a}(\theta)$  and its mean reward is  $\mu_{*}(\theta) = \mu_{A_{*}}(\theta)$ . For a fixed model parameter  $\theta$ , the  $n$ -round regret of a policy is defined as
$$
R (n; \theta) = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \mu_ {*} (\theta) - \mu_ {A _ {t}} (\theta) \Bigg | \theta \right],
$$
where the expectation is taken over both random observations  $Y_{t}$  and actions  $A_{t}$ . The suboptimality gap of action  $a$  is  $\Delta_{a} = \mu_{*}(\theta) - \mu_{a}(\theta)$  and the minimum gap is  $\Delta_{\mathrm{min}} = \min_{a\in \mathcal{A}\setminus \{A_{*}\}}\Delta_{a}$ .
Two settings are common in stochastic bandits. In the frequentist setting [Lai and Robbins, 1985, Auer et al., 2002, Abbasi-Yadkori et al., 2011], the learner has no additional information about  $\theta$  and its objective is to minimize the worst-case regret for any bounded  $\theta$ . We study the Bayesian setting [Gittins, 1979, Lai, 1987, Russo and Van Roy, 2014, Russo et al., 2018], where the model parameter  $\theta$  is drawn from a prior distribution  $h$  that is given to the learner as a side information. The goal of the learner is to minimize the  $n$ -round Bayes regret  $R(n) = \mathbb{E}[R(n; \theta)]$ , where the expectation is taken over the random model parameter  $\theta \sim h$ . Note that  $A_{*}$ ,  $\Delta_{a}$ , and  $\Delta_{\mathrm{min}}$  are random because they depend on the random instance  $\theta$ .
We study a Bayesian upper confidence bound algorithm called BayesUCB [Kaufmann et al., 2012]. The algorithm was analyzed in the Bayesian setting by Russo and Van Roy [2014]. The key idea in BayesUCB is to take the action with the highest UCB with respect to the posterior distribution of model parameter  $\theta$ . This differentiates it from frequentist algorithms, such as UCB1 [Auer et al., 2002] and LinUCB [Abbasi-Yadkori et al., 2011], where the UCBs are computed using a frequentist maximum likelihood estimate (MLE) of the model parameter.
Let  $H_{t} = (A_{\ell},Y_{\ell})_{\ell \in [t - 1]}$  be the history of taken actions and their observed rewards up to round  $t$ . The Bayesian UCB for the mean reward of action  $a$  at round  $t$  is
$$
U _ {t, a} = \mu_ {a} \left(\hat {\theta} _ {t}\right) + C _ {t, a}, \tag {1}
$$
where  $\hat{\theta}_t$  is the posterior mean estimate of  $\theta$  at round  $t$  and  $C_{t,a}$  is a confidence interval width for action  $a$  at round  $t$ . The posterior distribution of model parameter  $\theta$  is computed from a prior  $h$  and history  $H_t$  using Bayes' rule. The width is chosen so that  $|\mu_a(\hat{\theta}_t) - \mu_a(\theta)| \leq C_{t,a}$  holds with a high probability conditioned on any history  $H_t$ . Technically speaking,  $C_{t,a}$  is a half-width but we call it a width to simplify terminology.
Our algorithm is presented in Algorithm 1. We instantiate it in a Gaussian bandit in Section 3.1, in a Bernoulli bandit in Section 3.2, and in a linear bandit with Gaussian rewards in Section 3.3. These settings are of practical interest because they lead to computationally-efficient implementations that can be analyzed due to closed-form posteriors [Lu and Van Roy, 2019, Kveton et al., 2021, Basu et al., 2021, Wang et al., 2021, Hong et al., 2022]. While we focus on deriving logarithmic Bayes regret bounds for BayesUCB, we believe that similar analyses can be done for Thompson sampling [Thompson, 1933, Chapelle and Li, 2012, Agrawal and Goyal, 2012, 2013, Russo and Van Roy, 2014, Russo et al., 2018]. This extension is non-trivial because a key step in our analysis is that the action with the highest UCB is taken (Section 5.3).
3.1 Gaussian Bandit
In a  $K$ -armed Gaussian bandit, the action set is  $\mathcal{A} = [K]$  and the model parameter is  $\theta \in \mathbb{R}^K$ . Each action  $a \in \mathcal{A}$  has a Gaussian reward distribution,  $p_a(\cdot; \theta) = \mathcal{N}(\cdot; \theta_a, \sigma^2)$ , where  $\theta_a$  is its mean and
$\sigma > 0$  is a known reward noise. Thus  $\mu_{a}(\theta) = \theta_{a}$ . The model parameter  $\theta$  is drawn from a known Gaussian prior  $h(\cdot) = \mathcal{N}(\cdot; \mu_{0}, \sigma_{0}^{2} I_{K})$ , where  $\mu_{0} \in \mathbb{R}^{K}$  is a vector of prior means and  $\sigma_{0} > 0$  is a prior width.
The posterior distribution of the mean reward of action  $a$  at round  $t$  is  $\mathcal{N}(\cdot ;\hat{\theta}_{t,a},\hat{\sigma}_{t,a}^2)$ , where
$$
\hat {\sigma} _ {t, a} ^ {2} = \left(\sigma_ {0} ^ {- 2} + \sigma^ {- 2} N _ {t, a}\right) ^ {- 1}
$$
is the posterior variance,  $N_{t,a} = \sum_{\ell = 1}^{t - 1}\mathbb{1}\{A_\ell = a\}$  is the number of observations of action  $a$  up to round  $t$ , and
$$
\hat {\theta} _ {t, a} = \hat {\sigma} _ {t, a} ^ {2} \left(\sigma_ {0} ^ {- 2} \mu_ {0, a} + \sigma^ {- 2} \sum_ {\ell = 1} ^ {t - 1} \mathbb {1} \{A _ {\ell} = a \} Y _ {\ell}\right)
$$
is the posterior mean. This follows from a classic result, that the posterior distribution of the mean of a Gaussian random variable with a Gaussian prior is a Gaussian [Bishop, 2006]. The Bayesian UCB of action  $a$  at round  $t$  is  $U_{t,a} = \hat{\theta}_{t,a} + C_{t,a}$ , where  $C_{t,a} = \sqrt{2\hat{\sigma}_{t,a}^2\log(1 / \delta)}$  is the confidence interval width and  $\delta \in (0,1)$  is a failure probability of the confidence interval.
3.2 Bernoulli Bandit
In a  $K$ -armed Bernoulli bandit, the action set is  $\mathcal{A} = [K]$  and the model parameter is  $\theta \in \mathbb{R}^K$ . Each action  $a \in \mathcal{A}$  has a Bernoulli reward distribution,  $p_a(\cdot; \theta) = \mathrm{Ber}(\cdot; \theta_a)$ , where  $\theta_a$  is its mean. Hence  $\mu_a(\theta) = \theta_a$ . Each parameter  $\theta_a$  is drawn from a known prior  $\mathrm{Beta}(\cdot; \alpha_a, \beta_a)$ , where  $\alpha_a > 0$  and  $\beta_a > 0$  are positive and negative prior pseudo-counts, respectively.
The posterior distribution of the mean reward of action  $a$  at round  $t$  is  $\mathrm{Beta}(\cdot ;\alpha_{t,a},\beta_{t,a})$ , where
$$
\alpha_ {t, a} = \alpha_ {a} + \sum_ {\ell = 1} ^ {t - 1} \mathbb {1} \left\{A _ {\ell} = a \right\} Y _ {\ell}, \quad \beta_ {t, a} = \beta_ {a} + \sum_ {\ell = 1} ^ {t - 1} \mathbb {1} \left\{A _ {\ell} = a \right\} (1 - Y _ {\ell}).
$$
This follows from a classic result, that the posterior distribution of the mean of a Bernoulli random variable with a beta prior is a beta distribution [Bishop, 2006]. The corresponding Bayesian UCB is  $U_{t,a} = \hat{\theta}_{t,a} + C_{t,a}$ , where
$$
\hat {\theta} _ {t, a} = \frac {\alpha_ {t , a}}{\alpha_ {t , a} + \beta_ {t , a}}, C _ {t, a} = \sqrt {\frac {\log (1 / \delta)}{2 (\alpha_ {t , a} + \beta_ {t , a} + 1)}} = \sqrt {\frac {\log (1 / \delta)}{2 (\alpha_ {a} + \beta_ {a} + N _ {t , a} + 1)}},
$$
denote the posterior mean and confidence interval width, respectively, of action  $a$  at round  $t$ ; and  $\delta \in (0,1)$  is a failure probability of the confidence interval. The confidence interval is derived using the fact that  $\mathrm{Beta}(\cdot ;\alpha_{t,a},\beta_{t,a})$  is a sub-Gaussian distribution with variance proxy  $\frac{1}{4(\alpha_a + \beta_a + N_{t,a} + 1)}$  [Marchal and Arbel, 2017].
3.3 Linear Bandit with Gaussian Rewards
We also study linear bandits [Dani et al., 2008, Abbasi-Yadkori et al., 2011] with a finite number of actions  $\mathcal{A} \subseteq \mathbb{R}^d$  in  $d$  dimensions. The model parameter is  $\theta \in \mathbb{R}^d$ . All actions  $a \in \mathcal{A}$  have Gaussian reward distributions,  $p_a(\cdot; \theta) = \mathcal{N}(\cdot; a^\top \theta, \sigma^2)$ , where  $\sigma > 0$  is a known reward noise. Therefore, the mean reward of action  $a$  is  $\mu_a(\theta) = a^\top \theta$ . The parameter  $\theta$  is drawn from a known multivariate Gaussian prior  $h(\cdot) = \mathcal{N}(\cdot; \theta_0, \Sigma_0)$ , where  $\theta_0 \in \mathbb{R}^d$  is its mean and  $\Sigma_0 \in \mathbb{R}^{d \times d}$  is its covariance, represented by a positive semi-definite (PSD) matrix.
The posterior distribution of  $\theta$  at round  $t$  is  $\mathcal{N}(\cdot ;\hat{\theta}_t,\hat{\Sigma}_t)$ , where
$$
\hat {\theta} _ {t} = \hat {\Sigma} _ {t} \left(\Sigma_ {0} ^ {- 1} \theta_ {0} + \sigma^ {- 2} \sum_ {\ell = 1} ^ {t - 1} A _ {\ell} Y _ {\ell}\right), \quad \hat {\Sigma} _ {t} = (\Sigma_ {0} ^ {- 1} + G _ {t}) ^ {- 1}, \quad G _ {t} = \sigma^ {- 2} \sum_ {\ell = 1} ^ {t - 1} A _ {\ell} A _ {\ell} ^ {\top}.
$$
Here  $\hat{\theta}_t$  and  $\hat{\Sigma}_t$  are the posterior mean and covariance of  $\theta$ , respectively, and  $G_t$  is the outer product of the feature vectors of the taken actions up to round  $t$ . These formulas follow from a classic result,
that the posterior distribution of a linear model parameter with a Gaussian prior and observations is a Gaussian [Bishop, 2006]. The Bayesian UCB of action  $a$  at round  $t$  is  $U_{t,a} = a^{\top}\hat{\theta}_{t} + C_{t,a}$ , where  $C_{t,a} = \sqrt{2\log(1 / \delta)} \| a\|_{\hat{\Sigma}_t}$  is the confidence interval width,  $\delta \in (0,1)$  is a failure probability of the confidence interval, and  $\| a\|_M = \sqrt{a^\top Ma}$ .
4 Logarithmic Bayes Regret Upper Bounds
In this section, we present finite-time logarithmic Bayes regret bounds for BayesUCB. We derive them for both  $K$ -armed and linear bandits. One bound matches an existing lower bound of Lai [1987] asymptotically and all improve upon prior  $\tilde{O}(\sqrt{n})$  bounds. We discuss this in detail in Section 5.
4.1 BayesUCB in Gaussian Bandit
Our first regret bound is for BayesUCB in a  $K$ -armed Gaussian bandit. It depends on random gaps. To control the gaps, we clip them as  $\Delta_a^\varepsilon = \max \{\Delta_a,\varepsilon \}$ . The bound is stated below.
Theorem 1. For any  $\varepsilon >0$  and  $\delta \in (0,1)$ , the  $n$ -round Bayes regret of BayesUCB in a  $K$ -armed Gaussian bandit is bounded as
$$
R (n) \leq \mathbb {E} \left[ \sum_ {a \neq A _ {*}} \frac {8 \sigma^ {2} \log (1 / \delta)}{\Delta_ {a} ^ {\varepsilon}} - \frac {\sigma^ {2} \Delta_ {a} ^ {\varepsilon}}{\sigma_ {0} ^ {2}} \right] + C,
$$
where  $C = \varepsilon n + 2(\sqrt{2\log(1 / \delta)} +2K)\sigma_0Kn\delta$  is a low-order term.
The proof is in Appendix A.1. For  $\varepsilon = 1/n$  and  $\delta = 1/n$ , the bound is  $O(c_{\Delta} \log n)$ , where  $c_{\Delta}$  is a constant depending on the gaps of random bandit instances. The dependence on  $\sigma_0$  in the low-order term  $C$  can be reduced to  $\min \{ \sigma_0, \sigma \}$  by a more elaborate analysis, where the regret of taking each action for the first time is bounded separately. This also applies to Corollary 2.
Now we derive an upper bound on Theorem 1 that eliminates the dependence on random gaps. To state it, we need to introduce additional notation. For any action  $a$ , we denote all action parameters except for  $a$  by  $\theta_{-a} = (\theta_1, \ldots, \theta_{a-1}, \theta_{a+1}, \ldots, \theta_K)$  and the corresponding optimal action in  $\theta_{-a}$  by  $\theta_a^* = \max_{j \in \mathcal{A} \setminus \{a\}} \theta_j$ . We denote by  $h_a$  the prior density of  $\theta_a$  and by  $h_{-a}$  the prior density of  $\theta_{-a}$ . Since the prior is factored (Section 3.1), note that  $h(\theta) = h_a(\theta_a) h_{-a}(\theta_{-a})$  for any  $\theta$  and action  $a$ . To keep the result clean, we state it for a "sufficiently" large prior variance. A complete statement for all prior variances is given in Appendix B. We note that the setting of small prior variances favors Bayesian algorithms since their regret decreases with a more informative prior. In fact, we show in Appendix B that the regret of BayesUCB is  $O(1)$  for a sufficiently small  $\sigma_0$ .
Corollary 2. Let  $\sigma_0^2 \geq \frac{1}{8\log(1 / \delta)n^2\log\log n}$ . Then there exist functions  $\xi_{a}:\mathbb{R}\to \left[\frac{1}{n},\frac{1}{\sqrt{\log n}}\right]$  such that the  $n$ -round Bayes regret of BayesUCB in a  $K$ -armed Gaussian bandit is bounded as
$$
R (n) \leq \left[ 8 \sigma^ {2} \log (1 / \delta) \log n - \frac {\sigma^ {2}}{2 \sigma_ {0} ^ {2} \log n} \right] \sum_ {a \in \mathcal {A}} \int_ {\theta_ {- a}} h _ {a} (\theta_ {a} ^ {*} - \xi_ {a} (\theta_ {a} ^ {*})) h _ {- a} (\theta_ {- a}) \mathrm {d} \theta_ {- a} + C,
$$
where  $C = 8\sigma^2 K\log (1 / \delta)\sqrt{\log n} +2(\sqrt{2\log(1 / \delta)} +2K)\sigma_0Kn\delta +1$  is a low-order term.
The proof is in Appendix A.2. For  $\delta = 1 / n$ , the bound is  $O(c_h\log^2 n)$ , where  $c_{h}$  depends on prior  $h$  but not on the gaps of random bandit instances. This bound is motivated by Lai [1987]. The terms  $\xi_{a}$  arise due to the intermediate value theorem for function  $h_a$ . Similar terms appear in Lai [1987] but vanish in their final asymptotic claims. The rate  $1 / \sqrt{\log n}$  in the definition of  $\xi_{a}$  cannot be reduced to  $1 / \mathrm{poly}\log n$  without increasing dependence on  $n$  in other parts of the bound.
The complexity term  $\sum_{a\in \mathcal{A}}\int_{\theta_{-a}}h_a(\theta_a^* -\xi_a(\theta_a^*))h_{-a}(\theta_{-a})\mathrm{d}\theta_{-a}$  in Corollary 2 is the same as in Lai [1987] and can be interpreted as follows. Consider the asymptotic regime of  $n\to \infty$ . Then, since the range of  $\xi_{a}$  is  $\left[\frac{1}{n},\frac{1}{\sqrt{\log n}}\right]$ , the term simplifies to  $\sum_{a\in \mathcal{A}}\int_{\theta_{-a}}h_a(\theta_a^*)h_{-a}(\theta_{-a})\mathrm{d}\theta_{-a}$  and can be viewed as the distance between prior means. In a Gaussian bandit with  $K = 2$  actions, it has a closed form of  $\frac{1}{\sqrt{\pi\sigma_0^2}}\exp \left[-\frac{(\mu_{0,1} - \mu_{0,2})^2}{4\sigma_0^2}\right]$ . A general upper bound for  $K > 2$  actions is given below.
Lemma 3. In a  $K$ -armed Gaussian bandit with prior  $h(\cdot) = \mathcal{N}(\cdot ;\mu_0,\sigma_0^2 I_K)$ , we have
$$
\sum_ {a \in \mathcal {A}} \int_ {\theta_ {- a}} h _ {a} (\theta_ {a} ^ {*}) h _ {- a} (\theta_ {- a}) \mathrm {d} \theta_ {- a} \leq \frac {1}{2 \sqrt {\pi \sigma_ {0} ^ {2}}} \sum_ {a \in \mathcal {A}} \sum_ {a ^ {\prime} \neq a} \exp \left[ - \frac {(\mu_ {0 , a} - \mu_ {0 , a ^ {\prime}}) ^ {2}}{4 \sigma_ {0} ^ {2}} \right].
$$
The bound is proved in Appendix A.3 and has several interesting properties that capture low-regret regimes. First, as the prior becomes more informative and concentrated,  $\sigma_0\rightarrow 0$ , the bound goes to zero. Second, when the gaps of bandit instances sampled from the prior are large, low regret is also expected. This can happen when the prior means become more separated,  $|\mu_{0,a} - \mu_{0,a'}|\to \infty$ , or the prior becomes wider,  $\sigma_0\rightarrow \infty$ . Our bound goes to zero in both of these cases. This also implies that Bayes regret bounds are not necessarily monotone in prior parameters, such as  $\sigma_0$ .
4.2 UCB1 in Gaussian Bandit
Using a similar approach, we prove a Bayes regret bound for UCB1 [Auer et al., 2002]. We view it as BayesUCB (Section 3.1) where  $\sigma_0 = \infty$  and each action  $a\in \mathcal{A}$  is initially taken once at round  $t = a$ . This generalizes classic UCB1 to  $\sigma^2$ -sub-Gaussian noise. An asymptotic Bayes regret bound for UCB1 was proved by Lai [1987] (claim (i) in their Theorem 3). We derive a finite-time prior-dependent Bayes regret bound below.
Theorem 4. There exist functions  $\xi_{a}:\mathbb{R}\to \left[\frac{1}{n},\frac{1}{\sqrt{\log n}}\right]$  such that the  $n$  -round Bayes regret of UCB1 in a  $K$  -armed Gaussian bandit is bounded as