Title: Finite-Time Logarithmic Bayes Regret Upper Bounds
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian bandits

logarithmic regret bounds

multi-armed bandits

linear bandits
Abstract: We derive the first finite-time logarithmic Bayes regret upper bounds for Bayesian bandits. In a multi-armed bandit, we obtain $O(c_\Delta \log n)$ and $O(c_h \log^2 n)$ upper bounds for an upper confidence bound algorithm, where $c_h$ and $c_\Delta$ are constants depending on the prior distribution and the gaps of bandit instances sampled from it, respectively. The latter bound asymptotically matches the lower bound of Lai (1987). Our proofs are a major technical departure from prior works, while being simple and general. To show the generality of our techniques, we apply them to linear bandits. Our results provide insights on the value of prior in the Bayesian setting, both in the objective and as a side information given to the learner. They significantly improve upon existing $\tilde{O}(\sqrt{n})$ bounds, which have become standard in the literature despite the logarithmic lower bound of Lai (1987).

Finite-Time Logarithmic Bayes Regret Upper Bounds

Abstract

We derive the first finite-time logarithmic Bayes regret upper bounds for Bayesian bandits. In a multi-armed bandit, we obtain O ( c ∆ log n ) and O ( c h log 2 n ) upper bounds for an upper confidence bound algorithm, where c h and c ∆ are constants depending on the prior distribution and the gaps of bandit instances sampled from it, respectively. The latter bound asymptotically matches the lower bound of Lai (1987). Our proofs are a major technical departure from prior works, while being simple and general. To show the generality of our techniques, we apply them to linear bandits. Our results provide insights on the value of prior in the Bayesian setting, both in the objective and as a side information given to the learner. They significantly improve upon existing ˜ O ( √ n ) bounds, which have become standard in the literature despite the logarithmic lower bound of Lai (1987).

1 Introduction

A stochastic multi-armed bandit [Lai and Robbins, 1985, Auer et al., 2002, Lattimore and Szepesvari, 2019] is an online learning problem where a learner sequentially interacts with an environment over n rounds. In each round, the learner takes an action and receives its stochastic reward . The goal of the learner is to maximize its expected cumulative reward over n rounds. The mean rewards of the actions are unknown a priori but can be learned by taking the actions. Therefore, the learner faces the exploration-exploitation dilemma : explore , and learn more about the actions; or exploit , and take the action with the highest estimated reward. Bandits have been successfully applied to problems where uncertainty modeling and subsequent adaptation are beneficial. One example are recommender systems [Li et al., 2010, Zhao et al., 2013, Kawale et al., 2015, Li et al., 2016], where the actions are recommended items and their rewards are clicks. Another example is hyper-parameter optimization [Li et al., 2018], where the actions are values of the optimized parameters and their reward is the optimized metric.

Cumulative regret minimization in stochastic bandits has been traditionally studied in two settings: frequentist [Lai and Robbins, 1985, Auer et al., 2002, Abbasi-Yadkori et al., 2011] and Bayesian [Gittins, 1979, Tsitsiklis, 1994, Lai, 1987, Russo and Van Roy, 2014, Russo et al., 2018]. In the frequentist setting, the learner minimizes the regret with respect to a fixed unknown bandit instance. In the Bayesian setting, the learner minimizes the average regret with respect to bandit instances drawn from a prior distribution. The instance is unknown but the learner knows its prior distribution. The Bayesian setting allows surprisingly simple and insightful analyses of Thompson sampling. One fundamental result in this setting is that linear Thompson sampling [Russo and Van Roy, 2014] has a comparable regret bound to LinUCB in the frequentist setting [Abbasi-Yadkori et al., 2011, Agrawal and Goyal, 2013, Abeille and Lazaric, 2017]. Moreover, many recent meta- and multi-task bandit works [Bastani et al., 2019, Kveton et al., 2021, Basu et al., 2021, Simchowitz et al., 2021, Wang et al., 2021, Hong et al., 2022, Aouali et al., 2023] adopt the Bayes regret to analyze the stochastic

structure of their problems, that the bandit tasks are similar because their parameters are sampled i.i.d. from a task distribution.

Many bandit algorithms have frequentist regret bounds that match a lower bound. As an example, in a K -armed bandit with the minimum gap ∆ and horizon n , the gap-dependent O ( K ∆ -1 log n ) regret bound of UCB1 [Auer et al., 2002] matches the gap-dependent Ω( K ∆ -1 log n ) lower bound of Lai and Robbins [1985]. Moreover, the gap-free ˜ O ( √ Kn ) regret bound of UCB1 matches, up to logarithmic factors, the gap-free Ω( √ Kn ) lower bound of Auer et al. [1995]. The extra logarithmic factor in the ˜ O ( √ Kn ) bound can be eliminated by modifying UCB1 [Audibert and Bubeck, 2009]. In contrast, and despite the popularity of the model, matching upper and lower bounds mostly do not exist in the Bayesian setting. Specifically, Lai [1987] proved asymptotic c h log 2 n upper and lower bounds, where c h is a prior-dependent constant. However, all recent Bayes regret bounds are ˜ O ( √ n ) [Russo and Van Roy, 2014, 2016, Lu and Van Roy, 2019, Hong et al., 2020, Kveton et al., 2021]. This leaves open the question of finite-time logarithmic regret bounds in the Bayesian setting.

In this work, we answer this question positively and make the following contributions:

We derive the first finite-time logarithmic Bayes regret upper bounds for a Bayesian upper confidence bound (UCB) algorithm. The bounds are O ( c ∆ log n ) and O ( c h log 2 n ) , where c h and c ∆ are constants depending on the prior distribution h and the gaps of random bandit instances sampled from h , respectively. The latter matches the lower bound of Lai [1987] asymptotically. When compared to prior ˜ O ( √ n ) bounds, we better characterize low-regret regimes, where the random gaps are large.

To show the value of prior as a side information, we also derive a finite-time logarithmic Bayes regret upper bound for a frequentist UCB algorithm. The bound changes only little as the prior becomes more informative, while the regret bound for the Bayesian algorithm eventually goes to zero. The bounds match asymptotically when n →∞ and the prior is overtaken by data.

To show the generality of our approach, we prove a O ( dc ∆ log 2 n ) Bayes regret bound for a Bayesian linear bandit algorithm, where d denotes the number of dimensions and c ∆ is a constant depending on random gaps. This bound also improves with a better prior.

Our analyses are a major departure from all recent Bayesian bandit analyses, starting with Russo and Van Roy [2014]. Roughly speaking, we first bound the regret in a fixed bandit instance, similarly to frequentist analyses, and then integrate out the random gap.

We show the tightness of our bounds empirically and compare them to prior bounds.

This paper is organized as follows. In Section 2, we introduce the setting of Bayesian bandits. In Section 3, we present a Bayesian upper confidence bound algorithm called BayesUCB [Kaufmann et al., 2012]. In Section 4, we derive finite-time logarithmic Bayes regret bounds for BayesUCB , in both multi-armed and linear bandits. These are the first such bounds ever derived. In Section 5, we compare our bounds to prior works and show that one matches an existing lower bound [Lai, 1987] asymptotically. In Section 6, we evaluate the bounds empirically. We conclude in Section 7.

2 Setting

We start with introducing our notation. Random variables are capitalized, except for Greek letters like θ . For any positive integer n , we define [ n ] = { 1 , . . . , n } . The indicator function is 1 {·} . The i -th entry of vector v is v i . If the vector is already indexed, such as v j , we write v j,i . We denote the maximum and minimum eigenvalues of matrix M ∈ R d × d by λ 1 ( M ) and λ d ( M ) , respectively.

Our setting is defined as follows. We have a multi-armed bandit [Lai and Robbins, 1985, Lai, 1987, Auer et al., 2002, Abbasi-Yadkori et al., 2011] with an action set A . Each action a ∈ A is associated with a reward distribution p a ( · ; θ ) , which is parameterized by an unknown model parameter θ shared by all actions. The learner interacts with the bandit instance for n rounds indexed by t ∈ [ n ] . In each round t , it takes an action A t ∈ A and observes its stochastic reward Y t ∼ p A t ( · ; θ ) . The rewards are sampled independently across the rounds. We denote the mean of p a ( · ; θ ) by µ a ( θ ) and call it

the mean reward of action a . The optimal action is A ∗ = arg max a ∈A µ a ( θ ) and its mean reward is µ ∗ ( θ ) = µ A ∗ ( θ ) . For a fixed model parameter θ , the n -round regret of a policy is defined as

where the expectation is taken over both random observations Y t and actions A t . The suboptimality gap of action a is ∆ a = µ ∗ ( θ ) -µ a ( θ ) and the minimum gap is ∆ min = min a ∈A\{ A ∗ } ∆ a .

Two settings are common in stochastic bandits. In the frequentist setting [Lai and Robbins, 1985, Auer et al., 2002, Abbasi-Yadkori et al., 2011], the learner has no additional information about θ and its objective is to minimize the worst-case regret for any bounded θ . We study the Bayesian setting [Gittins, 1979, Lai, 1987, Russo and Van Roy, 2014, Russo et al., 2018], where the model parameter θ is drawn from a prior distribution h that is given to the learner as a side information. The goal of the learner is to minimize the n -round Bayes regret R ( n ) = E [ R ( n ; θ )] , where the expectation is taken over the random model parameter θ ∼ h . Note that A ∗ , ∆ a , and ∆ min are random because they depend on the random instance θ .

3 Algorithm

We study a Bayesian upper confidence bound algorithm called BayesUCB [Kaufmann et al., 2012]. The algorithm was analyzed in the Bayesian setting by Russo and Van Roy [2014]. The key idea in BayesUCB is to take the action with the highest UCB with respect to the posterior distribution of model parameter θ . This differentiates it from frequentist algorithms, such as UCB1 [Auer et al., 2002] and LinUCB [Abbasi-Yadkori et al., 2011], where the UCBs are computed using a frequentist maximum likelihood estimate (MLE) of the model parameter.

Let H t = ( A ℓ , Y ℓ ) ℓ ∈ [ t -1] be the history of taken actions and their observed rewards up to round t . The Bayesian UCB for the mean reward of action a at round t is

where ˆ θ t is the posterior mean estimate of θ at round t and C t,a is a confidence interval width for action a at round t . The posterior distribution of model parameter θ is computed from a prior h and history H t using Bayes' rule. The width is chosen so that | µ a ( ˆ θ t ) -µ a ( θ ) | ≤ C t,a holds with a high probability conditioned on any history H t . Technically speaking, C t,a is a half-width but we call it a width to simplify terminology.

Our algorithm is presented in Algorithm 1. We instantiate it in a Gaussian bandit in Section 3.1, in a Bernoulli bandit in Section 3.2, and in a linear bandit with Gaussian rewards in Section 3.3. These settings are of practical interest because they lead to computationally-efficient implementations that can be analyzed due to closed-form posteriors [Lu and Van Roy, 2019, Kveton et al., 2021, Basu et al., 2021, Wang et al., 2021, Hong et al., 2022]. While we focus on deriving logarithmic Bayes regret bounds for BayesUCB , we believe that similar analyses can be done for Thompson sampling [Thompson, 1933, Chapelle and Li, 2012, Agrawal and Goyal, 2012, 2013, Russo and Van Roy, 2014, Russo et al., 2018]. This extension is non-trivial because a key step in our analysis is that the action with the highest UCB is taken (Section 5.3).

3.1 Gaussian Bandit

In a K -armed Gaussian bandit, the action set is A = [ K ] and the model parameter is θ ∈ R K . Each action a ∈ A has a Gaussian reward distribution, p a ( · ; θ ) = N ( · ; θ a , σ 2 ) , where θ a is its mean and

σ > 0 is a known reward noise. Thus µ a ( θ ) = θ a . The model parameter θ is drawn from a known Gaussian prior h ( · ) = N ( · ; µ 0 , σ 2 0 I K ) , where µ 0 ∈ R K is a vector of prior means and σ 0 > 0 is a prior width.

The posterior distribution of the mean reward of action a at round t is N ( · ; ˆ θ t,a , ˆ σ 2 t,a ) , where

is the posterior variance, N t,a = ∑ t -1 ℓ =1 1 { A ℓ = a } is the number of observations of action a up to round t , and

is the posterior mean. This follows from a classic result, that the posterior distribution of the mean of a Gaussian random variable with a Gaussian prior is a Gaussian [Bishop, 2006]. The Bayesian UCB of action a at round t is U t,a = ˆ θ t,a + C t,a , where C t,a = √ 2ˆ σ 2 t,a log(1 /δ ) is the confidence interval width and δ ∈ (0 , 1) is a failure probability of the confidence interval.

3.2 Bernoulli Bandit

In a K -armed Bernoulli bandit, the action set is A = [ K ] and the model parameter is θ ∈ R K . Each action a ∈ A has a Bernoulli reward distribution, p a ( · ; θ ) = Ber( · ; θ a ) , where θ a is its mean. Hence µ a ( θ ) = θ a . Each parameter θ a is drawn from a known prior Beta( · ; α a , β a ) , where α a > 0 and β a > 0 are positive and negative prior pseudo-counts, respectively.

The posterior distribution of the mean reward of action a at round t is Beta( · ; α t,a , β t,a ) , where

This follows from a classic result, that the posterior distribution of the mean of a Bernoulli random variable with a beta prior is a beta distribution [Bishop, 2006]. The corresponding Bayesian UCB is U t,a = ˆ θ t,a + C t,a , where

denote the posterior mean and confidence interval width, respectively, of action a at round t ; and δ ∈ (0 , 1) is a failure probability of the confidence interval. The confidence interval is derived using the fact that Beta( · ; α t,a , β t,a ) is a sub-Gaussian distribution with variance proxy 1 4( α a + β a + N t,a +1) [Marchal and Arbel, 2017].

3.3 Linear Bandit with Gaussian Rewards

We also study linear bandits [Dani et al., 2008, Abbasi-Yadkori et al., 2011] with a finite number of actions A ⊆ R d in d dimensions. The model parameter is θ ∈ R d . All actions a ∈ A have Gaussian reward distributions, p a ( · ; θ ) = N ( · ; a ⊤ θ, σ 2 ) , where σ > 0 is a known reward noise. Therefore, the mean reward of action a is µ a ( θ ) = a ⊤ θ . The parameter θ is drawn from a known multivariate Gaussian prior h ( · ) = N ( · ; θ 0 , Σ 0 ) , where θ 0 ∈ R d is its mean and Σ 0 ∈ R d × d is its covariance, represented by a positive semi-definite (PSD) matrix.

The posterior distribution of θ at round t is N ( · ; ˆ θ t , ˆ Σ t ) , where

Here ˆ θ t and ˆ Σ t are the posterior mean and covariance of θ , respectively, and G t is the outer product of the feature vectors of the taken actions up to round t . These formulas follow from a classic result,

that the posterior distribution of a linear model parameter with a Gaussian prior and observations is a Gaussian [Bishop, 2006]. The Bayesian UCB of action a at round t is U t,a = a ⊤ ˆ θ t + C t,a , where C t,a = √ 2 log(1 /δ ) ∥ a ∥ ˆ Σ t is the confidence interval width, δ ∈ (0 , 1) is a failure probability of the confidence interval, and ∥ a ∥ M = √ a ⊤ Ma .

4 Logarithmic Bayes Regret Upper Bounds

In this section, we present finite-time logarithmic Bayes regret bounds for BayesUCB . We derive them for both K -armed and linear bandits. One bound matches an existing lower bound of Lai [1987] asymptotically and all improve upon prior ˜ O ( √ n ) bounds. We discuss this in detail in Section 5.

4.1 BayesUCB in Gaussian Bandit

Our first regret bound is for BayesUCB in a K -armed Gaussian bandit. It depends on random gaps. To control the gaps, we clip them as ∆ ε a = max { ∆ a , ε } . The bound is stated below.

Theorem 1. For any ε > 0 and δ ∈ (0 , 1) , the n -round Bayes regret of BayesUCB in a K -armed Gaussian bandit is bounded as

̸

where C = εn +2( √ 2 log(1 /δ ) + 2 K ) σ 0 Knδ is a low-order term.

The proof is in Appendix A.1. For ε = 1 /n and δ = 1 /n , the bound is O ( c ∆ log n ) , where c ∆ is a constant depending on the gaps of random bandit instances. The dependence on σ 0 in the low-order term C can be reduced to min { σ 0 , σ } by a more elaborate analysis, where the regret of taking each action for the first time is bounded separately. This also applies to Corollary 2.

Now we derive an upper bound on Theorem 1 that eliminates the dependence on random gaps. To state it, we need to introduce additional notation. For any action a , we denote all action parameters except for a by θ -a = ( θ 1 , . . . , θ a -1 , θ a +1 , . . . , θ K ) and the corresponding optimal action in θ -a by θ ∗ a = max j ∈A\{ a } θ j . We denote by h a the prior density of θ a and by h -a the prior density of θ -a . Since the prior is factored (Section 3.1), note that h ( θ ) = h a ( θ a ) h -a ( θ -a ) for any θ and action a . To keep the result clean, we state it for a 'sufficiently' large prior variance. A complete statement for all prior variances is given in Appendix B. We note that the setting of small prior variances favors Bayesian algorithms since their regret decreases with a more informative prior. In fact, we show in Appendix B that the regret of BayesUCB is O (1) for a sufficiently small σ 0 .

Corollary 2. Let σ 2 0 ≥ 1 8 log(1 /δ ) n 2 log log n . Then there exist functions ξ a : R → [ 1 n , 1 √ log n ] such that the n -round Bayes regret of BayesUCB in a K -armed Gaussian bandit is bounded as

where C = 8 σ 2 K log(1 /δ ) √ log n +2( √ 2 log(1 /δ ) + 2 K ) σ 0 Knδ +1 is a low-order term.

The proof is in Appendix A.2. For δ = 1 /n , the bound is O ( c h log 2 n ) , where c h depends on prior h but not on the gaps of random bandit instances. This bound is motivated by Lai [1987]. The terms ξ a arise due to the intermediate value theorem for function h a . Similar terms appear in Lai [1987] but vanish in their final asymptotic claims. The rate 1 / √ log n in the definition of ξ a cannot be reduced to 1 / poly log n without increasing dependence on n in other parts of the bound.

The complexity term ∑ a ∈A ∫ θ -a h a ( θ ∗ a -ξ a ( θ ∗ a )) h -a ( θ -a ) d θ -a in Corollary 2 is the same as in Lai [1987] and can be interpreted as follows. Consider the asymptotic regime of n →∞ . Then, since the range of ξ a is [ 1 n , 1 √ log n ] , the term simplifies to ∑ a ∈A ∫ θ -a h a ( θ ∗ a ) h -a ( θ -a ) d θ -a and can be viewed as the distance between prior means. In a Gaussian bandit with K = 2 actions, it has a closed form of 1 √ πσ 2 0 exp [ -( µ 0 , 1 -µ 0 , 2 ) 2 4 σ 2 0 ] . A general upper bound for K > 2 actions is given below.

Lemma 3. In a K -armed Gaussian bandit with prior h ( · ) = N ( · ; µ 0 , σ 2 0 I K ) , we have

̸

The bound is proved in Appendix A.3 and has several interesting properties that capture low-regret regimes. First, as the prior becomes more informative and concentrated, σ 0 → 0 , the bound goes to zero. Second, when the gaps of bandit instances sampled from the prior are large, low regret is also expected. This can happen when the prior means become more separated, | µ 0 ,a -µ 0 ,a ′ |→∞ , or the prior becomes wider, σ 0 →∞ . Our bound goes to zero in both of these cases. This also implies that Bayes regret bounds are not necessarily monotone in prior parameters, such as σ 0 .

4.2 UCB1 in Gaussian Bandit

Using a similar approach, we prove a Bayes regret bound for UCB1 [Auer et al., 2002]. We view it as BayesUCB (Section 3.1) where σ 0 = ∞ and each action a ∈ A is initially taken once at round t = a . This generalizes classic UCB1 to σ 2 -sub-Gaussian noise. An asymptotic Bayes regret bound for UCB1 was proved by Lai [1987] (claim (i) in their Theorem 3). We derive a finite-time prior-dependent Bayes regret bound below.

Theorem 4. There exist functions ξ a : R → [ 1 n , 1 √ log n ] such that the n -round Bayes regret of UCB1 in a K -armed Gaussian bandit is bounded as

where C = 8 σ 2 K log(1 /δ ) √ log n +2( √ 2 log(1 /δ ) + 2 K ) σKnδ + ∑ a ∈A E [∆ a ] + 1 .

The proof is in Appendix A.4. For δ = 1 /n , the bound is O ( c h log 2 n ) and similar to Corollary 2. The main difference is in the additional factor σ 2 2 σ 2 0 log n in Corollary 2, which decreases the bound. This means that the regret of BayesUCB improves as σ 0 decreases while that of UCB1 may not change much. In fact, the regret bound of BayesUCB is O (1) as σ 0 → 0 (Appendix B) while that of UCB1 remains logarithmic. This is expected because BayesUCB has more information about the random instance θ as σ 0 decreases, while the frequentist algorithm is oblivious to the prior.

4.3 BayesUCB in Bernoulli Bandit

Theorem 1 and Corollary 2 can be straightforwardly extended to Bernoulli bandits because

holds for any action a and history H t (Section 3.2). We state the extension below and prove it in Appendix A.5.

Theorem 5. For any ε > 0 and δ ∈ (0 , 1) , the n -round Bayes regret of BayesUCB in a K -armed Bernoulli bandit is bounded as

̸

where C = εn +2 Knδ is a low-order term.

Moreover, let λ = min a ∈A α a + β a +1 and λ ≤ 2 log(1 /δ ) n 2 log log n . Then

where C = 2 K log(1 /δ ) √ log n +2 Knδ +1 is a low-order term.

4.4 BayesUCB in Linear Bandit

Now we present a gap-dependent Bayes regret bound for BayesUCB in a linear bandit with a finite number of actions. The bound depends on a random minimum gap. To control the gap, we clip it as ∆ ε min = max { ∆ min , ε } .

Theorem 6. Suppose that ∥ θ ∥ 2 ≤ L ∗ holds with probability at least 1 -δ ∗ . Let ∥ a ∥ 2 ≤ L for any action a ∈ A . Then for any ε > 0 and δ ∈ (0 , 1) , the n -round Bayes regret of linear BayesUCB is bounded as

with probability at least 1 -δ ∗ , where σ 0 , max = √ λ 1 (Σ 0 ) L .

The proof is in Appendix A.6. For ε = 1 /n and δ = 1 /n , the bound is O ( dc ∆ log 2 n ) , where c ∆ is a constant depending on the gaps of random bandit instances. The bound is remarkably similar to the frequentist O ( d ∆ -1 min log 2 n ) bound in Theorem 5 of Abbasi-Yadkori et al. [2011], where ∆ min is the minimum gap. There are two differences. First, we integrate ∆ -1 min over the prior. Second, our bound decreases as the prior becomes more informative, σ 0 , max → 0 .

In a Gaussian bandit, the bound becomes O ( K E [1 / ∆ ε min ] log 2 n ) . Therefore, it is comparable to Theorem 1 up to an additional logarithmic factor in n . This is due to a more general proof technique, which allows for dependencies between the mean rewards of actions. We also note that Theorem 6 does not assume that the prior is factored, unlike Theorem 1.

5 Comparison to Prior Works

This section is organized as follows. In Section 5.1, we show that the bound in Theorem 5 matches an existing lower bound of Lai [1987] asymptotically. In Section 5.2, we compare our logarithmic bounds to prior ˜ O ( √ n ) bounds. Finally, in Section 5.3, we outline the key steps in our analyses and how they differ from prior works.

5.1 Matching Lower Bound

In frequentist bandit analyses, it is standard to compare asymptotic lower bounds to finite-time upper bounds because finite-time logarithmic lower bounds do not exist [Lattimore and Szepesvari, 2019]. We follow the same approach when arguing that our finite-time upper bounds are order optimal.

The results in Lai [1987] are for single-parameter exponential-family reward distributions, which excludes Gaussian rewards. Therefore, we argue about the tightness of Theorem 5 only. Specifically, we take the second bound in Theorem 5, set δ = 1 /n , and let n →∞ . In this case, λ 2 log n → 0 and ξ a ( · ) → 0 , and the bound matches up to constant factors the lower bound in Lai [1987] (claim (ii) in their Theorem 3), which is

Lai [1987] also matched this lower bound with an asymptotic upper bound for a frequentist policy.

Our finite-time upper bounds also reveal an interesting difference from the asymptotic lower bound in (2), which may deserve more future attention. More specifically, the regret bound of BayesUCB (Corollary 2) improves with prior information while that of UCB1 (Theorem 4) does not. We observe these improvements empirically as well (Section 6 and Appendix D). However, both bounds are the same asymptotically. This is because the benefit of knowing the prior vanishes in asymptotic analyses, since σ 2 2 σ 2 0 log n → 0 in Corollary 2 as n →∞ . This motivates the need for finite-time logarithmic Bayes regret lower bounds, which do not exist.

5.2 Prior Bayes Regret Upper Bounds

Theorem 1 and Corollary 2 are major improvements upon existing ˜ O ( √ n ) bounds. For instance, take a prior-dependent bound in Lemma 4 of Kveton et al. [2021], which holds for both BayesUCB and Thompson sampling due to a well-known equivalence of their analyses [Russo and Van Roy, 2014, Hong et al., 2020]. For δ = 1 /n , their leading term becomes

Similarly to Theorem 1 and Corollary 2, (3) decreases as the prior concentrates and becomes more informative, σ 0 → 0 . However, the bound is ˜ O ( √ n ) . Moreover, it does not depend on prior means µ 0 or the gaps of random bandit instances. Therefore, it cannot capture low-regret regimes due to large random gaps ∆ ε a in Theorem 1 or a small complexity term in Corollary 2. We demonstrate it empirically in Section 6.

When the random gaps ∆ ε a in Theorem 1 are small or the complexity term in Corollary 2 is large, our bounds can be worse than ˜ O ( √ n ) bounds. This is analogous to the relation of the gap-dependent and gap-free frequentist bounds [Lattimore and Szepesvari, 2019]. Specifically, a gap-dependent bound of UCB1 in a K -armed bandit with 1 -sub-Gaussian rewards (Theorem 7.1) is O ( K ∆ -1 min log n ) , where ∆ min is the minimum gap. A corresponding gap-free bound (Theorem 7.2) is O ( √ Kn log n ) . The latter is smaller when the gap is small, ∆ min = o ( √ ( K log n ) /n ) . To get the best bound, the minimum of the two should be taken, and the same is true in our Bayesian setting.

No prior-dependent Bayes regret lower bound exists in linear bandits. Thus we treat E [1 / ∆ ε min ] in Theorem 6 as the complexity term and do not further bound it as in Corollary 2. To compare our bound fairly to existing ˜ O ( √ n ) bounds, we derive an ˜ O ( √ n ) bound in Appendix C, by a relatively minor change in the proof of Theorem 6. A similar bound can be obtained by adapting the proofs of Lu and Van Roy [2019] and Hong et al. [2022] to a linear bandit with a finite number of actions. The leading term of the bound is

Similarly to Theorem 6, (4) decreases as the prior becomes more informative, σ 0 , max → 0 . However, the bound is ˜ O ( √ n ) and does not depend on the gaps of random bandit instances. Hence it cannot capture low-regret regimes due to a large random minimum gap ∆ min in Theorem 6. We validate it empirically in Section 6.

5.3 Technical Novelty

All modern Bayesian analyses follow Russo and Van Roy [2014], who derived the first finite-time ˜ O ( √ n ) Bayes regret bounds for BayesUCB and Thompson sampling. The key idea in their analyses is that conditioned on history, the optimal and taken actions are identically distributed, and that the upper confidence bounds are deterministic functions of the history. This is where the randomness of instances in Bayesian bandits is used. Using this, the regret at round t is bounded by the confidence interval width of the taken action, and the usual ˜ O ( √ n ) bounds can be obtained by summing up the confidence interval widths over n rounds.

The main difference in our work is that we first bound the regret in a fixed bandit instance, similarly to frequentist analyses. The bound involves ∆ -1 a and is derived using biased Bayesian confidence intervals. The rest of our analysis is Bayesian in two parts: we prove that the confidence intervals fail with a low probability and bound random ∆ -1 a , following a similar technique to Lai [1987]. The resulting logarithmic Bayes regret bounds cannot be derived using the techniques of Russo and Van Roy [2014], as these become loose when the confidence interval widths are introduced.

Asymptotic logarithmic Bayes regret bounds were derived in Lai [1987]. From this analysis, we use only the technique for bounding ∆ -1 a when proving Corollary 2 and Theorem 5. The central part of our proof is a finite-time per-instance bound on the number of times that a suboptimal action is taken. This quantity is bounded based on the assumption that the action with the highest UCB is taken. A

0.0

0.2

0.4

0.6

0.8

1.0

Prior width

¾

0

10 1

10 2

10 3

Regret

(a) Decreasing prior width

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Prior gap

¢0

10 1

10 2

10 3

(b) Increasing prior gap

BayesUCB

BayesUCB log bound

UCB1 log bound

O

(

p

n

)

bound

comparable argument in Theorem 2 of Lai [1987] is asymptotic and on average over random bandit instances.

6 Experiments

We experiment with UCB algorithms in two environments: Gaussian bandits (Section 3.1) and linear bandits with Gaussian rewards (Section 3.3). In both experiments, the horizon is n = 1000 rounds. All results are averaged over 10 000 random runs. Shaded regions in the plots are standard errors of the estimates. They are generally small because the number of runs is high.

6.1 Gaussian Bandit

The first problem is a K -armed bandit with K = 10 actions (Section 3.1). The prior width is σ 0 = 1 . The prior mean is µ 0 , where µ 0 , 1 = ∆ 0 and µ 0 ,a = 0 for a > 1 . We set ∆ 0 = 1 and call it the prior gap . We vary σ 0 and ∆ 0 in our experiments, and observe how the regret and its upper bounds change as the problem hardness varies (Sections 4.1 and 5.2).

We plot five trends: (a) Bayes regret of BayesUCB . (b) Bayes regret of UCB1 (Section 4.2), which is a comparable frequentist algorithm to BayesUCB . (c) A leading term of the BayesUCB regret bound in Theorem 1, where ε = 1 /n and δ = 1 /n . (d) A leading term of the UCB1 regret bound: This is the same as (c) with σ 0 = ∞ . (e) An existing ˜ O ( √ n ) regret bound in (3).

Our results are reported in Figure 1. We observe three major trends. First, the regret of BayesUCB decreases as the problem becomes easier, either σ 0 → 0 or ∆ 0 →∞ . It is also lower than that of UCB1 , which does not leverage the prior. Second, the regret bound of BayesUCB is tighter than that of UCB1 , due to capturing the benefit of the prior. Finally, the logarithmic regret bounds are tighter than the ˜ O ( √ n ) bound. In addition, the ˜ O ( √ n ) bound depends on the prior only through σ 0 and thus remains constant as the prior gap ∆ 0 changes.

In Appendix D, we compare BayesUCB to UCB1 more comprehensively for various K , σ , ∆ 0 , and σ 0 . In all experiments, BayesUCB has a lower regret than UCB1 . This also happens when the noise is not Gaussian, which a testament to the robustness of Bayesian methods to model misspecification.

6.2 Linear Bandit with Gaussian Rewards

The second problem is a linear bandit with K = 30 actions in d = 10 dimensions (Section 3.3). The prior covariance is Σ 0 = σ 2 0 I d . The prior mean is θ 0 , where θ 0 , 1 = ∆ 0 and θ 0 ,i = -1 for i > 1 . As in Section 6.1, we set ∆ 0 = 1 and call it the prior gap . The action set A is generated as follows. The first d actions are the canonical basis in R d . The remaining K -d actions are sampled uniformly at random from the positive orthant and scaled to unit length. This ensures that the first action has the highest mean reward, of ∆ 0 , under the prior mean θ 0 . We vary σ 0 and ∆ 0 , and observe how the regret and its upper bounds change as the problem hardness varies (Sections 4.4 and 5.2). We plot three trends: (a) Bayes regret of BayesUCB . (b) A leading term of the BayesUCB regret bound in Theorem 6, where σ 0 , max = σ 0 , ε = 1 /n , and δ = 1 /n . (c) An existing ˜ O ( √ n ) regret bound in (4).

0.0

0.2

0.4

0.6

0.8

1.0

Prior width

¾

0

10 1

10 2

10 3

10 4

Regret

(a) Decreasing prior width

1

2

3

4

5

6

7

8

9

10

Prior gap

¢0

10 1

10 2

10 3

10 4

(b) Increasing prior gap

BayesUCB

BayesUCB log bound

O

(

p

n

)

bound

Our results are reported in Figure 2 and we observe three major trends. First, the regret of BayesUCB decreases as the problem becomes easier, either σ 0 → 0 or ∆ 0 →∞ . Second, the regret bound of BayesUCB decreases as the problem becomes easier. Finally, our logarithmic regret bound can also be tighter than the ˜ O ( √ n ) bound. In particular, the ˜ O ( √ n ) bound depends on the prior only through σ 0 , and thus remains constant as the prior gap ∆ 0 changes. We discuss when our bounds could be looser than ˜ O ( √ n ) bounds in Section 5.2.

7 Conclusions

Finite-time logarithmic frequentist regret bounds are the standard way of analyzing K -armed bandits [Auer et al., 2002, Garivier and Cappe, 2011, Agrawal and Goyal, 2012]. In our work, we prove the first comparable finite-time bounds, logarithmic in n , in the Bayesian setting. This is a major step in theory and a significant improvement upon prior ˜ O ( √ n ) Bayes regret bounds that have become standard in the literature. Comparing to frequentist regret bounds, our bounds capture the value of prior information given to the learner. Our proof technique is general and we also apply it to linear bandits.

This work can be extended in many directions. First, our analyses only need closed-form posteriors, which are available for other reward distributions than Gaussian and Bernoulli. Second, our linear bandit analysis (Section 4.4) seems preliminary when compared to our multi-armed bandit analyses. As an example, the complexity term E [1 / ∆ ε min ] in Theorem 6 could be bounded as in Corollary 2 and Theorem 5. We do not do this because the main reason for deriving the O ( c h log 2 n ) bound in Theorem 5, an upper bound on the corresponding O ( c ∆ log n ) bound, is that it matches the lower bound in (2). No such instance-dependent lower bound exists in Bayesian linear bandits. Third, we believe that our approach can be extended to information-theory bounds [Russo and Van Roy, 2016] and discuss it in Appendix E. Fourth, although we only analyze BayesUCB , we believe that similar guarantees can be obtained for Thompson sampling. Finally, we would like to extend our results to reinforcement learning, for instance by building on the work of Lu and Van Roy [2019].

There have been recent attempts in theory [Wagenmaker and Foster, 2023] to design general adaptive algorithms with finite-time instance-dependent bounds based on optimal allocations. The promise of these methods is a higher statistical efficiency than exploring by optimism, which we adopt in this work. One of their shortcomings is that they are not guaranteed to be computationally efficient, as discussed in Section 2.2 of Wagenmaker and Foster [2023]. This work is also frequentist.

References

Yasin Abbasi-Yadkori, David Pal, and Csaba Szepesvari. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems 24 , pages 2312-2320, 2011.

Marc Abeille and Alessandro Lazaric. Linear Thompson sampling revisited. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , 2017.

Shipra Agrawal and Navin Goyal. Analysis of Thompson sampling for the multi-armed bandit problem. In Proceedings of the 25th Annual Conference on Learning Theory , pages 39.1-39.26, 2012.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In Proceedings of the 30th International Conference on Machine Learning , pages 127-135, 2013.

Imad Aouali, Branislav Kveton, and Sumeet Katariya. Mixed-effect Thompson sampling. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , 2023.

Jean-Yves Audibert and Sebastien Bubeck. Minimax policies for adversarial and stochastic bandits. In Proceedings of the 22nd Annual Conference on Learning Theory , 2009.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Proceedings of the 36th Annual Symposium on Foundations of Computer Science , pages 322-331, 1995.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47:235-256, 2002.

Hamsa Bastani, David Simchi-Levi, and Ruihao Zhu. Meta dynamic pricing: Transfer learning across experiments. CoRR , abs/1902.10918, 2019. URL https://arxiv.org/abs/1902.10918 .

Soumya Basu, Branislav Kveton, Manzil Zaheer, and Csaba Szepesvari. No regrets for learning the prior in bandits. In Advances in Neural Information Processing Systems 34 , 2021.

Christopher Bishop. Pattern Recognition and Machine Learning . Springer, New York, NY, 2006.

Olivier Chapelle and Lihong Li. An empirical evaluation of Thompson sampling. In Advances in Neural Information Processing Systems 24 , pages 2249-2257, 2012.

Varsha Dani, Thomas Hayes, and Sham Kakade. Stochastic linear optimization under bandit feedback. In Proceedings of the 21st Annual Conference on Learning Theory , pages 355-366, 2008.

Aurelien Garivier and Olivier Cappe. The KL-UCB algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th Annual Conference on Learning Theory , pages 359-376, 2011.

John Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society. Series B (Methodological) , 41:148-177, 1979.

Joey Hong, Branislav Kveton, Manzil Zaheer, Yinlam Chow, Amr Ahmed, and Craig Boutilier. Latent bandits revisited. In Advances in Neural Information Processing Systems 33 , 2020.

Joey Hong, Branislav Kveton, Manzil Zaheer, and Mohammad Ghavamzadeh. Hierarchical Bayesian bandits. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics , 2022.

Emilie Kaufmann, Olivier Cappe, and Aurelien Garivier. On Bayesian upper confidence bounds for bandit problems. In Proceedings of the 15th International Conference on Artificial Intelligence and Statistics , pages 592-600, 2012.

Jaya Kawale, Hung Bui, Branislav Kveton, Long Tran-Thanh, and Sanjay Chawla. Efficient Thompson sampling for online matrix-factorization recommendation. In Advances in Neural Information Processing Systems 28 , pages 1297-1305, 2015.

Branislav Kveton, Mikhail Konobeev, Manzil Zaheer, Chih-Wei Hsu, Martin Mladenov, Craig Boutilier, and Csaba Szepesvari. Meta-Thompson sampling. In Proceedings of the 38th International Conference on Machine Learning , 2021.

Tze Leung Lai. Adaptive treatment allocation and the multi-armed bandit problem. The Annals of Statistics , 15(3):1091-1114, 1987.

Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics , 6(1):4-22, 1985.

Tor Lattimore and Csaba Szepesvari. Bandit Algorithms . Cambridge University Press, 2019.

Lihong Li, Wei Chu, John Langford, and Robert Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web , 2010.

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research , 18(185):1-52, 2018.

Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. Collaborative filtering bandits. In Proceedings of the 39th Annual International ACM SIGIR Conference , 2016.

Xiuyuan Lu and Benjamin Van Roy. Information-theoretic confidence bounds for reinforcement learning. In Advances in Neural Information Processing Systems 32 , 2019.

Olivier Marchal and Julyan Arbel. On the sub-Gaussianity of the beta and Dirichlet distributions. Electronic Communications in Probability , 22:1-14, 2017.

Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research , 39(4):1221-1243, 2014.

Daniel Russo and Benjamin Van Roy. An information-theoretic analysis of Thompson sampling. Journal of Machine Learning Research , 17(68):1-30, 2016.

Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on Thompson sampling. Foundations and Trends in Machine Learning , 11(1):1-96, 2018.

Max Simchowitz, Christopher Tosh, Akshay Krishnamurthy, Daniel Hsu, Thodoris Lykouris, Miro Dudik, and Robert Schapire. Bayesian decision-making under misspecified priors with applications to meta-learning. In Advances in Neural Information Processing Systems 34 , 2021.

William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

John Tsitsiklis. A short proof of the gittins index theorem. Neural Computation , 4(1):194-199, 1994.

Andrew Wagenmaker and Dylan Foster. Instance-optimality in interactive decision making: Toward a non-asymptotic theory. In Proceedings of the 36th Annual Conference on Learning Theory , 2023.

Zhi Wang, Chicheng Zhang, Manish Kumar Singh, Laurel Riek, and Kamalika Chaudhuri. Multitask bandit learning through heterogeneous feedback aggregation. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics , 2021.

Xiaoxue Zhao, Weinan Zhang, and Jun Wang. Interactive collaborative filtering. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management , pages 1411-1420, 2013.

A Proofs

We present a general approach for deriving finite-time logarithmic Bayes regret bounds. We start with K -armed bandits and then extend it to linear bandits.

A.1 Proof of Theorem 1

Let E t = { ∀ a ∈ A : | θ a -ˆ θ t,a | ≤ C t,a } be the event that all confidence intervals at round t hold. Fix ε > 0 . We start with decomposing the n -round regret as

We bound the first term using the design of BayesUCB and its closed-form posteriors.

Case 1: Event E t occurs and the gap is large, ∆ A t ≥ ε . Since E t occurs and the action with the highest UCB is taken, the regret at round t can be bounded as

In the second inequality, we use that θ A ∗ ≤ U t,A ∗ on event E t . This implies that on event E t , action a can be taken only if

which can be rearranged as

Therefore, the number of times that action a is taken in n rounds while all confidence intervals hold, N a = ∑ n t =1 1 { A t = a, E t } , is bounded as

Now we apply this inequality to bound the first term in (5) as

̸

n

Case 2: The gap is small, ∆ A t < ε . Then naively ∑ t =1 E [∆ A t 1 { ∆ A t < ε } ] < εn . Case 3: Event E t does not occur. The last term in (5) can be bounded as

To bound the resulting terms, we use that θ a -ˆ θ t,a | H t ∼ N (0 , ˆ σ 2 t,a ) .

Lemma 7. For any action a ∈ A , round t ∈ [ n ] , and history H t ,

̸

Proof. Let E t,a = { | θ a -ˆ θ t,a | ≤ C t,a } . We start with decomposing ¯ E t into individual ¯ E t,a as E [ ( θ a -ˆ θ t,a ) 1 { ¯ E t } ∣ ∣ ∣ H t ] ≤ E [ | θ a -ˆ θ t,a | 1 { ¯ E t,a } ∣ ∣ ∣ H t ] + ∑ a ′ = a E [ | θ a -ˆ θ t,a | 1 { ¯ E t,a ′ } ∣ ∣ ∣ H t ] .

To bound the first term, we use that θ a -ˆ θ t,a | H t ∼ N (0 , ˆ σ 2 t,a ) . Thus

To bound the second term, we use the independence of the distributions for a and a ′ ,

The probability is at most 2 δ and the expectation can be bounded as

This completes the proof.

The first two terms in (8) can be bounded using a union bound over a ∈ A and Lemma 7. For the last term, we use that C t,a ≤ √ 2 σ 2 0 log(1 /δ ) and a union bound in P ( ¯ E t ∣ ∣ H t ) to get

Finally, we sum up the upper bounds on (8) over all rounds t ∈ [ n ] .

A.2 Proof of Corollary 2

The key idea in the proof is to integrate out the random gap in (7). Fix action a ∈ A and thresholds ε 2 > ε > 0 . We consider two cases.

Case 1: Diminishing gaps ε < ∆ a ≤ ε 2 . Let

and N a be defined as in Appendix A.1. Then

where the inequality is by the definition of ξ a . Now the inner integral is independent of h a and thus can be easily bounded. Specifically, the upper bound in (6) and simple integration yield

For ε = 1 /n and ε 2 = 1 / √ log n , we get

The last inequality holds for σ 2 0 ≥ 1 8 log(1 /δ ) n 2 log log n .

Case 2: Large gaps ∆ a > ε 2 . Here we use (6) together with ε 2 = 1 / √ log n to get

Finally, we chain all inequalities.

A.3 Proof of Lemma 3

We have that

̸

where the last step is by completing the square and integrating out θ a ′ .

A.4 Proof of Theorem 4

The regret bound of UCB1 is proved similarly to Theorem 1 and Corollary 2. This is because UCB1 can be viewed as BayesUCB where σ 0 = ∞ and each action a ∈ A is initially taken once at round t = a . Since σ 0 = ∞ , the confidence interval becomes

The proof differs in two steps. First, the regret in the first K rounds is bounded by ∑ a ∈A E [∆ a ] . Second, the concentration argument (Case 3 in Appendix A.1) changes because the bandit instance θ is fixed and the estimated model parameter ˆ θ t is random. We detail it below.

Case 3: Event E t does not occur. The last term in (5) can be bounded as

To bound the resulting terms, we use that θ a -ˆ θ t,a | θ ∼ N (0 , σ 2 /N t,a ) .

̸

̸

̸

̸

Lemma 8. For any action a ∈ A , round t > K , and N t,a ≥ 1 ,

̸

To bound the first term, we use that θ a -ˆ θ t,a | θ ∼ N (0 , σ 2 /N t,a ) . Thus

To bound the second term, we use the independence of the distributions for a and a ′ ,

The probability is at most 2 δ and the expectation can be bounded as

This completes the proof.

The first two terms in (11) can be bounded using a union bound over a ∈ A and Lemma 8. For the last term, we use that C t,a ≤ √ 2 σ 2 log(1 /δ ) and a union bound in P ( ¯ E t ∣ ∣ θ ) to get

Finally, we sum up the upper bounds on (11) over all rounds t ∈ [ n ] .

A.5 Proof of Theorem 5

Let E t = { ∀ a ∈ A : | θ a -ˆ θ t,a | ≤ C t,a } be the event that all confidence intervals at round t hold. Fix ε > 0 . We decompose the n -round regret as in (5) and then bound each resulting term next.

Case 1: Event E t occurs and the gap is large, ∆ A t ≥ ε . As in Appendix A.1,

In the second inequality, we use that θ A ∗ ≤ U t,A ∗ on event E t . This implies that on event E t , action a can be taken only if

Now we apply this inequality to bound the first term in (5) as

̸

Case 2: The gap is small, ∆ A t < ε . Then naively ∑ n t =1 E [∆ A t 1 { ∆ A t < ε } ] < εn .

Case 3: Event E t does not occur. Since θ a ∈ [0 , 1] , the last term in (5) can be bounded as

This completes the first part of the proof.

The second claim is proved as in Appendix A.2 and we only comment on what differs. For ε = 1 /n and ε 2 = 1 / √ log n , (9) becomes

The last inequality holds for λ ≤ 2 log(1 /δ ) n 2 log log n . Moreover, (10) becomes

This completes the second part of the proof.

A.6 Proof of Theorem 6

Let

be an event that high-probability confidence intervals for mean rewards at round t hold. Our proof has three parts.

Case 1: Event E t occurs and the gap is large, ∆ A t ≥ ε . Then

The first inequality follows from definitions of ∆ A t and ∆ ε min ; and that the gap is large, ∆ A t ≥ ε . The second inequality holds because A ⊤ ∗ θ -A ⊤ t θ ≥ 0 by definition and U t,A t -U t,A ∗ ≥ 0 by the design of BayesUCB . The third inequality holds because A ⊤ ∗ θ -U t,A ∗ ≤ 0 on event E t . Specifically, for any action a ∈ A on event E t ,

The last inequality follows from the definition of event E t . Specifically, for any action a ∈ A on event E t ,

Case 2: The gap is small, ∆ A t ≤ ε . Then naively ∆ A t ≤ ε .

Case 3: Event E t does not occur. Then ∆ A t 1 { ¯ E t } ≤ 2 ∥ A t ∥ 2 ∥ θ ∥ 2 1 { ¯ E t } ≤ 2 LL ∗ 1 { ¯ E t } , where 2 LL ∗ is a trivial upper bound on ∆ A t . We bound the event in expectation as follows.

Lemma 9. For any round t ∈ [ n ] and history H t , we have that P ( ¯ E t ∣ ∣ H t ) ≤ 2 Kδ .

Proof. First, note that for any history H t ,

By definition, θ -ˆ θ t | H t ∼ N ( 0 d , ˆ Σ t ) , and therefore a ⊤ ( θ -ˆ θ t ) / ∥ a ∥ ˆ Σ t | H t ∼ N (0 , 1) for any action a ∈ A . It immediately follows that

This completes the proof.

Finally, we chain all inequalities, add them over all rounds, and get

The sum can bounded using a worst-case argument below, which yields our claim.

Lemma 10. The sum of posterior variances is bounded as

Proof. We start with an upper bound on the posterior variance of the mean reward estimate of any action. In any round t ∈ [ n ] , by Weyl's inequalities, we have

Thus, when ∥ a ∥ 2 ≤ L for any action a ∈ A , we have max a ∈A ∥ a ∥ ˆ Σ t ≤ √ λ 1 (Σ 0 ) L = σ 0 , max .

Now we bound the sum of posterior variances ∑ n t =1 ∥ A t ∥ 2 ˆ Σ t . Fix round t and note that

for

This upper bound is derived as follows. For any x ∈ [0 , u ] ,

Then we set x = σ -2 A ⊤ t ˆ Σ t A t and use the definition of σ 0 , max .

The next step is bounding the logarithmic term in (13), which can be rewritten as

Because of that, when we sum over all rounds, we get telescoping and the total contribution of all terms is at most

This completes the proof.

B Complete Statement of Corollary 2

Theorem 11. Let σ 2 0 ≥ 1 8 log(1 /δ ) n 2 log log n . Then there exist functions ξ a : R → [ 1 n , 1 √ log n ] such that the n -round Bayes regret of BayesUCB in a K -armed Gaussian bandit is bounded as

where C = 8 σ 2 K log(1 /δ ) √ log n +(2 √ 2 log(1 /δ ) + 1) σ 0 Knδ +1 is a low-order term.

Moreover, when σ 2 0 < 1 8 log(1 /δ ) n 2 log log n , the regret is bounded as

Proof. The first claim is proved in Appendix A.2. The second claim can be proved as follows. Take Theorem 1, set ε = 0 , and consider the three cases in Appendix A.1.

Case 1: Event E t occurs and the gap is large, ∆ A t ≥ ε . On event E t , action a can be taken only if

Therefore, the corresponding n -round regret is bounded by 1 .

Case 2: The gap is small, ∆ A t < ε . This case cannot happen because ε = 0 .

Case 3: Event E t does not occur. The n -round regret is bounded by

This completes the proof.

C Gap-Free Regret Bound of BayesUCB in Linear Bandit

Let E t be the event in (12). Our proof has three parts.

Case 1: Event E t occurs and the gap is large, ∆ A t ≥ ε . Then

The first inequality holds because U t,A t -U t,A ∗ ≥ 0 by the design of BayesUCB . The second one uses that A ⊤ ∗ θ -U t,A ∗ ≤ 0 . Specifically, for any action a ∈ A on event E t ,

The last inequality follows from the definition of event E t . Specifically, for any action a ∈ A on event E t ,

Cases 2 and 3 are bounded as in Appendix A.6. Now we chain all inequalities, add them over all rounds, and get

where the last inequality uses the Cauchy-Schwarz inequality and the concavity of the square root. Finally, the sum ∑ n t =1 ∥ A t ∥ 2 ˆ Σ t is bounded using Lemma 10. This completes the proof.

0

10

20

30

40

50

60

70

80

Instance

10 -1

10 0

10 1

10 2

10 3

Regret difference

(a) Gaussian noise

0

10

20

30

40

50

60

70

80

Instance

10 -1

10 0

10 1

10 2

10 3

Regret difference

(b) Box noise

D Comparison of BayesUCB and UCB1

We report the difference in regret of UCB1 and BayesUCB on 81 Bayesian bandit instances. These instances are obtained by all combinations of K ∈ { 5 , 10 , 20 } actions, reward noise σ ∈ { 0 . 5 , 1 , 2 } , prior gap ∆ 0 ∈ { 0 . 5 , 1 , 2 } , and prior width σ 0 ∈ { 0 . 5 , 1 , 2 } . The horizon is n = 1000 rounds and all results are averaged over 1 000 random runs.

Our results are reported in Figure 3. In Figure 3a, the noise is Gaussian N (0 , σ 2 ) . In Figure 3b, the noise is σ with probability 0 . 5 and -σ otherwise. Therefore, this noise is σ 2 -sub-Gaussian, of the same magnitude as N (0 , σ 2 ) but far from it in terms of the distribution. This tests the robustness of BayesUCB to Gaussian posterior updates. UCB1 only needs σ 2 -sub-Gaussian noise. In both plots, and in all 81 Bayesian bandit instances, BayesUCB has a lower regret than UCB1 . It is also remarkably robust to noise misspecification, although we cannot prove it.

E Note on Information-Theory Bounds

Our approach could be used to derive Bayesian information-theory bounds [Russo and Van Roy, 2016]. The key step in these bounds, where the information-theory term I t,a for action a at round t arises, is ∆ A t ≤ Γ √ I t,A t , where Γ is the highest possible ratio of regret to information gain. As in Case 1 in Appendix A.6, the n -round regret can be bounded as

The term ∑ n t =1 I t,A t can be bounded using a worst-case argument by a O (log n ) bound.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proves logarithmic (in horizon $n$) regret bounds for Bayesian bandits using the BayesUCB algorithm. They specifically study the settings of Gaussian MABs and Gaussian linear bandits. This improves on prior works which show the more standard $\\sqrt{n}$ bounds on regret.\n\nSTRENGTHS:\n- Generally the writing is good. Problem setup and results are clear, as well as points of comparison with classical asymptotic guarantees.\n- The proofs are simple and easy to understand.\n- Numerical simulations are run to back up the validity of the theory.\n\nWEAKNESSES:\n- On the technical side, I'm not sure where the contributions are. The abstract states that \"our proofs mark a technical departure from prior works\". Can this be made more explicit in the technical discussion?\n- Several questions about the experiments, see next section. Generally the scope of the experiments is rather limited (which may be fine since this is primarily a theoretical work). \n\nSome comments on writing:\n- line 65: \"any bounded $\\theta$\": since we have not specified a specific parametric model here (it is not even clear that $\\theta$ must lie in some Euclidean space) a description of $\\theta$ being bounded seems premature here.\n- as a matter of writing style, it is a bit strange to put part of the inequality inline and the other half in display equations (such as line 120).\n\nQUESTIONS:\n1. Can the authors elaborate on how the techniques in the paper differ from prior works, and/or are clearer and simpler to understand? \n\n    a) From my understanding, the main parts of the analysis seem pretty similar to the standard frequentist analysis (i.e., the construction of confidence intervals and bounding the regret when the confidence interval event holds).\n\n2. The parameter $\\epsilon$ seems a bit mysterious to me. It seems like it is a parameter that is used only in the analysis. Perhaps the results could be more clearly stated in the main text by just setting $\\epsilon$ to the canonical values.\n\n3. The bounds that are obtained are not that easy to interpret.\n\n    a) Any intuition for what the $\\xi_a$ functions accomplish (e.g., in line 225)? Can one improve the range for $\\xi_a$ to be $\\Theta(1/\\mathrm{poly}(n))$? A slightly unsatisfying aspect of the claim when one takes $n\\to \\infty$ is that $1/\\sqrt{\\log n}$ goes to zero very slowly.\n\n    b) What is the intuition for the $\\sum_a \\int_{\\theta_{-a}}$ term? It seems like this comes from prior analysis. Why does it capture the dependence on the prior?\n\n    c) The linear gaussian bandit bound can be used to recover (a version) of the gaussian MAB bound. Can you discuss this? Is this tight in the relevant problem parameters? If not, why?\n\n4. line 147-148: the claim about the setting where $\\sigma_0 \\to \\infty$ is not clear to me why the complexity term should approach zero. Can this be formalized?\n\n5. Can the ideas be extended beyond Gaussian settings, and/or can they handle some degree of model misspecification (e.g., what happens if the rewards are close to gaussian in some distance metric).\n\n6. Experiments:\n    \n     a) Did you tune the confidence widths for the BayesUCB and UCB1 algorithm?\n     b) Can confidence bands be plotted?\n     c) I think the most interesting part is the comparison between BayesUCB and UCB1. Your theory suggests that BayesUCB improves over UCB1 because it takes into account the prior. Can this be demonstrated more robustly, i.e. for a range of experimental parameters. Also, we would expect this to happen in well-specified settings where the rewards are actually Gaussian. It would be interesting to see if this phenomena is robust and holds for real-world datasets (otherwise, in some sense, it might not be interesting to study BayesUCB...)\n    d) Why are your plotted bounds not monotone in $\\sigma_0 and \\Delta_0$? (Taking a step back, is this something that one should expect?)\n    e) In some regimes, your bounds for the linear setting are worse than the established $\\sqrt{n}$ bounds. Why is this so? Does this point to some looseness in the analysis?\n\n7) At a high level, why is BayesUCB interesting to study, and how does it compare to other Bayesian approaches, e.g., Thompson sampling? Which should one choose to use in practice?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a Bayesian setting of the multi-armed bandit problem and provides finite-time regret bounds. For the Gaussian bandits with independent priors, the authors provide regret analyses for BayesUCB and UCB1 algorithms. Both bounds scale as $O(c_h \\log^2 n)$ and match the lower bound by Lai (1987) up to constant factors. For the Linear bandits with Gaussian rewards, the authors provide an analysis for BayesUCB and the bound scales as $ O(c_\\Delta \\text{polylog}n)$. The empirical results corroborate the tightness of the bounds.\n\nSTRENGTHS:\nThis paper gives the first finite time bound for the Bayesian regret minimization problem with logarithmic regret for Gaussian reward and Gaussian prior setting. In the case of the multi-armed (unstructured) bandit setting, for both BayesUCB and UCB1, the order of $O(c_h \\log^2 n)$ Bayesian regret bound is given. For the linear Gaussian bandit case with bounded inverse gap, the bound does not scale with the number of arms and scales as $O(c_\\Delta \\text{polylog} n)$.\n\nWEAKNESSES:\nLai (1987) provides an upper bound that exactly matches the lower bound, including the universal constants. The authors' bound does not seem to match it asymptotically. \n\n\nThere is an assumption that the prior is factored. (Lai (1987) has weaker condition on the prior distribution (3.16) - (3.18) in Theorem 3.) I believe this assumption should be more explicitly stated/discussed in the main theorem.\n\nQUESTIONS:\nIf possible, \n\n- Could you provide the reasoning/intuition behind why it would be difficult to have matching upper bounds with constants asymptotically together with the finite round bounds?\n- Why the factored assumption is necessary or may not be? \n- Could you make contrast your proof with the proof of Lai (1987)? What techniques/devices make it possible to bound with a finite sample?\n- Why would it be difficult to get a lower bound with the linear bandit setting?\n- Would it be possible to get Bayesian regret bound for the linear bandit setting by integrating the complexity term without finite inverse gap  assumption?\n\nLIMITATIONS:\nSee the Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the Bayesian bandits problem and provide first finite-time logarithmic regret upper bounds by analyzing the BayesUCB algorithm from a new perspective, departing from previous studies. Their theoretical results demonstrate the significance of the prior. Additionally, the authors showcase the versatility of their analysis techniques by applying them to  linear bandits. Furthermore, they substantiate the tightness of their bounds through numerical evaluations.\n\nSTRENGTHS:\nThe authors provide first finite-time logarithmic regret upper bounds for Bayesian bandits, which improve the existing $\\tilde{O}(\\sqrt{n})$ bounds and highlight the significance of the prior. \nThe authors provide experimental results for both Gaussian bandits and linear bandits with Gaussian rewards. \nThe experimental results substantiate the theoretical findings. \nThe results and the addressed problem hold substantial significance and are likely to capture the interest of a broad audience in Neurips.\n\nWEAKNESSES:\nI have two primary concerns regarding this work and would like to request the authors to provide additional clarifications. \n\n==Weakness 1==\n\nWhile the authors present finite-time logarithmic regret upper bounds for Bayesian bandits, it is worth noting that Lai [1987] provide both asymptotic logarithmic upper and lower bounds. Therefore, it would be beneficial for the authors to explicitly highlight the key challenges associated with analyzing the finite-time case. This clarification would help readers better understand the distinctive aspects of the two settings.\n\n==Weakness 2==\n\nFrom a theoretical standpoint, the authors demonstrate the tightness of their results by comparing them with the asymptotic lower bound presented in Lai [1987]. This may raise concerns about the rigors. Additionally, it introduces a logical paradox concerning the contribution of the paper. If the theoretical analysis between the finite-time and asymptotic cases significantly differs, the comparison may not be sufficiently robust. On the other hand, if the comparison holds, it might suggest that the current results have somewhat limited contributions.\n\n==Minor Comment==\n\nRegarding the title, I suggest revising it to \"Finite-Time Logarithmic Bayes Regret Upper Bounds\" instead of the current title \"Logarithmic Bayes Regret Bounds.\" This change may be appropriate because Lai [1987] already covers both asymptotic logarithmic upper and lower bounds for Bayesian bandits.\n\nReference \n\nTze Leung Lai. Adaptive treatment allocation and the multi-armed bandit problem. The Annals of Statistics, 15(3):1091–1114, 1987.\n\nQUESTIONS:\nPlease see the points mentioned in the weaknesses above and provide comments on them.\n\nLIMITATIONS:\nI don't see any issue on societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper derives a finite-time regret bound for BayesUCB in Bayesian bandits. In Gaussian bandits, the bound grows linearly in $c_h \log^2(n)$, where $n$ is the time horizon, and $c_h$ is a constant determined by the prior. The bound matches that given by Lai (1987). A regret bound of BayesUCB in linear bandit is also established. Numerical experiments have also been conducted to support the theoretical findings.

STRENGTHS:
The paper studies an important problem and derives useful regret bounds. The paper is also well-organized and easy to follow.

WEAKNESSES:
In terms of presentation/ writing, I wish the paper could be more polished. Some of the examples are listed in “Questions.”

QUESTIONS:
There are some minor issues in writing. Below are a couple examples: 
1. In line 57, the paper reads “$A_t \in \mathcal{A}$.” However, $A_t$ is a random variable taking values in $\mathcal{A}$, so the statement is not precise. 
2. In line 83, it reads” $C_{t,a}$ is a confidence interval.” However, $C_{t,a}$ should be half the length of the confidence interval. 
3. Typo in line 97: should remove “a” from “the posterior distribution is a Gaussian.”

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the multi-arm bandit (MAB) problem, which is an online learning problem where a learner interacts with an environment over multiple rounds. In each round the learner would make an action, and each different action will incur a reward sampled from some distribution parametrized by an unknown $\\theta$. The goal is to get highest expected cumulative reward overtime, or in other words, to minimize the cumulative \"regret\" of not choosing the best action in each round. Here \"best\" means an action incurring the highest expected reward. The paper focuses on Bayesian MAB, where the unknown parameter $\\theta$ itself is generated from some known prior.\n\nThe paper proves several improved regret bounds for various Bayesian MAB settings, including: proving finite-time logarithmic Bayesian regret bounds for a Bayesian UCB algorithm in a Gaussian bandit, deriving a finite-time logarithmic Bayesian regret bound for a frequentist UCB algorithm to demonstrate the value of the prior as side information, proving a Bayesian regret bound for a Bayesian linear bandit algorithm, and empirically comparing the tightness of the bounds to prior bounds.\n\nSTRENGTHS:\nThe authors provide a logarithmic regret bounds for some basic Bayesian MAB problems, which improves exponentially on previous best result and matches an asymptotic lower bound. This seems to me a breakthrough.\n\nWEAKNESSES:\nN/A\n\nQUESTIONS:\nFirst, The paper under consideration falls outside my domain and I never bid for it (Regrettably I wasn't given any alternative submissions to assess). Despite my limited familiarity with MAB, the problems delineated in the manuscript, such as the Gaussian bandits, seem both intuitive and natural. The lower bound result referenced, by Lai et al., dates back several decades. Given this context, I presume these matters have been thoroughly examined within the domain. \n\nGiven this, one thing that appears perplexing to me is the paper's claim of exponential improvements over prior best results. Such a significant leap over such basic problems, usually suggests the introduction of some truly innovative ideas in the analysis. However, the proof seems quite innocuous and it's not immediately clear to me what is the core idea that leads to the improvement.\n\nIt will be very helpful if the authors can add one or two paragraphs elucidating previous results. Specifically, what is the major barrier that prevents them from achieving this logarithmic bound, and how the authors manages to overcome that barrier.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}