Title: Optimal Regret Is Achievable with Bounded Approximate Inference Error: An Enhanced Bayesian Upper Confidence Bound Framework
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian bandits

approximate Bayesian inference

Bayesian Upper Confidence Bound

optimal regret order

bounded inference error
Abstract: Bayesian bandit algorithms with approximate Bayesian inference have been widely used in real-world applications. However, there is a large discrepancy between the superior practical performance of these approaches and their theoretical justification. Previous research only indicates a negative theoretical result: Thompson sampling could have a worst-case linear regret $\Omega(T)$ with a constant threshold on the inference error measured by one $\alpha$-divergence. To bridge this gap, we propose an Enhanced Bayesian Upper Confidence Bound (EBUCB) framework that can efficiently accommodate bandit problems in the presence of approximate inference. Our theoretical analysis demonstrates that for Bernoulli multi-armed bandits, EBUCB can achieve the optimal regret order $O(\log T)$ if the inference error measured by two different $\alpha$-divergences is less than a constant, regardless of how large this constant is. To our best knowledge, our study provides the first theoretical regret bound that is better than $o(T)$ in the setting of constant approximate inference error. Furthermore, in concordance with the negative results in previous studies, we show that only one bounded $\alpha$-divergence is insufficient to guarantee a sub-linear regret.

Optimal Regret Is Achievable with Bounded Approximate Inference Error: An Enhanced Bayesian Upper Confidence Bound Framework

Abstract

Bayesian bandit algorithms with approximate Bayesian inference have been widely used in real-world applications. However, there is a large discrepancy between the superior practical performance of these approaches and their theoretical justification. Previous research only indicates a negative theoretical result: Thompson sampling could have a worst-case linear regret Ω( T ) with a constant threshold on the inference error measured by one α -divergence. To bridge this gap, we propose an Enhanced Bayesian Upper Confidence Bound (EBUCB) framework that can efficiently accommodate bandit problems in the presence of approximate inference. Our theoretical analysis demonstrates that for Bernoulli multi-armed bandits, EBUCB can achieve the optimal regret order O (log T ) if the inference error measured by two different α -divergences is less than a constant, regardless of how large this constant is. To our best knowledge, our study provides the first theoretical regret bound that is better than o ( T ) in the setting of constant approximate inference error. Furthermore, in concordance with the negative results in previous studies, we show that only one bounded α -divergence is insufficient to guarantee a sub-linear regret.

1 Introduction

The stochastic bandit problem, dated back to [37], is an important sequential decision-making problem that aims to find optimal adaptive strategies to maximize cumulative reward. At each time step, the learning agent chooses an action among all possible actions and observes its corresponding reward (but not others), and thus requires a balance between exploration and exploitation. Previous theoretical studies mainly focus on exact Bayesian bandit problems, requiring access to exact posterior distributions. However, their work cannot be easily applied to complex models such as deep neural networks, where maintaining exact posterior distributions tends to be intractable [36]. In contrast, approximate Bayesian bandit methods are widely employed in real-world applications with stateof-the-art performances [36, 41, 33, 45, 17, 50]. In comparison with exact algorithms, approximate Bayesian bandit algorithms are more challenging to analyze, as the inaccessibility of exact posterior sampling adds another level of discrepancy, and the resulting theory and solutions hence also differ substantially.

Few theoretical studies have been developed around Bayesian bandit approaches with approximate inference, despite their superior practical performance. [28] gave a theoretical analysis of an approximate sampling method called Ensemble Sampling, which possessed constant Kullback-Leibler divergence (KL divergence) error from the exact posterior and thus indicated a linear regret. [34]

showed that with a constant threshold on the inference error in terms of α -divergence (a generalization of KL divergence), regardless of how small the threshold is, Thompson sampling with general approximate inference could have a linear regret in the worst case. Their work also showed that Thompson sampling combined with a small amount of forced exploration could achieve a o ( T ) regret upper bound, but no better result than o ( T ) was shown. Moreover, this improvement was mostly credited to the forced exploration rather than the intrinsic property of Thompson sampling. It appears that [34] illustrated a paradox that approximate Bayesian bandit methods worked well empirically but failed theoretically. Thus, further study regarding the fundamental understanding of approximate Bayesian bandit methods is necessary.

Motivated by the negative results in [34], [30] leveraged an efficient Markov chain Monte Carlo (MCMC) Langevin algorithm in multi-armed Thompson sampling so that the inference error would vanish along with increased sample numbers. [48] extended [30] to the contextual bandit problems and integrated contextual Thompson sampling with the Langevin algorithm that allowed the approximate posterior distribution to be sufficiently close to the exact posterior distribution. Hence, both works [48, 30] had a similar feature in terms of vanishing inference error. However, in other inference approaches, such as variational inference [8], the approximate posteriors might incur a systematic computational bias. To accommodate the latter scenario, we consider a general Bayesian inference approach that allows bounded inference error:

Is it possible to achieve the optimal regret order O (log T ) with a constant (non-vanishing) threshold on the inference error?

This question is not well investigated in previous literature, even for Bernoulli bandits. [34] showed that the answer is No for Thompson sampling when the inference error measured by only one α -divergence is bounded. In this study, we will provide a novel theoretical framework and point out that the answer could be Yes when the inference error measured by two different α -divergences is bounded where one α is greater than 1, and the other α is less than 0. This assumption guarantees that the approximate posterior is close to the exact posterior from two different "directions". Our finding implies that the problem of sub-optimal regret in the presence of approximate inference may not arise from the constant but from the design of the inference error. Our study takes the first step in deriving positive answers in the presence of constant approximate inference error, which provides some theoretical support for the superior performance of approximate Bayesian bandit methods.

In this study, we extend the work of the Bayesian Upper Confidence Bound (BUCB) [21, 20, 42, 17] to the setting of approximate inference and propose an enhanced Bayesian bandit algorithm that can efficiently accommodate approximate inference, termed as Enhanced Bayesian Upper Confidence Bound (EBUCB). In particular, we redesign the quantile choice in the algorithm to address the challenge of approximate inference: The original choice of t -1 provides the best regret bound without approximate inference, but in the presence of approximate inference, it leads to an undesirable quantile shift which degrades the performance. By adjusting the quantile choice, we theoretically demonstrate that EBUCB can achieve the optimal regret order O (log T ) if the inference error measured by two different α -divergences ( α 1 > 1 and α 2 < 0 ) is bounded. We also provide insights in the other direction: Instead of two different α -divergences, controlling one α -divergence alone is not sufficient to guarantee a sub-linear regret for both Thompson sampling, BUCB, and EBUCB. This further suggests that naive approximate inference methods that only minimize one α -divergence alone could perform poorly, and thus it is critical to design approaches with two different α -divergences reduced.

Our main contributions are summarized as follows:

1) We propose a general Bayesian bandit framework, named EBUCB, to address the challenge of approximate inference. Our theoretical study shows that for Bernoulli bandits, EBUCB can achieve a O (log T ) regret upper bound when the inference error measured by two different α -divergences is bounded. To the best of our knowledge, with constant approximate inference error, there is no existing result showing a regret upper bound that is better than o ( T ) , even for Bernoulli bandits. 2) We develop a novel sensitivity analysis of quantile shift with respect to inference error. This provides a fundamental tool to analyze Bayesian quantiles in the presence of approximate inference, which holds promise for broader applications, e.g., when the inference error is time-dependent. 3) We demonstrate that one bounded α -divergence alone is insufficient to guarantee a sub-linear regret.

Worst-case examples are constructed and illustrated where Thompson sampling/BUCB/EBUCB has Ω( T ) regret if only one α -divergence is bounded. Hence, special consideration on reducing two different α -divergences is necessary for real-world applications.

4) Our experimental evaluations corroborate our theory well, showing that our EBUCB is consistently superior to BUCB and Thompson sampling on multiple approximate inference settings.

Related Work. Bandit problems and their theoretical optimality have been extensively studied over decades [27, 10]. The seminal paper [25] (and subsequently [11]) established the first problemdependent frequentist regret lower bound, showing that without any prior knowledge on the distributions, a regret of order (log T ) is unavoidable. Two popular lines of Bayesian bandit algorithms, Thompson sampling [1, 22, 16] and BUCB [21, 20], had been shown to match the lower bound, which indicated the theoretical optimality of those algorithms. Beyond Gaussian processes [42] and linear models [3, 39], exact computation of the posterior distribution is generally intractable, and thus, the approximate Bayesian inference techniques are necessary.

Some recent work focused on designing specialized methods to construct Bayesian indices since previous studies had demonstrated that Thompson sampling with constant inference error could exhibit linear regret in the worst-case scenario [28, 34]. [30] constructed Langevin algorithms to generate approximate samples with decreasing inference error and showed an optimal problemdependent frequentist regret. [32] proposed variational Bayesian optimistic sampling, suggesting solving a convex optimization problem over the simplex at every time step. Unlike these researches, our study presents general results that only depend on the error threshold of approximate inference, rather than some specific approximate inference approaches.

Beyond Bayesian, another mainstream of bandit algorithms to address the exploration-exploitation tradeoff is upper confidence bound (UCB)-type frequentist algorithms [5-7, 13, 15, 40, 51, 24, 29, 43]. [12] revealed that Thompson sampling empirically outperformed UCB algorithms in practice, partly because UCB was typically conservative, as its configuration was data-independent which led to over-exploration [18]. BUCB [21] could be viewed as a middle ground between Thompson sampling and UCB. On the other hand, empirical studies [20] showed that Thompson sampling and BUCB performed similarly well in general.

2 Methodology

The stochastic multi-armed bandit problem consists of a set of K actions (arms), each with a stochastic scalar reward following a probability distribution ν i ( i = 1 , ..., K ). At each time step t = 1 , ..., T , where T is the time horizon, the agent chooses an action A t ∈ [ K ] and in return observes an independent reward X t drawn from the associated probability distribution ν A t . The goal is to devise a strategy A = ( A t ) t ∈ [ T ] , to maximize the accumulated rewards through the observations from historical interactions.

In general, a wise strategy should be sequential, in the sense that the upcoming actions are determined and adjusted by the past observations: letting F t = σ ( A 1 , X 1 , ..., A t , X t ) be the σ -field generated by the observations up to time t , A t is σ ( F t -1 , U t ) -measurable, where U t is a uniform random variable independent from F t -1 (as algorithms may be randomized). More precisely, let µ j ( j ∈ [ K ] ) denote the mean reward of the action j (i.e., the mean of the distribution ν i ), and without loss of generality, we assume that µ 1 = max j ∈ [ K ] µ j . Then maximizing the rewards is equivalent to minimizing the (frequentist) regret, which is defined as the expected difference between the reward accumulated by an 'ideal' strategy (a strategy that always playing the best action), and the reward accumulated by a strategy A :

The expectation is taken with respect to both the randomness in the sequence of successive rewards from each action j , denoted by ( Y j,s ) s ∈ N , and the possible randomization of the algorithm, ( U t ) t ∈ [ T ] . Let N j ( t ) = ∑ t s =1 ✶ ( A s = j ) denote the number of draws from action j up to time t , so that X t = Y A t ,N At ( t ) . Moreover, let ˆ µ j,s = 1 s ∑ s k =1 Y j,k be the empirical mean of the first s rewards from action j and let ˆ µ j ( t ) be the empirical mean of action j after t rounds of the bandit algorithm. Therefore ˆ µ j ( t ) = 0 if N j ( t ) = 0 , ˆ µ j ( t ) = ˆ µ j,N j ( t ) otherwise.

Note that the true mean rewards µ = ( µ 1 , ..., µ K ) are fixed and unknown to the agent. In order to perform Thompson sampling, or more generally, Bayesian approaches, we artificially define a prior distribution Π 0 on µ . Let Π t be the exact posterior distribution of µ |F t -1 with density function π t

Algorithm 1

Enhanced Bayesian Upper Confidence Bound (EBUCB) with Approximate Inference Input: T (time horizon), Π 0 = Q 0 (initial prior on µ ), c (parameters of the quantile), and a real-value increasing sequence { γ t } such that γ t → 1 as t →∞ for t = 1 to T do for each action j = 1 , ..., K do Compute qu j ( t ) = Qu ( γ t , Q t -1 ,j ) . end for Draw action A t = arg max j =1 ,...,K qu j ( t ) Get reward X t = Y A t ,N At ( t ) Obtain the approximate distribution Q t end for with marginal distributions π t, 1 , ..., π t,K for actions 1 , ..., K . Specifically, if at time step t , the agent chooses action A t = j and consequently observes X t = Y A t ,N At ( t ) , the Bayesian update for action j is

whereas for i = j , π t,i = π t -1 ,i . At each time step t , we assume that the exact posterior computation in (2) cannot be obtained explicitly and an approximate inference method is able to give us an approximate distribution Q t (instead of Π t ). We use q t to denote the density function of Q t .

̸

First, we consider a standard case where the exact posterior is accessible. In Thompson sampling [1, 44], a sample ˆ m is drawn from the posterior distribution Π t -1 and then an action A t is selected using the following strategy: A t = i if ˆ m i = max j ˆ m j . In BUCB [21], we compute the quantile of the posterior distribution qu j ( t ) = Qu (1 -1 t (log T ) c , Π t -1 ,j ) for each action j , where Qu ( γ, ρ ) is the quantile function associated to the distribution ρ , such that P ρ ( X ≤ Qu ( γ, ρ )) = γ . Then we select action A t as follows: A t = i if qu i ( t ) = max j qu j ( t ) .

Next, we move to a more concrete example, Bernoulli multi-armed bandit problems with a standard setting used in seminal papers [1, 2, 21, 22]. In these problems, each (stochastic) reward follows a Bernoulli distribution ν i ∼ Bernoulli ( µ i ) and these distributions are independent of each other. The prior Π 0 ,j is typically chosen to be the independent and identically distributed (i.i.d.) Beta (1 , 1) , or the uniform distribution for every action j . Then the posterior distribution for action j is a Beta distribution Π t,j = Beta (1 + S j ( t ) , 1 + N j ( t ) -S j ( t )) , where S j ( t ) = ∑ t s =1 ✶ { A s = j } X t is the empirical cumulative reward from action j up to time t . Then, Thompson sampling/BUCB chooses the samples/quantiles of the posterior Π t,j = Beta (1 + S j ( t ) , 1+ N j ( t ) -S j ( t )) respectively at each time step.

In the presence of approximate inference, Thompson sampling draws the sample ˆ m from Q t -1 , as the exact Π t -1 is not accessible. Correspondingly, we modify the specific sequence of quantiles chosen by the BUCB algorithm with a general sequence of { γ t } -quantiles and term it as Enhanced Bayesian Upper Confidence Bound (EBUCB) algorithm. The detailed pseudo algorithm of EBUCB is described in Algorithm 1. Note that the choice of γ t should address the presence of inference error and should be trailed to the specific definition of inference error; See Remark 3.10 in Section 3.

3 Theoretical Analysis

In this section, we present a theoretical analysis of EBUCB. In Section 3.1, we provide the necessary background of α -divergence on approximate inference error measurement. Then in Section 3.2, we develop a novel sensitivity analysis of quantile shift with respect to inference error. This provides a fundamental tool to analyze Bayesian quantiles in the presence of approximate inference. The general results therein will be used for our derivation for the regret upper bound of EBUCB in Section 3.3, and are also potentially useful for broad applications, e.g., when the inference error is time-dependent. Lastly, in Section 3.4, we provide examples where Thompson sampling/BUCB/EBUCB has a linear regret with arbitrarily small inference error measured by one α -divergence alone. All proofs are given in the Appendix.

3.1 The Alpha Divergence for Inference Error Measurement

The α -divergence, generalizing the KL divergence, is a common way to measure errors in inference methods.

Definition 3.1. The α -divergence between two distributions P 1 and P 2 with density functions p 1 ( x ) and p 2 ( x ) is defined as: D α ( P 1 , P 2 ) = 1 α ( α -1) (∫ p 1 ( x ) α p 2 ( x ) 1 -α dx -1 ) , where α ∈ R and the case of α = 0 and 1 is defined as the limit.

Note that different studies use the α parameter in different ways. Herein, our definition of α -divergence does not follow Renyi's definition of α -divergence [35]; Instead, we follow a generalized version of Tsallis's α -divergence, which is adopted by [52, 31, 34]. Compared with Renyi's α -divergence, Tsallis's α -divergence does not involve a log function, and it has the following property:

Proposition 3.2 (Positivity and symmetry) . For any α ∈ R , D α ( P 1 , P 2 ) ≥ 0 and D α ( P 1 , P 2 ) = D 1 -α ( P 2 , P 1 ) .

The α -divergence contains many distances such as KL ( P 2 , P 1 )( α → 0) , KL ( P 1 , P 2 )( α → 1) , Hellinger distance ( α = 0 . 5) , and χ 2 divergence ( α = 2) . α -divergence is widely used in variational inference [8, 23, 26], which is one of the most popular approaches in Bayesian approximate inference. Moreover, it was also adopted in previous studies on Thompson sampling with approximate inference [34, 28]. In particular, the KL divergence ( ( α = 1) -divergence) is:

KL ( P 1 , P 2 ) = ∫ p 1 ( x ) log ( p 1 ( x ) p 2 ( x ) ) dx. In approximate Bayesian inference, the exact posterior distribution Π t and the approximate distribution Q t may differ from each other. To provide a statistical analysis of approximate sampling methods, we use the α -divergence as the measurement of inference error (statistical distance) between Π t and Q t . Our starting point is the following:

Assumption 3.3. Suppose that there exists a positive value ϵ ∈ (0 , + ∞ ) and two different parameters α 1 > 1 and α 2 < 0 such that

This assumption is adapted from [34] but we enhance theirs with two bounded α -divergences, as [34] showed one bounded α -divergence was not sufficient to guarantee the sublinear regret. However, in the following, we show that the optimal regret order O (log T ) is indeed achievable under Assumption 3.3 with two bounded α -divergences. Intuitively, P 2 is flatten to cover P 1 's entire support when minimizing D α ( P 1 , P 2 ) with a large α (greater than 1), while when α is small (less than 0), P 2 fits the P 1 's dominant mode; See [31] for the implication of α -divergence. Therefore, Assumption 3.3 guarantees that the approximate posterior is close to the exact posterior from two different 'directions'. It is worth mentioning that when one α -divergence is small, it does not necessarily imply that any other α -divergences are large or infinite. In fact, as long as the two distributions have densities with the same support, then any α -divergence between them is finite. Note that Assumption 3.1 does not require the threshold ϵ to be small; instead, ϵ can be any finite positive number. We pinpoint that this assumption, as well as our subsequent results, are very general in the sense that it does not depend on any specific methods of approximate inference. To enhance credibility on Assumption 3.3, we make several additional remarks in Section A.

3.2 Quantile Shift with Inference Error

In this section, we develop a novel sensitivity analysis of quantile shift with respect to inference error, which implies that under Assumption 3.3, the γ -quantiles of Π t,j and Q t,j only differs from a bound depending on ϵ . We provide a general result first, which is rigorously stated as follows:

Theorem 3.4. Consider any two distributions P 1 and P 2 with densities p 1 ( x ) and p 2 ( x ) . Let R i denote the quantile function of the distribution P i , i.e., R i ( p ) := Qu ( p, P i ) ( i = 1 , 2 ). Let 0 < γ < 1 . Let δ γ,ϵ satisfy that R 1 ( γ ) = R 2 ( γ + δ γ,ϵ ) where -γ ≤ δ γ,ϵ ≤ 1 -γ .

a) If D α ( P 1 , P 2 ) ≤ ϵ where α > 1 , then

Note that when α > 1 , ( ϵα ( α -1) + 1) 1 1 -α < 1 and (1 -γ ) α α -1 < 1 -γ .

b) If D α ( P 1 , P 2 ) ≤ ϵ where α < 0 , then

Note that when α < 0 , ( ϵα ( α -1) + 1) 1 1 -α > 1 and (1 -γ ) α α -1 > 1 -γ .

c) Suppose that α ∈ (0 , 1) and ϵ ≥ -1 α ( α -1) . Then for any δ γ,ϵ ∈ [ -γ, 1 -γ ] , there exist two distributions P 1 and P 2 such that D α ( P 1 , P 2 ) ≤ ϵ. This implies that the condition D α ( P 1 , P 2 ) ≤ ϵ cannot control the quantile shift between P 1 and P 2 in general when α ∈ (0 , 1) .

Theorem 3.4 states that γ -quantile of the distribution P 1 is the ( γ + δ γ,ϵ ) -quantile of the distribution P 2 where the quantile shift δ γ,ϵ has the following properties. a) The upper bound of δ γ,ϵ is close to 0 if D α ( P 1 , P 2 ) with α > 1 is bounded; b) The lower bound of δ γ,ϵ is close to 0 if D α ( P 1 , P 2 ) with α < 0 is bounded; c) A slightly large bound on D α ( P 1 , P 2 ) with 0 < α < 1 cannot control the shift δ γ,ϵ in general, which gives the intuition that α ∈ (0 , 1) is not implemented in Assumption 3.3.

This theorem is distribution-free, in the sense that the bound of δ γ,ϵ does not depend on any specific distributions (noting that distribution changes as t evolves in bandit problems). In particular, a)+b) in Theorem 3.4 shows that C ϵ,α 1 (1 -γ ) α 1 α 1 -1 ≥ δ γ,ϵ -(1 -γ ) ≥ -C ϵ,α 2 (1 -γ ) α 2 α 2 -1 , with α 1 > 1 and α 2 < 0 where C ϵ,α = ( ϵα ( α -1) + 1) 1 1 -α is independent of γ or distributions, and thus independent of the time step t in our EBUCB algorithm. This observation is important in the robustness of using quantiles in the EBUCB. The proof of Theorem 3.4 relies on the following lemma, which provides a quantile-based representation of α -divergence.

Lemma 3.5. Under the same conditions in Theorem 3.4, we have that for any α -divergence, D α ( P 1 , P 2 ) = ∫ 1 0 ( d du R -1 2 ( R 1 ( u )) ) 1 -α du -1 .

3.3 Finite-Time Regret Bound for EBUCB

In this section, we derive a finite-time upper bound of problem-dependent frequentist regret for our EBUCB algorithm in Bernoulli multi-armed bandit problems. Without loss of generality, we assume action 1 is optimal in the subsequent theorems.

To begin with, we first express the frequentist regret as R ( T, A ) := E [ ∑ T t =1 ( µ 1 -µ A t ) ] = ∑ K j =1 ( µ 1 -µ j ) E [ N j ( t )] by (1). Therefore, it is sufficient to study each term E [ N j ( t )] in order to bound the regret R ( T, A ) . For ( p, q ) ∈ [0 , 1] 2 , we denote the Bernoulli KL divergence between two points by d ( p, q ) = p log( p q ) + (1 -p ) log( 1 -p 1 -q ) , with 0 log 0 = 0 log(0 / 0) = 0 and x log( x/ 0) = + ∞ for x > 0 by convention. We also denote that d + ( p, q ) = d ( p, q ) { p < q } for convenience.

✶ Note that π t,j ( x ) is the density of Beta (1 + S j ( t ) , 1 + N j ( t ) -S j ( t )) so its (closed) support is [0 , 1] . We put a basic assumption on the approximate distribution Q t,j .

Assumption 3.6. Q t,j has the density q t,j ( x ) whose support is [0 , 1] for any t ∈ [ T ] , j ∈ [ K ] .

The following is our main theorem, which establishes a finite-time regret bound for our EBUCB algorithm.

Theorem 3.7. Suppose Assumptions 3.3 and 3.6 hold. Let M ϵ, 1 = ( ϵα 1 ( α 1 -1) + 1) 1 1 -α 1 < 1 , ˜ α 1 = α 1 α 1 -1 > 0 , M ϵ, 2 = ( ϵα 2 ( α 2 -1) + 1) 1 1 -α 2 > 1 , and ˜ α 2 = α 2 α 2 -1 > 0 . For any ξ > 0 , choosing the parameter c such that c ˜ α 2 ≥ 5 in the EBUCB algorithm and setting γ t = 1 -1 t ζ (log T ) c ( ζ > 0) , the number of draws of any sub-optimal action j ≥ 2 is upper-bounded by

if 0 < ζ ˜ α 2 < 1 , and if ζ ˜ α 2 ≥ 1 .

Theorem 3.7 provides an exact finite-time regret bound and the o ( · ) term in Theorem 3.7 has an exact finite-time closed-form expression that holds for any time horizon T ; See Step 4 in the proof of Theorem 3.7 in Appendix C. We only show the most dominant term of the regret bound and shrink the rest to the o ( · ) term to improve the readability of the main paper. Note that this bound has explicit dependence on ϵ , which is M -1 ϵ, 1 and M ϵ, 2 in Step 4 in the proof of Theorem 3.7. Obviously, the error terms M -1 ϵ, 1 and M ϵ, 2 in the bound increase as ϵ increases. However, this dependence on ϵ does not impact the dominating term too much. The exact posterior will be more 'concentrated' on the true mean with small variability as the time t increases, and the impact from the error will vanish; See Remark 3.9.

It is easy to see that to minimize the regret upper bound, we may choose ζ = 1 ˜ α 2 in Theorem 3.7.

Corollary 3.8. Under the same conditions in Theorem 3.7, for any ξ > 0 , choosing the parameter c such that c ˜ α 2 ≥ 5 in the EBUCB algorithm and setting γ t = 1 -1 t 1 / ˜ α 2 (log T ) c , the number of draws

˜

of any sub-optimal action j ≥ 2 is upper-bounded by E [ N j ( T )] ≤ (1+ ξ ) ˜ α 2 d ( µ j ,µ 1 ) log( T ) + o (log T ) .

α

1

This result states that with the ϵ error threshold, the regret of the EBUCB algorithm is bounded above by O (log T ) regardless of how large ϵ is, which reaches the same order (log T ) of the problemdependent frequentist regret lower bound [25]. In comparison with the exact lower bound, there is a slight difference in the multiplier before the order (log T ) : Our upper bound in Corollary 3.8 has the additional multiplier ˜ α 1 ˜ α 2 > 1 , which arises from the approximate inference when estimating the posterior distributions (Assumption 3.3). If in addition Assumption 3.3 holds for any α 1 > 1 and any α 2 < 0 , then we can let ˜ α 1 ˜ α 2 → 1 by taking α 1 → + ∞ and α 2 →-∞ to match the exact lower bound.

In the absence of approximate inference, [21] showed that E [ N j ( T )] ≤ (1+ ξ ) d ( µ j ,µ 1 ) log( T ) + o (log T ) matching the exact lower bound. Prior to our work, it was unclear in the literature whether the optimal regret order O (log T ) could be achieved in the presence of bounded approximate inference error. Our result provides a positive answer to this question, despite the fact that the inference error may cause the multiplier ˜ α 1 ˜ α 2 before the order (log T ) . To the best of our knowledge, this is the first algorithm providing the regret upper bound that is better than o ( T ) in [34] with bounded approximate inference error.

As discussed in [21], the horizon-dependent term (log T ) c in Corollary 3.8 is only an artifact of the theoretical analysis to obtain a finite-time regret upper bound. In practice, the model with choice c = 0 (i.e., without the horizon-dependent term) already achieves superior performance. This is confirmed by our experiments in Section 4. A similar observation in BUCB was indicated in [21].

Remark 3.9 . It might appear a little surprising that the result in Corollary 3.8 indicates a regret upper bound with the dominating term O (log T ) that does not depend on ϵ , as one may expect that a large ϵ allows the 'full swap' between the posterior of the optimal action and the posterior of a suboptimal action, making any Bayesian-based approaches unable to distinguish them. However, benefiting from historical observations, the exact posterior will be more 'concentrated' on the true mean with small variability, which will keep enlarging the α -divergence between two actions. This indicates that, for a fixed ϵ , the α -divergence between the exact posteriors of two actions can be sufficiently large as t increases and ultimately exceeds ϵ . Hence, the 'full swap' will not happen when t is large.

Remark 3.10 . The 1 t 1 / ˜ α 2 in EBUCB, instead of the original 1 t in BUCB, is a delicate choice to address the tradeoff between making the regret optimal without approximate inference and the presence of inference error. On a technical level, a power ζ close to 1 in t ζ improves the regret bound without the presence of approximate inference but simultaneously leads to high-level quantile shift caused by approximate inference. Choosing ζ = 1 ˜ α 2 is a subtle balance of these two.

The technical derivation of Theorem 3.7 depends on analyzing the quantiles of the approximate distributions used in the EBUCB algorithm. In particular, one of the major techniques in our analysis is Lemma C.1 in Appendix C. It provides explicit upper and lower bounds on the tails of approximate distributions to control the quantiles designed by the EBUCB algorithm. It is obtained by combining the quantile shift between the approximate and exact posterior distributions that developed in Theorem 3.4 (Section 3.2) with the tight bounds on the quantiles of the exact posterior distributions (the proof of Lemma 1 in [21]). This result is then used to bound the expectation of a decomposition of N j ( T ) in Lemma C.2 that links N j ( T ) to the over-estimation of the optimal arm.

3.4 Negative Results

We show that one bounded α -divergence alone cannot guarantee a sub-linear regret. We provide two worst-case examples, one where Thompson sampling has a linear regret, and the other where BUCB/EBUCB has a linear regret, even when the inference error measured by one α -divergence is small. A similar study on Thompson sampling was conducted in [34] with a special focus on the inference error on the joint distribution of all actions. In our study, nevertheless, we focus on a setting where the inference error on the distribution of each action is assumed; See Remark A.3. Therefore, the examples in [34] cannot be directly applied in our setting. Moreover, our second example shows that BUCB/EBUCB could have a linear regret if only one α -divergence is considered, which is new.

Assumption 3.11. Suppose that there exists a positive value ϵ ∈ (0 , + ∞ ) such that

We establish the following theorem for Thompson sampling:

Theorem 3.12. Consider a Bernoulli multi-armed bandit problem where the number of actions is K = 2 and µ 1 > µ 2 . The prior Π 0 ,j is chosen to be the i.i.d. Beta (1 , 1) , or the uniform distribution for every action j = 1 , 2 . For any given α < 1 and any error threshold ϵ > 0 , there exists a sequence of distributions Q t -1 such that for all t ≥ 1 :

1) The probability of sampling from Q t -1 choosing action 2 is greater than a positive constant independent of t .

2) Q t -1 satisfies Assumptions 3.6 and 3.11.

Therefore Thompson sampling from the approximate distribution Q t -1 will cause a finite-time linear frequentist regret: R ( T, A ) = Ω( T ) .

This theorem shows that making one α -divergence a small constant alone, even for each action j , is not sufficient to guarantee a sub-linear regret of Thompson sampling. Note that Theorem 3.12 is an enhancement of the results in [34] in the sense that the Q t constructed by our theorem satisfies more restrictive assumptions. We can derive a similar observation for the BUCB/EBUCB algorithm as follows:

Theorem 3.13. Consider a Bernoulli multi-armed bandit problem where the number of actions is K = 2 and µ 1 > µ 2 . The prior Π 0 ,j is chosen to be the i.i.d. Beta (1 , 1) , or the uniform distribution for every action j = 1 , 2 . Consider the general EBUCB algorithm described in Algorithm 1. For any given α < 1 and any error threshold ϵ > 0 , there exists a constant T 0 (only depending on ϵ , α , and the sequence { γ t } ) and a sequence of distributions Q t -1 such that for all t ≥ 1 :

1) The EBUCB algorithm always chooses action 2 when t ≥ T 0 .

2) Q t -1 satisfies Assumptions 3.6 and 3.11.

Therefore the EBUCB algorithm from the approximate distribution Q t -1 will cause a finite-time linear frequentist regret: R ( T, A ) = Ω( T ) .

This theorem shows that making one α -divergence a small constant alone, even for each action j , is insufficient to guarantee a sub-linear regret of BUCB/EBUCB. We emphasize that the examples in Theorems 3.12 and 3.13 are in the worst-case sense, indicating that there exist worst-case examples where Thompson sampling/EBUCB exhibits a linear regret if only one α -divergence is bounded. However, this does not imply that EBUCB and Thompson sampling would fail on average in the presence of approximate inference. In fact, Theorem 3.7 shows that a sub-linear regret can be achieved if the inference error measured by two different α -divergences is bounded.

4 Experiments

In this section, we conduct numerical experiments to show the correctness of our theory. 2 In Section 4.1, we compare the performance of EBUCB with the following baselines: BUCB (using its originally proposed quantile and Q t as Π t since the exact posterior distribution Π t is unavailable) and Thompson sampling (using Q t as Π t ). In Section 4.2, we construct worst-case examples showing that both EBUCB and Thompson sampling can degenerate to linear regret if only one α -divergence is bounded. We consider the Bernoulli multi-armed bandit problem which has two actions with mean rewards

(P1) with w=0.9

(P1) with w=0.8

(P1) with w=0.7

14

16

17.5

12

14

15.0

10

Regret

12.5

10

12

Cumulative

10.0

7.5

5.0

2.5

0.0

200

400

Time Step

600

800

1000

0

200

400

Time Step

600

800

1000

200

400

Time Step

600

800

1000

(P2) with w=0.9

(P2) with w=0.8

(P2) with w=0.7

10

10

10

200

400

Time Step

600

800

1000

0

200

400

Time Step

600

800

1000

0

200

400

Time Step

600

800

1000

EBUCB (c = 0)

Thompson Sampling

BUCB (c = 0)

BUCB (c = 5)

EBUCB (c =5)

ThompsonSampling

BUCB (c = 0)

EBUCB (c =0)

175

100

r = 1.1

r=1.011

r = 1.111

80

r= 2

r=10

150

r=1.005

r =1.003

150

= 1.076

r= 1.058

Regret

Regret

125

Regret

125

Cumulative P

60

Cumulative

100

Cumulative

100

40

75

75

50

50

20

25

25

0

0

0

0

100

200

Time Step

300

400

500

0

100

200

Time Step

300

400

500

0

100

200

300

400

500

Time Step

[0 . 7 , 0 . 3] , and use Beta (1 , 1) as the prior distribution of mean reward for each action. At each time step t , the exact posterior distribution for each action is Beta (1 + S j ( t ) , 1 + N j ( t ) -S j ( t )) , where S j ( t ) is the empirical cumulative reward from action j up to time t and N j ( t ) is the number of draws from action j up to time t .

4.1 Generally Misspecified Posteriors

Suppose the posterior distributions are misspecified to the following distributions:

where w = 0 . 9 , 0 . 8 , 0 . 7 . Figure 1 presents the results of EBUCB and the baselines. Overall, EBUCB achieves consistently superior performance than the baselines, and it outperforms BUCB with considerable improvements. These results confirm the effectiveness of EBUCB across multiple settings. Moreover, EBUCB performs well without the horizon-dependent term (i.e., c = 0 in Corollary 3.8). This brings EBUCB practical advantages in real-world applications, as it does not require advanced knowledge of the horizon (i.e., anytime ). A similar observation of BUCB was also noticed in [21].

4.2 Worst-Case Misspecified Posteriors

We consider the worst-case examples, Equations (13) and (14), presented in the proof of Theorems 3.12 and 3.13, where the posterior distributions are misspecified using one α -divergence. The results of Thompson sampling, BUCB, and EBUCB are displayed in Figure 2. From these worst-case examples, we observe that: 1) Thompson sampling exhibits a linear regret after t ≥ 1 . As shown in Theorem 3.12, the linear coefficient (i.e., the slope) of the regret depends on the level r that corresponds to the inference error. Specifically, the slope of the regret is increased along with the increased value of r , as illustrated in both Figure 2 and the proof of Theorem 3.12. 2) BUCB/EBUCB exhibits a linear regret with constant slope µ 1 -µ 2 after t ≥ T 0 , where T 0 is the time threshold introduced in Theorem 3.13 after which BUCB/EBUCB always chooses the sub-optimal action. The artificial choice of r is to make T 0 = 100 , 200 , 333 where γ T 0 = 1 r ; See the proof of Theorem 3.13.

In summary, our experiments evidently demonstrate the superior performance of our proposed EBUCB on multi-armed bandit problems with generally misspecified posteriors. Our results also align closely with our theory that making one α -divergence a small constant alone is insufficient to guarantee a sub-linear regret of Thompson sampling/BUCB/EBUCB. Hence, making two different α -divergences bounded is necessary for the sub-linear regret upper bound.

5 Conclusions and Future Work

In this paper, we propose a general Bayesian bandit algorithm, Enhanced Bayesian Upper Confidence Bound (EBUCB), that achieves superior performance for Bernoulli bandit problems with approximate inference. We prove that, if the inference error measured by two different α -divergences is less than a constant, EBUCB can achieve the optimal regret order O (log T ) . Additionally, we construct worse-case examples to show the necessity of bounding two different α -divergences, which is further validated by our experiments. We consider the study of other problem settings as meaningful future research that could be built upon our current framework, e.g., extending to the general exponential family bandit problems by leveraging the techniques in [20]. We will also extend our current framework to contextual bandit problems and investigate the performance of contextual Bayesian bandit algorithms with approximate inference [36].

Acknowledgments and Disclosure of Funding

This work has been supported in part by the National Science Foundation under grants CAREER CMMI-1834710 and IIS-1849280, and the Cheung-Kong Innovation Doctoral Fellowship. The authors thank the anonymous reviewers for their constructive comments which have helped greatly improve the quality of our paper.

References

S. Agrawal and N. Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on Learning Theory , pages 39-1, 2012.

S. Agrawal and N. Goyal. Further optimal regret bounds for thompson sampling. In Artificial Intelligence and Statistics , pages 99-107, 2013.

S. Agrawal and N. Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning , pages 127-135, 2013.

C. Andrieu, N. De Freitas, A. Doucet, and M. I. Jordan. An introduction to mcmc for machine learning. Machine Learning , 50(1):5-43, 2003.

P. Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47(2):235-256, 2002.

P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77, 2002.

D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association , 112(518):859-877, 2017.

R. Brekelmans, V. Masrani, F. Wood, G. Ver Steeg, and A. Galstyan. All in the exponential family: Bregman duality in thermodynamic variational inference. In International Conference on Machine Learning , pages 1111-1122. PMLR, 2020.

S. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. arXiv preprint arXiv:1204.5721 , 2012.

A. N. Burnetas and M. N. Katehakis. Optimal adaptive policies for sequential allocation problems. Advances in Applied Mathematics , 17(2):122-142, 1996.

O. Chapelle and L. Li. An empirical evaluation of thompson sampling. Advances in Neural Information Processing Systems , 24:2249-2257, 2011.

V. Dani, T. P. Hayes, and S. M. Kakade. Stochastic linear optimization under bandit feedback. 21st Annual Conference on Learning Theory , 2008.

K. Daudel et al. Mixture weights optimisation for alpha-divergence variational inference. Advances in Neural Information Processing Systems , 34:4397-4408, 2021.

A. Garivier and O. Cappé. The kl-ucb algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th Annual Conference on Learning Theory , pages 359-376. JMLR Workshop and Conference Proceedings, 2011.

A. Gopalan, S. Mannor, and Y. Mansour. Thompson sampling for complex online problems. In International Conference on Machine Learning , pages 100-108. PMLR, 2014.

D. Guo, S. I. Ktena, P. K. Myana, F. Huszar, W. Shi, A. Tejani, M. Kneier, and S. Das. Deep bayesian bandits: Exploring in online personalized recommendations. In Fourteenth ACM Conference on Recommender Systems , pages 456-461, 2020.

B. Hao, Y. Abbasi Yadkori, Z. Wen, and G. Cheng. Bootstrapping upper confidence bound. Advances in Neural Information Processing Systems , 32:12123-12133, 2019.

J. Hernandez-Lobato, Y. Li, M. Rowland, T. Bui, D. Hernández-Lobato, and R. Turner. Blackbox alpha divergence minimization. In International Conference on Machine Learning , pages 1511-1520. PMLR, 2016.

E. Kaufmann. On bayesian index policies for sequential resource allocation. The Annals of Statistics , 46(2):842-865, 2018.

E. Kaufmann, O. Cappé, and A. Garivier. On bayesian upper confidence bounds for bandit problems. In Artificial Intelligence and Statistics , pages 592-600. PMLR, 2012.

E. Kaufmann, N. Korda, and R. Munos. Thompson sampling: An asymptotically optimal finitetime analysis. In International Conference on Algorithmic Learning Theory , pages 199-213. Springer, 2012.

D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

N. Korda, B. Szorenyi, and S. Li. Distributed clustering of linear bandits in peer to peer networks. In International Conference on Machine Learning , pages 1301-1309. PMLR, 2016.

T. L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics , 6(1):4-22, 1985.

Y. Li and R. E. Turner. Rényi divergence variational inference. Advances in Neural Information Processing Systems , 29, 2016.

Y. Li, Y. Wang, and Y. Zhou. Nearly minimax-optimal regret for linearly parameterized bandits. In Conference on Learning Theory , pages 2173-2174. PMLR, 2019.

X. Lu and B. Van Roy. Ensemble sampling. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 3260-3268, 2017.

K. Mahadik, Q. Wu, S. Li, and A. Sabne. Fast distributed bandits for online recommendation systems. In Proceedings of the 34th ACM International Conference on Supercomputing , pages 1-13, 2020.

E. Mazumdar, A. Pacchiano, Y. Ma, M. Jordan, and P. Bartlett. On approximate thompson sampling with langevin algorithms. In International Conference on Machine Learning , pages 6797-6807. PMLR, 2020.

T. Minka et al. Divergence measures and message passing. Technical report, Citeseer, 2005.

B. O'Donoghue and T. Lattimore. Variational bayesian optimistic sampling. Advances in Neural Information Processing Systems , 34, 2021.

I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. Advances in Neural Information Processing Systems , 29:4026-4034, 2016.

M. Phan, Y. Abbasi-Yadkori, and J. Domke. Thompson sampling with approximate inference. arXiv preprint arXiv:1908.04970 , 2019.

A. Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics , volume 4, pages 547-562. University of California Press, 1961.

C. Riquelme, G. Tucker, and J. Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. In International Conference on Learning Representations , 2018.

H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58(5):527-535, 1952.

F. Ruiz and M. Titsias. A contrastive divergence for combining variational inference and mcmc. In International Conference on Machine Learning , pages 5537-5545. PMLR, 2019.

D. Russo and B. Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research , 39(4):1221-1243, 2014.

Y. Seldin, C. Szepesvári, P. Auer, and Y. Abbasi-Yadkori. Evaluation and analysis of the performance of the exp3 algorithm in stochastic environments. In European Workshop on Reinforcement Learning , pages 103-116. PMLR, 2013.

J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, M. Prabhat, and R. Adams. Scalable bayesian optimization using deep neural networks. In International Conference on Machine Learning , pages 2171-2180. PMLR, 2015.

N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995 , 2009.

B. Szorenyi, R. Busa-Fekete, I. Hegedus, R. Ormándi, M. Jelasity, and B. Kégl. Gossip-based distributed stochastic bandit algorithms. In International Conference on Machine Learning , pages 19-27. PMLR, 2013.

W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3/4):285-294, 1933.

I. Urteaga and C. Wiggins. Variational inference for the multi-armed contextual bandit. In International Conference on Artificial Intelligence and Statistics , pages 698-706. PMLR, 2018.

N. Wan, D. Li, and N. Hovakimyan. f-divergence variational inference. Advances in Neural Information Processing Systems , 33:17370-17379, 2020.

D. Wang, H. Liu, and Q. Liu. Variational inference with tail-adaptive f-divergence. Advances in Neural Information Processing Systems , 31, 2018.

P. Xu, H. Zheng, E. V. Mazumdar, K. Azizzadenesheli, and A. Anandkumar. Langevin monte carlo for contextual bandits. In International Conference on Machine Learning , pages 2483024850. PMLR, 2022.

Y. Yang, D. Pati, and A. Bhattacharya. α -variational inference with statistical guarantees. The Annals of Statistics , 48(2):886 - 905, 2020.

W. Zhang, D. Zhou, L. Li, and Q. Gu. Neural thompson sampling. In International Conference on Learning Representations , 2021.

D. Zhou, L. Li, and Q. Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning , pages 11492-11502. PMLR, 2020.

H. Zhu and R. Rohwer. Information geometric measurements of generalisation. Technical Report. Aston University, Birmingham, UK , 1995.

Appendices

We provide further results and discussions in this appendix. Section A presents detailed discussions on Assumption 3.3. Section B presents proofs for the results in Section 3.2 in the main paper. Section C presents proofs for the results in Section 3.3. Section D presents proofs for the results in Section 3.4. Section E presents additional experimental results.

Appendix A Remarks on Assumption 3.3

Remark A.1 (Constant Threshold) . A constant threshold assumption on inference error appears in [34]. In this study, we adopt a similar assumption as [34], since no standard assumptions are in place due to the infancy of this field. In practice, keeping inference error below a constant threshold is not only feasible but also can be improved. As a concrete example, [30] showed that the inference error of an efficient Langevin MCMC algorithm decreases at the order of O (1 / √ N j ( t )) . [48] also establish similar consistency results. On a high level, as more data are collected, the approximate and exact posteriors both become more concentrated, but they could concentrate to the same point of true mean rewards, as [30] showed. In addition to MCMC algorithms [4] which typically produce a consistent posterior, our assumption also fits other Bayesian inference algorithms, such as variational inference [8], that output approximate posteriors with a systematic computational bias.

In fact, the negative results in the presence of constant inference error shown by [34] motivate another research direction: A natural idea for obtaining positive results is to construct a 'highly-effective" Bayesian inference approach with vanishing inference error as the number of samples increases, as studied in [30, 48]. However, our results demonstrate that the O (log T ) regret upper bound is achievable even with non-vanishing inference error, without requiring specific 'highly-effective' Bayesian inference approaches.

Remark A.2 ( α -divergence) . Studies beyond KL divergence [38, 9], using α -divergence [26, 19, 14, 49] or more generally f -divergence [46, 47], have appeared in recent research. These studies demonstrated the potential use of general α -divergence and other types of divergence in practical model design. Moreover, from a practical perspective, the two α -divergences between the estimated and the exact posterior distributions could be small, even if only one α -divergence is used to construct the approximate distribution. This might be the reason for the superior performance of approximate Bayesian bandit algorithms that consider only the KL divergence in practice.

Remark A.3 (Independent Prior) . In multi-armed bandit problems, the prior distribution of mean rewards Π 0 is typically chosen to be independent among each action Π 0 , 1 , ..., Π 0 ,K , as this is the most reasonable and natural way without further information [1, 21, 20, 30]. In this case, the posteriors must also be independent among each action because of the Bayesian update (2). Therefore, our study focuses on the inference error on the distribution of each action in (3), which appears to be more realistic than the inference error on the joint distribution of all actions assumed in [34].

Appendix B Proofs of Results in Section 3.2

̸

Proof of Theorem 3.4. Note that R 1 ( γ ) = R 2 ( γ + δ γ,ϵ ) implies R -1 2 ( R 1 ( γ )) = γ + δ γ,ϵ and R -1 1 ( R 2 ( γ + δ γ,ϵ )) = γ . Obviously R -1 2 ( R 1 (0)) = 0 and R -1 2 ( R 1 (1)) = 1 . On a high level, our proof technique is to split the D α ( P 1 , P 2 ) into two parts using the quantile-based representation of α -divergence in Lemma 3.5, and then use Jensen's inequality (since 1 α ( α -1) x 1 -α is a convex function of x ≥ 0 for any α = 0 , 1 ) to derive a bound for δ γ,ϵ .

By Lemma 3.5 and Jensen's inequality,

First, we note that for any γ ∈ (0 , 1) and any δ γ,ϵ ∈ [ -γ, 1 -γ ] , the above inequality is indeed achievable. To see this, consider

Simple calculations give

where F 1 is the cdf of P 1 , and

Therefore, the P 1 and P 2 constructed in Equation (6) show that the inequality (5) is indeed achievable.

Next, we consider the function

Taking the derivative of g ( δ ) with respect to δ , we obtain

It is easy to see that g ( δ ) is strictly decreasing when δ ∈ ( -γ, 0) and strictly increasing when δ ∈ (0 , 1 -γ ) with minimum g (0) = 0 . This fact shows that ϵ ≥ g ( δ ) is equivalent to δ in an interval around 0, i.e., δ ∈ [ δ, δ ] ⊂ [ -γ, 1 -γ ] . More explicitly, we can obtain an explicit bound for δ, δ . In the following, we will show that in some cases, the bound for δ becomes too loose in the sense that δ can be close to -γ or δ can be close to 1 -γ , in which cases we cannot obtain any useful information.

1) Suppose that α > 1 . When δ γ,ϵ > 0 , we have

as 0 < δ γ,ϵ ≤ 1 -γ . Therefore ϵ ≥ D α ( P 1 , P 2 ) implies that

which is equivalent to

Hence we have

Remark. Note that when α > 1 , the lower bound of δ γ,ϵ can be similarly derived but it becomes vicious. To see this, we notice that whenever

we have that

as -γ ≤ δ γ,ϵ < 0 and

which ensures ϵ ≥ D α ( P 1 , P 2 ) . However, the lower bound in Equation (7) is too loose:

which can be sufficiently close to -γ if ϵ is sufficiently large.

b) Suppose that α < 0 . When δ γ,ϵ < 0 , we have

as -γ ≤ δ γ,ϵ < 0 . Therefore ϵ ≥ D α ( P 1 , P 2 ) implies that

which is equivalent to

Hence we have

Remark. Note that when α < 0 , the upper bound of δ γ,ϵ can be similarly derived but it becomes vicious. To see this, we notice that whenever

we have that

as -γ ≤ δ γ,ϵ < 0 and

which ensures ϵ ≥ D α ( P 1 , P 2 ) . However, the lower bound in Equation (9) is too loose:

so the lower bound in Equation (9) can be 1 -γ if ϵ is large.

c) Note that when α ∈ (0 , 1) , α ( α -1) < 0 and thus we have

and

as -γ ≤ δ γ,ϵ ≤ 1 -γ . Hence, as long as ϵ ≥ -1 α ( α -1) , we always have

for any δ γ,ϵ ∈ [ -γ, 1 -γ ] . Moreover, as we discussed below Equation (5), for any δ γ,ϵ ∈ [ -γ, 1 -γ ] , there exist two distributions P 1 and P 2 such that

which shows that D α ( P 1 , P 2 ) ≤ ϵ holds for such P 1 and P 2 . This implies that the condition D α ( P 1 , P 2 ) ≤ ϵ cannot control the quantile shift between P 1 and P 2 in general when α ∈ (0 , 1) .

Proof of Lemma 3.5. Let F 1 and F 2 be the cdfs of P 1 and P 2 respectively. Since F 1 and F 2 are absolute continuous and strictly increasing (as they have positive densities), we have that F i ( R i ( u )) = u for 0 ≤ u ≤ 1 . Taking the derivative with respect to both sides of F i ( R i ( u )) = u , we obtain

where R ′ i ( u ) := dR i ( u ) du . Note that we have the following equality:

Using integration by substitution, for α -divergence, we obtain that

Similarly, for KL divergence, we obtain that

Appendix C Proofs of Results in Section 3.3

We first prove two useful lemmas.

LemmaC.1. Under the same condition in Theorem 3.7, the quantiles of the approximate distributions qu j ( t ) chosen by the EBUCB algorithm satisfies the following bound:

where

corresponds to Q t,j -1 ), we have

since D α 1 ( Q t,j -1 , Π t,j -1 ) ≤ ϵ and we have use the fact that Qu is non-decreasing. Now we apply the proof of Lemma 1 in [21], the tight bounds of the quantiles of the Beta distributions, to obtain

Similarly, by Theorem 3.4 part b) (where P 1 corresponds to Q t,j -1 ), we have that

since D α 2 (Π t,j -1 , Q t,j -1 ) ≤ ϵ and we have use the fact that Qu is non-decreasing. Now we apply the proof of Lemma 1 in [21], the tight bounds of the quantiles of the Beta distributions, to obtain

Therefore, we conclude that

Based on Lemma C.1, we can obtain a UCB-type decomposition of the number of draws of any sub-optimal action j ≥ 2 as follows.

Lemma C.2. Under the same condition in Theorem 3.7, we have that for any constant β T ,

Proof of Lemma C.2. We have that, by definition,

where the last inequality follows from the fact that q 1 ( t ) ≥ u 1 ( t ) and when A t = 2 , q 1 ( t ) ≤ q 2 ( t ) ≤ u 2 ( t ) .

Therefore, to obtain Theorem 3.7, it is sufficient to analyze the two terms in Lemma C.2.

Proof of Theorem 3.7. Without loss of generality, we let j = 2 . (Note that we have assumed the action 1 is optimal.) By Lemma C.2, we only need to bound the two following two terms:

Let β T = √ 1 log T . We further split D 1 into two parts:

Step 1 : Consider D 1 , 1 .

Note that

in u 1 ( t ) when N 1 ( t ) + 2 ≤ (log T ) 2 . Hence we have that

when N 1 ( t ) + 2 ≤ (log T ) 2 . This shows that

Similarly to the proof in [21], with a straightforward adaptation of the proof of theorem 10 in [15], we obtain the following self-normalized inequality

Lemma C.3.

where

Lemma C.3 leads to the following upper bound of D 1 , 1 :

for c ˜ α 2 ≥ 5 .

Step 2 : Consider D 1 , 2 . Note that in this term, the optimal action 1 has been sufficiently drawn to be well estimated, so we can use a loose bound

This right-hand side only depends on the draws from action 1 and has been studied in Theorem 1 in [21], so we apply their results:

Step 3 : Consider D 2 . Using the same technique as in lemma 7 in [15], D 2 is bounded by

For ξ > 0 , we let

Then D 2 can be rewritten as

where the last inequality follows from the same technique in the proof of Theorem 1 in [21], by noting that the function g ( q ) = d + (ˆ µ 2 ( s ) , q ) is convex and differentiable and g ′ ( q ) = q -ˆ µ 2 ( s ) q (1 -q ) ✶ { ( q > ˆ µ 2 ( s ) } . Hence, for T ≥ exp ( ( 2(1+ ξ )(1+ ξ/ 2) ξµ 1 (1 -µ 1 ) d ( µ 2 ,µ 1 ) ) 2 ) , we obtain d ( µ 2 ,µ 1 ) 1+ ξ + β T 2 µ 1 (1 -µ ) ≤ d ( µ 2 ,µ 1 ) 1+ ξ/ 2 . Following the proof of Theorem 1 in [21] (as well as [15]), we obtain

Step 4 : Combing the above results, we obtain that if 0 < ζ ˜ α 2 < 1 ,

and if ζ ˜ α 2 ≥ 1 ,

Note that the final step (Step 4) in the above proof provides an exact finite-time regret bound that holds for any time horizon T . It explicitly expresses the o ( · ) term in Theorem 3.7. The error terms M -1 ϵ, 1 and M ϵ, 2 depending on ϵ appear in this regret bound explicitly. Obviously, M -1 ϵ, 1 and M ϵ, 2 in the bound increase as ϵ increases.

In general, our above derivations depend on bounds for specific distributions (Beta posterior distributions with inference errors in our setting). It is a direction to generalize these results to more general bandit problems or more general families of distributions. For instance, [20] extends the setting of Beta posterior distribution to the exponential family that includes Gaussian (without approximate inference). Combining [20] with our techniques in Sections 3.2 and 3.3 may lead to analyzing the exponential family with approximate inference. This, however, requires some additional careful technical derivation beyond our current bounds in Bernoulli with approximate inference, which is a future research direction.

Appendix D Proofs of Results in Section 3.4

Proof of Theorem 3.12. We can explicitly construct such a distribution Q t as follows:

where F t, 1 is the cumulative distribution function (cdf) of Π t, 1 and r > 1 , b t ∈ (0 , 1) will be specified later.

First, note that by setting

When α = 0 :

we have D α (Π t, 2 , Q t, 2 ) = 0 and q t, 2 = π t, 2 with the same support [0 , 1] , satisfying Assumptions 3.6 and 3.11 on action j = 2 .

We set b t = Qu ( 1 2 , Π t, 2 ) ∈ (0 , 1) , the 1 2 -quantile of the distribution Π t, 2 (or equivalently, Q t, 2 ). Let F t, 1 be the cdf of Π t, 1 . We have F t, 1 ( b t ) ∈ (0 , 1) . For r > 1 , we set

Step 1: We show that q t, 1 ( x 1 ) is indeed a density satisfying Assumption 3.6 on action j = 1 . First of all, it is obvious that q t, 1 > 0 on (0 , 1) as π t, 1 > 0 on (0 , 1) . Moreover

Step 2: We show that there exists an r > 1 (independent of t ) such that q t, 1 satisfies Assumption 3.11 on action j = 1 .

We have that when α < 0 or 0 < α < 1 :

We note that as r > 1 . Hence we have

Then for 1 < r < ( ϵα ( α -1) + 1) -1 α (only if ϵα ( α -1) + 1 > 0 , otherwise we put + ∞ as the upper bound on r ), we have that

We note that as r > 1 . Hence we have

Then for 1 < r < e ϵ , we have that

Step 3: We show that the probability of sampling from Q t -1 choosing action 2 is greater than a positive constant 1 2 (1 -1 r ) , which thus leads to a linear regret.

In fact, the probability of sampling from Q t -1 choosing action 2 is given by P Q t -1 ( x 2 ≥ x 1 ) . Therefore we have that

since Q t -1 , 1 and Q t -1 , 2 are independent.

since b t -1 is the 1 2 -quantile of the distribution Π t -1 , 2 and p t -1 , 2 > 0 on (0 , 1) .

by our construction of Q t -1 , 1 . Therefore we have that

We conclude that the lower bound of the average expected regret is given by

leading to a linear regret.

Proof of Theorem 3.13. We can explicitly construct such a distribution Q t as follows:

where F t, 2 is the cdf of Π t, 2 and r > 1 , b t ∈ (0 , 1) will be specified later. First, note that by setting

we have D α (Π t, 1 , Q t, 1 ) = 0 and q t, 1 = π t, 1 with the same support [0 , 1] , satisfying Assumptions 3.6 and 3.11 on action j = 1 .

We set b t = Qu ( γ t +1 , Π t, 1 ) ∈ (0 , 1) , the γ t +1 -quantile of the distribution Π t, 1 (or equivalently, Q t, 1 ). Let F t, 2 be the cdf of Π t, 2 . We have F t, 2 ( b t ) ∈ (0 , 1) . For r > 1 , we set

Step 1: We show that q t, 2 ( x 2 ) is indeed a density satisfying Assumption 3.6 on action j = 2 . First of all, it is obvious that q t, 2 > 0 on (0 , 1) as π t, 2 > 0 on (0 , 1) . Moreover

Step 2: We show that there exists an r > 1 (independent of t ) such that q t, 2 satisfies Assumption 3.11 on action j = 2 .

We have that when α < 0 or 0 < α < 1 :

We note that as r > 1 . Hence we have

Then for 1 < r < ( ϵα ( α -1) + 1) -1 α (only if ϵα ( α -1) + 1 > 0 , otherwise we put + ∞ as the upper bound on r ), we have that

When α = 0 ,

We note that as r > 1 . Hence we have

Then for 1 < r < e ϵ , we have that

Therefore, we conclude that there exists an r > 1 (independent of t ) such that q t, 2 satisfies Assumption 3.11 on action j = 2 . Take this r > 1 and notice that since γ t → 1 as t → + ∞ , there must exists a T 0 > 0 such that for any t ≥ T 0 , we have that γ t > 1 r .

Step 3: We show that the EBUCB algorithm always chooses action 2 when t ≥ T 0 , which thus leads to a linear regret.

We note that when t ≥ T 0 , by definition,

Therefore after time step t ≥ T 0 , the EBUCB algorithm will always choose the action 2 . We conclude that the lower bound of the average expected regret is given by

leading to a linear regret.

Appendix E Additional Experiments

In this section, we present additional experimental results. We enrich our experiments by studying an increasing number of arms as well as multiple new problem instances with different inference errors. These results further support our findings in Section 4 that EBUCB without the horizon-dependent term (i.e., c = 0 ) performs the best.

Suppose the posterior distributions are misspecified to the following distributions:

where j ∈ [ K ] . We conduct two experiments: 1) The Bernoulli multi-armed bandit problem has K actions with the following mean rewards:

K = 2 : mean rewards = [0.7, 0.3]

K = 4 : mean rewards = [0.9, 0.7, 0.5, 0.3]

K = 8 : mean rewards = [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2]

K = 16 : mean rewards = [0.9, 0.85, 0.8, 0.75, 0.7, 0.65, 0.6, 0.55, 0.5, 0.45, 0.4, 0.35, 0.3, 0.25, 0.2, 0.15]

Let Γ = 2 or 0 . 5 . Let w = 0 . 9 . The results are shown in Figure 3 below.

2) We also study different Γ values in the appropriate distribution that lead to different inference errors. Consider Γ = 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 2 , 5 , 10 , 15 in the experiments. Let K = 2 with mean rewards = [0.7, 0.3]. Let w = 0 . 9 .

The results are shown in Figure 4 below.

which implies that

= 0.5;K = 2

= 0.5; K = 4

30

14

12

25

10

20

8

15

9

10

4

2

5

0

0

0

200

400

600

800

1000

0

200

400

600

800

1000

= 0.5;K = 8

= 0.5;K = 16

120

40

100

30

80

60

20

40

10

20

0

0

0

200

400

600

800

1000

0

200

400

600

800

1000

= 2; K = 2

= 2; K = 4

30

10

25

20

15

10

2

5

0

0

0

200

400

600

800

1000

0

200

400

600

800

1000

= 2;K= 8

=2;K = 16

40

80

30

60

20

40

10

20

0

0

200

400

600

800

1000

0

200

400

600

800

1000

Go

= 0.05

= 0.1

60

40

50

40

30

30

20

20

10

10

0

0

200

400

600

800

1000

0

200

400

600

800

1000

= 0.2

= 0.5

25

14

12

20

10

15

8

10

6

4

2

0

0

200

400

600

800

1000

0

200

400

600

800

1000

= 2

=J

10

10

2

0

0

0

200

400

600

800

1000

0

200

400

600

800

1000

「 = 10

= 15

10

10

8

8

N

200

400

600

800

1000

0

200

400

600

800

1000

Peer reviews:
comments: SUMMARY:
In this paper, the authors consider Bayesian bandit algorithms where exact posterior is not available and
only approximations are available.
The authors prove that if $\alpha_1$-divergence and $\alpha_2$-divergence between the exact posterior and approximation are small,
then a modification of Bayesian UCB (EBUCB) achieves $O(\log T)$ regret.
In the previous work, it is known that Thompson sampling with approximate inference has $\Omega(T)$ regret.
The authors also show negative results for EBUCB and Thompson sampling under the condition on 
only one $\alpha$-divergence.
Finally, in synthetic environments the authors confirm their theoretical findings.

STRENGTHS:
1. This paper solves an important problem (Bayesian bandit algorithms using an approximation of the posterior) and provides a nice result (the upper bound has $O(\log T)$-regret and the dominant term does not depend on $\varepsilon$, which is surprising).
2. Although the environments are simple and synthetic, they conducted experiments.

WEAKNESSES:
1. In Corollary 3.8, the dependence of $\varepsilon$ is hidden (not discussed).
2. The optimal parameters of the algorithm are not known in practice.

QUESTIONS:
1. In Corollary 3.8, does the small $o$ notation hides $\varepsilon$? Could you briefly discuss its dependence?

LIMITATIONS:
Limitations are not discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This paper studies Bayesian bandits with approximate inference errors. The authors proposed an algorithm called the Enhanced Bayesian Upper Confidence Bound (EBUCB). Under a two-bounded $\alpha$-divergence assumption, the authors show that EBUCB can achieve the optimal logarithmic regret. The authors also show that with sub-linear regret cannot be achieved with only one-bounded $\alpha$-divergence.

STRENGTHS:
The authors provide the first $\log(T)$-type of regret for Bayesian bandits with constant approximation error (under 2-bounded $\alpha$-approximation). This result is obtained based on a novel sensitivity analysis of quantile shift. Both the result and the analysis look interesting to me. The authors also show that, under 1-bounded $\alpha$-approximation, one cannot obtain sub-linear regret; this negative result further justifies the necessity of 2-bounded $\alpha$-approximation (Assumption 3.3).

WEAKNESSES:
My main concern is that the developed results are only for Bernoulli bandits, not for more general distributions (not even for the Gaussian distribution). Can authors comment on why Bernoulli is needed in the current analysis? Or what prevents the derived results from being extended to other distributions?

Another question I have is regarding the constant approximation error for the 2-bounded $\alpha$-approximation (i.e., the $\epsilon$ term in Assumption 3.3). It seems a bit weird to me that the regret bound doesn't depend (too much) on the $\epsilon$ (e.g., in Corollary 3.8). Can authors comment on the reasons behind this?

QUESTIONS:
See the weaknesses part.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the standard multi armed bandit problem with a prior on rewards, allowing the design of Bayesian algorithms such as Thompson sampling and Bayesian UCB. The problem of interest is when the exact posterior distributions are not available. Rather an approximate posterior is available. It has been known that even with a small constant alpha divergence error between the true and approximate posterior, Thompson sampling does not converge. This paper shows that with a two bounded alpha divergence (see Assumption 3.3) Bayesian UCB achives order optimal regret bound.\n\nSTRENGTHS:\nThe problem of designing Bayesian optimization algorithms which find the optimal action in the presence of approximate distributions is very interesting.\n\nWEAKNESSES:\nThe setting is motivated with the difficulty of obtaining true distributions that often arises in complex models and when using methods such as variationally inference. The results are however proven on a system of Bernoulli distributions where the posterior is available in closed form. I think this break the logic on motivation to a great extent. \n\nWhile I tried to read all the proofs and details, I could not obtain a good intuition into why small alpha divergence is not enough and two alpha divergence works; how can this be used when applied to for example MCMC or variational inference.\n\nQUESTIONS:\n1. Could authors provide more intuition into the significance of their results for the complex settings where the distributions are approximated. It seems the results are limited to simple settings where the posterior is easily obtained. Could authors intuitively explain why small alpha divergence is not enough and two alpha divergence works. \n\n2. As small alpha divergence is not enough and only two alpha divergence works in this setting, one would expect Theorem 3.7 to fail when $\\alpha_1=\\alpha_2$. In line 254, it is stated that we may choose $\\zeta=\\frac{1}{\\tilde{\\alpha}_2}$. If, in addition, we set $\\alpha_1=\\alpha_2$, implying $\\tilde{\\alpha}_1=\\tilde{\\alpha}_2$, the upper bound in Theorem 3.7 still seems to work. Could authors explain what happens here when $\\alpha_1=\\alpha_2$ and if the Theorem fails.\n\nLIMITATIONS:\nAs mentioned above, the main limitation of the paper seems to be the simple setting, where the posteriors are available in closed form. The results do not seem to be extendable to more complex setting where approximate distributions are actually relevant.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}