Variational Bayesian Optimistic Sampling

√ AT ) Bayesian regret for a problem with A actions after T rounds . We extend the regret analysis for optimistic policies to bilinear saddle-point problems which include zero-sum matrix games and constrained bandits as special cases . In this case we show that Thompson sampling can produce policies outside of the optimistic set and suffer linear regret in some instances . Finding a policy inside the optimistic set amounts to solving a convex optimization problem and we call the resulting algorithm ‘ variational Bayesian optimistic sampling ’ ( VBOS ) . The procedure works for any posteriors , i.e. , it does not require the posterior to have any special properties , such as log-concavity , unimodality , or smoothness . The variational view of the problem has many useful properties , including the ability to tune the exploration-exploitation tradeoff , add regularization , incorporate constraints , and linearly parameterize the policy . 1 Introduction . In this manuscript we consider online learning , in particular the multi-armed stochastic bandit problem [ 19 ] and its extension to bilinear saddle-point problems . Thompson sampling ( TS ) is a wellknown Bayesian algorithm to tackle this problem [ 37 , 35 ] . At each iteration TS samples an action according to the posterior probability that it is the optimal action . However , it never computes these probabilities , instead TS samples from the posterior of each arm and then acts greedily with respect to that sample , which is an implicit sample from the posterior probability of optimality . TS enjoys strong theoretical guarantees [ 1 , 35 ] as well as excellent practical performance [ 6 ] . Moreover , it has an elegant information-theoretic interpretation of an agent that is making a tradeoff between information accumulation and regret [ 34 ] . That being said it does have several weaknesses . Firstly , TS requires samples from the true posterior , which may be intractable for some problems . It is known that TS with even small error in its posterior samples can suffer linear regret [ 32 ] . In many cases of interest the posterior distribution is highly intractable to sample from and running MCMC chains , one for each arm , can be wasteful and slow and still not yield precise enough samples for good performance , though recent work has attempted to improve this situation [ 22 ] . Moreover , TS is unable to take into account other preferences , such as constraints or the presence of other agents . In particular , it was recently shown that TS can suffer linear regret in two-player zero sum games [ 27 ] and we shall demonstrate the same phenomenon in this work empirically for constrained bandits . Finally , when performing TS the user never has direct access to the posterior probability of optimality , only samples from that distribution ( though the distribution can be estimated by taking many samples ) . In some practical applications having access to the policy is desirable , for example to ensure safety constraints or to allocate budgets . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper we present ‘ variational Bayesian optimistic sampling ’ ( VBOS ) , a new Bayesian approach to online learning . At every step the VBOS algorithm solves a convex optimization problem over the simplex . The solution to this problem is a policy that satisfies a particular ‘ optimism ’ condition . The problem is always convex , no matter what form the posteriors take . The only requirement is the ability to compute or estimate the cumulant generating function of the posteriors . VBOS and TS can be considered as part of the same family of algorithms since , in the bandit case , they both produce policies inside the optimistic set . That being said , VBOS has several advantages over TS . First , in some cases estimating or bounding the cumulant generating function may be easier than sampling from the posterior . Second , incorporating additional requirements such as constraints , opponents or tuning the exploration exploitation tradeoff is easy in VBOS - just change the optimization problem . Finally , when using VBOS we always have direct access to the policy , rather than just implicit samples from it , as well as an explicit upper bound on the expected value of the problem , which TS does not give us . Here we list the contributions of this work . 1.1 Contributions . 1 . We derive a set of ‘ optimistic ’ Bayesian policies for the stochastic multi-armed bandit that satisfy a particular optimism inequality and which includes the TS policy . 2 . We present a new , simple proof of a Õ ( √ AT ) regret-bound for a bandit with A arms after T timesteps that covers any policy in the optimistic set ( Õ ignores logarithmic factors ) . 3 . We derive variational Bayesian optimistic sampling ( VBOS ) , which is a policy in the optimistic set that can be computed by solving a convex optimization problem . 4 . We show that , unlike TS , the variational formulation easily handles more complicated problems , such as bilinear saddle-point problems which include as special cases zero-sum two player games and constrained bandits . 1.2 Preliminaries . A stochastic multi-armed bandit problem is a sequential learning problem in which an agent interacts with an environment in order to maximize its total cumulative reward . Initially the agent does not know how the rewards are distributed and must learn about them from experience . In each round t ∈ N the agent selects an action at ∈ { 1 , . . . , A } and receives a reward rt = µat + ηtat where η t at ∈ R is zero-mean noise associated with action at at time t. We define Ft = ( a1 , r1 , . . . , at−1 , rt−1 ) to be the sequence of actions and rewards observed by the agent prior to round t , and as shorthand we shall use the notation Et ( · ) = E ( · | Ft ) . An algorithm alg is a measurable mapping that takes the entire history at time t and produces a probability distribution , the policy , from which the agent samples its action . In order to assess the quality of an algorithm alg we consider the regret , or shortfall in cumulative rewards relative to the optimal value , Regret ( µ , alg , T ) = Eη , alg [ T∑ t=1 max i µi − rt ] , ( 1 ) where Eη , alg denotes the expectation with respect to the noise process η and any randomness in the algorithm . This quantity ( 1 ) depends on the unknown expected reward µ ∈ RA , which is fixed at the start of play and kept the same throughout . To assess the quality of learning algorithms designed to work across some familyM of problems we define BayesRegret ( φ , alg , T ) = Eµ∼φRegret ( µ , alg , T ) ( 2 ) where φ is a prior probability measure over µ ∈M that assigns relative importance to each problem instance and which we assume is known to the algorithm . 2 Thompson sampling and a variational approach . The algorithm that minimizes the Bayesian regret ( 2 ) is called the Bayes-optimal policy , but it is believed to be intractable to compute in most cases [ 10 ] . With that in mind we would like an algorithm that is computationally tractable and still guarantees a good bound on the Bayesian regret . TS is a Bayesian algorithm that at each time step samples possible values of each arm from their respective posteriors and acts greedily with respect to those samples ; see Algorithm 1 . This simple strategy provides a principled approach to the exploration-exploitation tradeoff in the stochastic multi-armed bandit problem . In this section we derive a set of ‘ optimistic ’ policies , which includes the TS policy , and we shall show that any algorithm that produces policies in this set satisfies a guaranteed Bayesian regret bound . For brevity we defer all proofs to the appendices . 2.1 Bounding the conditional expectation . The Bayesian regret ( 2 ) depends on the expectation of the maximum over a collection of random variables . Here we shall derive an upper bound on this quantity , starting with a generic upper bound on the conditional expectation . If X : Ω → Rd is a random variable in Ld1 ( i.e. , E|Xi| < ∞ for each i = 1 , . . . , d ) then we denote by ΨX : Rd → R ∪ { ∞ } the cumulant generating function of X − EX , i.e. , ΨX ( β ) = logE exp ( β > ( X − EX ) ) . This function is always convex , and we shall assume throughout this manuscript that it is also closed and proper for any random variables we encounter . With this definition we can present a bound on the conditional expectation . Lemma 1 . Let X : Ω → R be a random variable on ( Ω , F , P ) satisfying X ∈ L1 , and let A ∈ F be an event with P ( A ) > 0 . Then , for any τ ≥ 0 E [ X|A ] ≤ EX + τΨX ( 1/τ ) − τ logP ( A ) . ( 3 ) The right hand side of ( 3 ) involves the perspective of ΨX and so is convex in τ [ 4 ] and we can minimize over τ ≥ 0 to find the tightest bound , which we do next . Recall that the convex conjugate of function f : Rd → R is denoted f∗ : Rd → R and is given by f∗ ( y ) = supx ( x > y − f ( x ) ) [ 4 ] . Theorem 1 . Let X : Ω → R be a random variable such that the interior of the domain of ΨX is non-empty , then under the same assumptions as Lemma 1 we have 1 E [ X|A ] ≤ EX + ( Ψ∗X ) −1 ( − logP ( A ) ) . For example , if X ∼ N ( 0 , σ2 ) , then for any event A we can bound the conditional expectation as E [ X|A ] ≤ σ √ −2 logP ( A ) , no matter how X and A are related . This inequality has clear connections to Kullback ’ s inequality [ 8 ] and the Donsker-Varadhan variational representation of KL-divergence [ 11 , Thm . 3.2 ] . The function Ψ∗ is referred to as the Cramér function or the rate function in large deviations theory [ 39 ] . The inverse of the rate function comes up in several contexts , such as queuing theory and calculating optimal insurance premiums [ 21 ] . Cramér ’ s theorem tells us that the probability of a large deviation of the empirical average of a collection of IID random variables decays exponentially with more data , and that the decay constant is the rate function . It is no surprise then that the regret bound we derive will depend on the inverse of the rate function , i.e. , when the tails decay faster we incur less regret . 2.2 The optimistic set . Using Theorem 1 we next derive the maximal inequality we shall use to bound the regret . Lemma 2 . Let µ : Ω → RA , µ ∈ LA1 , be a random variable , let i ? = argmaxi µi ( ties broken arbitrarily ) and denote by Ψi : = Ψµi , then Emax i µi ≤ A∑ i=1 P ( i ? = i ) ( Eµi + ( Ψ∗i ) −1 ( − logP ( i ? = i ) ) ) . Note that Lemma 2 makes no assumption of independence of each entry of µ , the bound holds even in the case of dependency . 1If ΨX = 0 , then we take ( Ψ∗X ) −1 = 0 . In the context of online learning πTSi = P ( i ? = i ) is exactly the TS policy when µi have the laws of the posterior . For ease of notation we introduce the ‘ optimism ’ map Gtφ : ∆A → R which for a random variable µ : Ω→ RA distributed according to φ ( · | Ft ) is given by Gtφ ( π ) : = A∑ i=1 πi ( Etµi + ( Ψt∗i ) −1 ( − log πi ) ) , ( 4 ) where ∆A denotes the probability simplex of dimension A − 1 and Ψti : = Ψµi|Ft . Note that Gtφ is concave since Ψti is convex [ 4 ] . Moreover , if the random variables are non-degenerate then Gtφ is differentiable and the gradient can be computed using Danskin ’ s theorem [ 7 ] . With this notation we can write Et maxi µi ≤ Gtφ ( πTS ) . This brings us to the ‘ optimistic ’ set of probability distributions , which corresponds to the set of policies that satisfy that same inequality . Definition 1 . Let µ : Ω → RA be a random variable distributed according to φ ( · | Ft ) , then we define the optimistic set as Ptφ : = { π ∈ ∆A | Et max i µi ≤ Gtφ ( π ) } . Immediately we have two facts about the set Ptφ . First , it is non-empty since πTS ∈ Ptφ , and second , it is a convex set since the optimism map Gtφ is concave .

Summary: This work develops an analog to Thompson sampling by upper-bounding the expected regret in sequential decision-making problems. The two terms in the upper bound loosely resemble the evidence lower bound of variational inference: the first term encourages selecting arms with high expected reward; the second term depends on the inverse of the rate function and penalizes heavy tails and encourages exploration. The resulting variational Thompson sampling algorithm is evaluated on a random game and a constrained bandit problem.