Variational Bayesian Optimistic Sampling

√ AT ) Bayesian regret for a problem with A actions after T rounds . We extend the regret analysis for optimistic policies to bilinear saddle-point problems which include zero-sum matrix games and constrained bandits as special cases . In this case we show that Thompson sampling can produce policies outside of the optimistic set and suffer linear regret in some instances . Finding a policy inside the optimistic set amounts to solving a convex optimization problem and we call the resulting algorithm ‘ variational Bayesian optimistic sampling ’ ( VBOS ) . The procedure works for any posteriors , i.e. , it does not require the posterior to have any special properties , such as log-concavity , unimodality , or smoothness . The variational view of the problem has many useful properties , including the ability to tune the exploration-exploitation tradeoff , add regularization , incorporate constraints , and linearly parameterize the policy . 1 Introduction . In this manuscript we consider online learning , in particular the multi-armed stochastic bandit problem [ 19 ] and its extension to bilinear saddle-point problems . Thompson sampling ( TS ) is a wellknown Bayesian algorithm to tackle this problem [ 37 , 35 ] . At each iteration TS samples an action according to the posterior probability that it is the optimal action . However , it never computes these probabilities , instead TS samples from the posterior of each arm and then acts greedily with respect to that sample , which is an implicit sample from the posterior probability of optimality . TS enjoys strong theoretical guarantees [ 1 , 35 ] as well as excellent practical performance [ 6 ] . Moreover , it has an elegant information-theoretic interpretation of an agent that is making a tradeoff between information accumulation and regret [ 34 ] . That being said it does have several weaknesses . Firstly , TS requires samples from the true posterior , which may be intractable for some problems . It is known that TS with even small error in its posterior samples can suffer linear regret [ 32 ] . In many cases of interest the posterior distribution is highly intractable to sample from and running MCMC chains , one for each arm , can be wasteful and slow and still not yield precise enough samples for good performance , though recent work has attempted to improve this situation [ 22 ] . Moreover , TS is unable to take into account other preferences , such as constraints or the presence of other agents . In particular , it was recently shown that TS can suffer linear regret in two-player zero sum games [ 27 ] and we shall demonstrate the same phenomenon in this work empirically for constrained bandits . Finally , when performing TS the user never has direct access to the posterior probability of optimality , only samples from that distribution ( though the distribution can be estimated by taking many samples ) . In some practical applications having access to the policy is desirable , for example to ensure safety constraints or to allocate budgets . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper we present ‘ variational Bayesian optimistic sampling ’ ( VBOS ) , a new Bayesian approach to online learning . At every step the VBOS algorithm solves a convex optimization problem over the simplex . The solution to this problem is a policy that satisfies a particular ‘ optimism ’ condition . The problem is always convex , no matter what form the posteriors take . The only requirement is the ability to compute or estimate the cumulant generating function of the posteriors . VBOS and TS can be considered as part of the same family of algorithms since , in the bandit case , they both produce policies inside the optimistic set . That being said , VBOS has several advantages over TS . First , in some cases estimating or bounding the cumulant generating function may be easier than sampling from the posterior . Second , incorporating additional requirements such as constraints , opponents or tuning the exploration exploitation tradeoff is easy in VBOS - just change the optimization problem . Finally , when using VBOS we always have direct access to the policy , rather than just implicit samples from it , as well as an explicit upper bound on the expected value of the problem , which TS does not give us . Here we list the contributions of this work . 1.1 Contributions . 1 . We derive a set of ‘ optimistic ’ Bayesian policies for the stochastic multi-armed bandit that satisfy a particular optimism inequality and which includes the TS policy . 2 . We present a new , simple proof of a Õ ( √ AT ) regret-bound for a bandit with A arms after T timesteps that covers any policy in the optimistic set ( Õ ignores logarithmic factors ) . 3 . We derive variational Bayesian optimistic sampling ( VBOS ) , which is a policy in the optimistic set that can be computed by solving a convex optimization problem . 4 . We show that , unlike TS , the variational formulation easily handles more complicated problems , such as bilinear saddle-point problems which include as special cases zero-sum two player games and constrained bandits . 1.2 Preliminaries . A stochastic multi-armed bandit problem is a sequential learning problem in which an agent interacts with an environment in order to maximize its total cumulative reward . Initially the agent does not know how the rewards are distributed and must learn about them from experience . In each round t ∈ N the agent selects an action at ∈ { 1 , . . . , A } and receives a reward rt = µat + ηtat where η t at ∈ R is zero-mean noise associated with action at at time t. We define Ft = ( a1 , r1 , . . . , at−1 , rt−1 ) to be the sequence of actions and rewards observed by the agent prior to round t , and as shorthand we shall use the notation Et ( · ) = E ( · | Ft ) . An algorithm alg is a measurable mapping that takes the entire history at time t and produces a probability distribution , the policy , from which the agent samples its action . In order to assess the quality of an algorithm alg we consider the regret , or shortfall in cumulative rewards relative to the optimal value , Regret ( µ , alg , T ) = Eη , alg [ T∑ t=1 max i µi − rt ] , ( 1 ) where Eη , alg denotes the expectation with respect to the noise process η and any randomness in the algorithm . This quantity ( 1 ) depends on the unknown expected reward µ ∈ RA , which is fixed at the start of play and kept the same throughout . To assess the quality of learning algorithms designed to work across some familyM of problems we define BayesRegret ( φ , alg , T ) = Eµ∼φRegret ( µ , alg , T ) ( 2 ) where φ is a prior probability measure over µ ∈M that assigns relative importance to each problem instance and which we assume is known to the algorithm . 2 Thompson sampling and a variational approach . The algorithm that minimizes the Bayesian regret ( 2 ) is called the Bayes-optimal policy , but it is believed to be intractable to compute in most cases [ 10 ] . With that in mind we would like an algorithm that is computationally tractable and still guarantees a good bound on the Bayesian regret . TS is a Bayesian algorithm that at each time step samples possible values of each arm from their respective posteriors and acts greedily with respect to those samples ; see Algorithm 1 . This simple strategy provides a principled approach to the exploration-exploitation tradeoff in the stochastic multi-armed bandit problem . In this section we derive a set of ‘ optimistic ’ policies , which includes the TS policy , and we shall show that any algorithm that produces policies in this set satisfies a guaranteed Bayesian regret bound . For brevity we defer all proofs to the appendices . 2.1 Bounding the conditional expectation . The Bayesian regret ( 2 ) depends on the expectation of the maximum over a collection of random variables . Here we shall derive an upper bound on this quantity , starting with a generic upper bound on the conditional expectation . If X : Ω → Rd is a random variable in Ld1 ( i.e. , E|Xi| < ∞ for each i = 1 , . . . , d ) then we denote by ΨX : Rd → R ∪ { ∞ } the cumulant generating function of X − EX , i.e. , ΨX ( β ) = logE exp ( β > ( X − EX ) ) . This function is always convex , and we shall assume throughout this manuscript that it is also closed and proper for any random variables we encounter . With this definition we can present a bound on the conditional expectation . Lemma 1 . Let X : Ω → R be a random variable on ( Ω , F , P ) satisfying X ∈ L1 , and let A ∈ F be an event with P ( A ) > 0 . Then , for any τ ≥ 0 E [ X|A ] ≤ EX + τΨX ( 1/τ ) − τ logP ( A ) . ( 3 ) The right hand side of ( 3 ) involves the perspective of ΨX and so is convex in τ [ 4 ] and we can minimize over τ ≥ 0 to find the tightest bound , which we do next . Recall that the convex conjugate of function f : Rd → R is denoted f∗ : Rd → R and is given by f∗ ( y ) = supx ( x > y − f ( x ) ) [ 4 ] . Theorem 1 . Let X : Ω → R be a random variable such that the interior of the domain of ΨX is non-empty , then under the same assumptions as Lemma 1 we have 1 E [ X|A ] ≤ EX + ( Ψ∗X ) −1 ( − logP ( A ) ) . For example , if X ∼ N ( 0 , σ2 ) , then for any event A we can bound the conditional expectation as E [ X|A ] ≤ σ √ −2 logP ( A ) , no matter how X and A are related . This inequality has clear connections to Kullback ’ s inequality [ 8 ] and the Donsker-Varadhan variational representation of KL-divergence [ 11 , Thm . 3.2 ] . The function Ψ∗ is referred to as the Cramér function or the rate function in large deviations theory [ 39 ] . The inverse of the rate function comes up in several contexts , such as queuing theory and calculating optimal insurance premiums [ 21 ] . Cramér ’ s theorem tells us that the probability of a large deviation of the empirical average of a collection of IID random variables decays exponentially with more data , and that the decay constant is the rate function . It is no surprise then that the regret bound we derive will depend on the inverse of the rate function , i.e. , when the tails decay faster we incur less regret . 2.2 The optimistic set . Using Theorem 1 we next derive the maximal inequality we shall use to bound the regret . Lemma 2 . Let µ : Ω → RA , µ ∈ LA1 , be a random variable , let i ? = argmaxi µi ( ties broken arbitrarily ) and denote by Ψi : = Ψµi , then Emax i µi ≤ A∑ i=1 P ( i ? = i ) ( Eµi + ( Ψ∗i ) −1 ( − logP ( i ? = i ) ) ) . Note that Lemma 2 makes no assumption of independence of each entry of µ , the bound holds even in the case of dependency . 1If ΨX = 0 , then we take ( Ψ∗X ) −1 = 0 . In the context of online learning πTSi = P ( i ? = i ) is exactly the TS policy when µi have the laws of the posterior . For ease of notation we introduce the ‘ optimism ’ map Gtφ : ∆A → R which for a random variable µ : Ω→ RA distributed according to φ ( · | Ft ) is given by Gtφ ( π ) : = A∑ i=1 πi ( Etµi + ( Ψt∗i ) −1 ( − log πi ) ) , ( 4 ) where ∆A denotes the probability simplex of dimension A − 1 and Ψti : = Ψµi|Ft . Note that Gtφ is concave since Ψti is convex [ 4 ] . Moreover , if the random variables are non-degenerate then Gtφ is differentiable and the gradient can be computed using Danskin ’ s theorem [ 7 ] . With this notation we can write Et maxi µi ≤ Gtφ ( πTS ) . This brings us to the ‘ optimistic ’ set of probability distributions , which corresponds to the set of policies that satisfy that same inequality . Definition 1 . Let µ : Ω → RA be a random variable distributed according to φ ( · | Ft ) , then we define the optimistic set as Ptφ : = { π ∈ ∆A | Et max i µi ≤ Gtφ ( π ) } . Immediately we have two facts about the set Ptφ . First , it is non-empty since πTS ∈ Ptφ , and second , it is a convex set since the optimism map Gtφ is concave .

Summary: This paper presents and establishes regret bounds for a new heuristic, inspired by Thompson sampling, for selecting actions in stochastic multi-arm bandit and bilinear saddle-point problems. As in Thompson sampling, a belief state is maintained over the values of each arm. However, instead of selecting an action according to its probability of optimality, one instead computes a policy (distribution over actions) that maximizes a novel objective. The objective can be understood as combining a "pure exploitation" term (the expected reward of the policy, under the current belief distribution), and an "exploration" term that depends only on the relative concentrations (and not the expected values) of the posteriors for each arm.