No Regrets for Learning the Prior in Bandits

1 Introduction . We study the problem of maximizing the total reward , or minimizing the total regret , in a sequence of stochastic bandit instances [ 29 , 4 , 31 ] . We consider a Bayesian version of the problem , where the bandit instances are drawn from some distribution . More specifically , the learning agent interacts with m bandit instances in m tasks , with one instance per task . The interaction with each task is for n rounds and with K arms . The reward distribution of arm i ∈ [ K ] in task s ∈ [ m ] is pi ( · ; θs , ∗ ) , where θs , ∗ is a shared parameter of all arms in task s. When arm i is pulled in task s , the agent receives a random reward from pi ( · ; θs , ∗ ) . The parameters θ1 , ∗ , . . . , θm , ∗ are drawn independently of each other from a task prior P ( · ; µ∗ ) . The task prior is parameterized by an unknown meta-parameter µ∗ , which is drawn from a meta-prior Q . The agent does not know µ∗ or θ1 , ∗ , . . . , θm , ∗ . However , it knows Q and the parametric forms of all distributions , which help it to learn about µ∗ . This is a form of meta-learning [ 39 , 40 , 7 , 8 ] , where the agent learns to act from interactions with bandit instances . A simple approach is to ignore the hierarchical structure of the problem and solve each bandit task independently with some bandit algorithm , such as Thompson sampling ( TS ) [ 38 , 11 , 2 , 37 ] . This may be highly suboptimal . To illustrate this , imagine that arm 1 is optimal for any θ in the support of P ( · ; µ∗ ) . If µ∗ was known , any reasonable algorithm would only pull arm 1 and have zero regret over any horizon . Likewise , a clever algorithm that learns µ∗ should eventually pull arm 1 most of the time , and thus have diminishing regret as it interacts with a growing number of tasks . Two challenges arise when designing the clever algorithm . First , can it be computationally efficient ? Second , what is the regret due to adapting to µ∗ ? We make the following contributions . First , we propose a Thompson sampling algorithm for our problem , which we call AdaTS . AdaTS maintains a distribution over the meta-parameter µ∗ , which concentrates over time and is marginalized out when interacting with individual bandit instances . Second , we propose computationally-efficient implementations of AdaTS for multi-armed bandits [ 29 , 4 ] , linear bandits [ 14 , 1 ] , and combinatorial semi-bandits [ 19 , 13 , 26 ] . These implementations are for specific reward distributions and conjugate task priors . Third , we bound the n-round Bayes regret of AdaTS in linear bandits and semi-bandits , and multi-armed bandits as a special case . The Bayes regret is defined by taking an expectation over all random quantities , including µ∗ ∼ Q . Our bounds show that not knowing µ∗ has a minimal impact on the regret as the number of tasks grows , of only Õ ( √ mn ) . This is in a sharp contrast to prior work [ 28 ] , where this is Õ ( √ mn2 ) . Finally , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . our experiments show that AdaTS quickly adapts to the unknown meta-parameter µ∗ , is robust to meta-prior misspecification , and performs well even in challenging classification problems . We present a general framework for learning to explore from similar past exploration problems . One potential application is cold-start personalization in recommender systems where users are tasks . The users have similar preferences , but neither the individual preferences nor their similarity is known in advance . Another application could be online regression with bandit feedback ( Appendix E.2 ) where individual regression problems are tasks . Similar examples in the tasks have similar mean responses , which are unknown in advance . 2 Setting . We first introduce our notation . The set { 1 , . . . , n } is denoted by [ n ] . The indicator 1 { E } denotes that event E occurs . The i-th entry of vector v is vi . If the vector or its index are already subindexed , we write v ( i ) . We use Õ for the big-O notation up to polylogarithmic factors . A diagonal matrix with entries v is denoted diag ( v ) . We use the terms “ arm ” and “ action ” interchangeably , depending on the context . Our setting was proposed in Kveton et al . [ 28 ] and is defined as follows . Each bandit problem instance has K arms . Each arm i ∈ [ K ] is defined by distribution pi ( · ; θ ) with parameter θ ∈ Θ . The parameter θ is shared among all arms . The mean of pi ( · ; θ ) is denoted by r ( i ; θ ) . The learning agent interacts with m instances , one at each of m tasks . At the beginning of task s ∈ [ m ] , an instance θs , ∗ ∈ Θ is sampled i.i.d . from a task prior P ( · ; µ∗ ) , which is parameterized by µ∗ . The agent interacts with θs , ∗ for n rounds . In round t ∈ [ n ] , it pulls one arm and observes a stochastic realization of its reward . We denote the pulled arm in round t of task s by As , t ∈ [ K ] , the realized rewards of all arms in round t of task s by Ys , t ∈ RK , and the reward of arm i ∈ [ K ] by Ys , t ( i ) ∼ pi ( · ; θs , ∗ ) . We assume that the realized rewards Ys , t are i.i.d . with respect to both s and t. A graphical model of our environment is drawn in Figure 1 . We define the distribution-specific parameters µq , Σq , Σ0 , and σ2 when we instantiate our framework . Our terminology is summarized in Appendix A . The n-round regret of an agent or algorithm over m tasks with task prior P ( · ; µ∗ ) is defined as R ( m , n ; µ∗ ) = m∑ s=1 E [ n∑ t=1 r ( As , ∗ ; θs , ∗ ) − r ( As , t ; θs , ∗ ) ∣∣∣∣∣µ∗ ] , ( 1 ) where As , ∗ = arg max i∈ [ K ] r ( i ; θs , ∗ ) is the optimal arm in the random problem instance θs , ∗ in task s ∈ [ m ] . The above expectation is over problem instances θs , ∗ ∼ P ( · ; µ∗ ) , their realized rewards , and also pulled arms . Note that µ∗ is fixed . Russo and Van Roy [ 36 ] showed that the Bayes regret , which matches the definition in ( 1 ) in any task , of Thompson sampling in a K-armed bandit with n rounds is Õ ( √ Kn ) . So , when TS is applied independently in each task , R ( m , n ; µ∗ ) = Õ ( m √ Kn ) . Our goal is to attain a comparable regret without knowing µ∗ . We frame this problem in a Bayesian fashion , where µ∗ ∼ Q before the learning agent interacts with the first task . The agent knows Q and we call it a meta-prior . Accordingly , we consider R ( m , n ) = E [ R ( m , n ; µ∗ ) ] as a metric and call it the Bayes regret . Our approach is motivated by hierarchical Bayesian models [ 20 ] , where the uncertainty in prior parameters , such as µ∗ , is represented by another distribution , such as Q . In these models , Q is called a hyper-prior and µ∗ is called a hyper-parameter . We attempt to learn µ∗ from sequential interactions with instances θs , ∗ ∼ P ( · ; µ∗ ) , which are also unknown . The agent can only observe their noisy realizations Ys , t . 3 Algorithm . Our algorithm is presented in this section . To describe it , we need to introduce several notions of history , the past interactions of the agent . We denote by Hs = ( As , t , Ys , t ( As , t ) ) nt=1 the history in Algorithm 1 AdaTS : Instance-adaptive exploration in Thompson sampling . 1 : Initialize meta-prior Q0 ← Q 2 : for s = 1 , . . . , m do 3 : Compute meta-posterior Qs ( Proposition 1 ) 4 : Compute uncertainty-adjusted task prior Ps ( Proposition 1 ) 5 : for t = 1 , . . . , n do 6 : Compute posterior of θ in task s , Ps , t ( θ ) ∝ Ls , t ( θ ) Ps ( θ ) 7 : Sample θ̃s , t ∼ Ps , t , pull arm As , t ← arg max i∈ [ K ] r ( i ; θ̃s , t ) , and observe Ys , t ( As , t ) task s and by H1 : s = H1 ⊕ · · · ⊕Hs a concatenated vector of all histories in the first s tasks . The history up to round t in task s is Hs , t = ( As , ` , Ys , ` ( As , ` ) ) t−1 ` =1 and all history up to round t in task s is H1 : s , t = H1 : s−1 ⊕Hs , t . We denote the conditional probability distribution given history H1 : s , t by Ps , t ( · ) = P ( · |H1 : s , t ) and the corresponding conditional expectation by Es , t [ · ] = E [ · |H1 : s , t ] . Our algorithm is a form of Thompson sampling [ 38 , 11 , 2 , 37 ] . TS pulls arms proportionally to being optimal with respect to the posterior . In particular , let Ls , t ( θ ) = ∏t−1 ` =1 pAs , ` ( Ys , ` ( As , ` ) ; θ ) be the likelihood of observations in task s up to round t. If the prior P ( · ; µ∗ ) was known , the posterior of instance θ in round t would be P TSs , t ( θ ) ∝ Ls , t ( θ ) P ( θ ; µ∗ ) . TS would sample θ̃t ∼ P TSs , t and pull arm At = arg max i∈ [ K ] r ( i ; θ̃t ) . We address the case of unknown µ∗ . The key idea in our method is to maintain a posterior density of µ∗ , which we call a meta-posterior . This density represents uncertainty in µ∗ given history . In task s , we denote it by Qs and define it such that P ( µ∗ ∈ B |H1 : s−1 ) = ∫ µ∈B Qs ( µ ) dκ1 ( µ ) holds for any set B , where κ1 is the reference measure for µ . We use this more general notation , as opposing to dµ , because µ can be both continuous and discrete . When solving task s , Qs is used to compute an uncertainty-adjusted task prior Ps , which is a posterior density of θs , ∗ given history . Formally , Ps is a density such that P ( θs , ∗ ∈ B |H1 : s−1 ) = ∫ θ∈B Ps ( θ ) dκ2 ( θ ) holds for any set B , where κ2 is the reference measure for θ . After computing Ps , we run TS with prior Ps to solve task s. To maintain Qs and Ps , we find it useful expressing them using a recursive update rule below . Proposition 1 . Let Ls ( θ ) = ∏n ` =1 pAs , ` ( Ys , ` ( As , ` ) ; θ ) be the likelihood of observations in task s. Then for any task s ∈ [ m ] , Ps ( θ ) = ∫ µ P ( θ ; µ ) Qs ( µ ) dκ1 ( µ ) , Qs ( µ ) = ∫ θ Ls−1 ( θ ) P ( θ ; µ ) dκ2 ( θ ) Qs−1 ( µ ) . The claim is proved in Appendix A . The proof uses the Bayes rule , where we carefully account for the fact that the observations are collected adaptively , the pulled arm in round t of task s depends on history H1 : s , t . The pseudocode of our algorithm is in Algorithm 1 . Since the algorithm adapts to the unknown task prior P ( · ; µ∗ ) , we call it AdaTS . AdaTS can be implemented efficiently when Ps is a conjugate prior for rewards , or a mixture of conjugate priors . We discuss several exact and efficient implementations starting from Section 3.1 . The design of AdaTS is motivated by MetaTS [ 28 ] , which also maintains a meta-posterior Qs . The difference is that MetaTS samples µ̃s ∼ Qs in task s to be optimistic with respect to the unknown µ∗ . Then it runs TS with prior P ( · ; µ̃s ) . While simple and intuitive , the sampling of µ̃s induces a high variance and leads to a conservative worst-case analysis . We improve MetaTS by avoiding the sampling step . This leads to tighter and more general regret bounds ( Section 4 ) , beyond multi-armed bandits ; while the practical performance also improves significantly ( Section 5 ) .

Summary: This work presents a Thompson sampling algorithm for the Bayesian "meta-learning" bandit problem, where bandit instances are drawn from a "task" distribution of known form with unknown location parameter, itself drawn from a known meta-prior.  The key ideas in the proposed AdaTS algorithm are 1. to maintain a distribution over the unknown task prior parameter, and 2. to marginalize such uncertainties by learning, across instances, a posterior over it.  Leveraging hierarchical Bayesian modeling and conjugate priors, the authors present AdaTS, with implementations for multi-armed/linear bandits and combinatorial semi-bandits.  Bayesian regret bounds for the $n$-round $m$-instance AdaTS are shown to be of order $O(\sqrt{mn})$.  Experiments on synthetic datasets illustrate the added benefits of the proposed approach.