No Regrets for Learning the Prior in Bandits

1 Introduction . We study the problem of maximizing the total reward , or minimizing the total regret , in a sequence of stochastic bandit instances [ 29 , 4 , 31 ] . We consider a Bayesian version of the problem , where the bandit instances are drawn from some distribution . More specifically , the learning agent interacts with m bandit instances in m tasks , with one instance per task . The interaction with each task is for n rounds and with K arms . The reward distribution of arm i ∈ [ K ] in task s ∈ [ m ] is pi ( · ; θs , ∗ ) , where θs , ∗ is a shared parameter of all arms in task s. When arm i is pulled in task s , the agent receives a random reward from pi ( · ; θs , ∗ ) . The parameters θ1 , ∗ , . . . , θm , ∗ are drawn independently of each other from a task prior P ( · ; µ∗ ) . The task prior is parameterized by an unknown meta-parameter µ∗ , which is drawn from a meta-prior Q . The agent does not know µ∗ or θ1 , ∗ , . . . , θm , ∗ . However , it knows Q and the parametric forms of all distributions , which help it to learn about µ∗ . This is a form of meta-learning [ 39 , 40 , 7 , 8 ] , where the agent learns to act from interactions with bandit instances . A simple approach is to ignore the hierarchical structure of the problem and solve each bandit task independently with some bandit algorithm , such as Thompson sampling ( TS ) [ 38 , 11 , 2 , 37 ] . This may be highly suboptimal . To illustrate this , imagine that arm 1 is optimal for any θ in the support of P ( · ; µ∗ ) . If µ∗ was known , any reasonable algorithm would only pull arm 1 and have zero regret over any horizon . Likewise , a clever algorithm that learns µ∗ should eventually pull arm 1 most of the time , and thus have diminishing regret as it interacts with a growing number of tasks . Two challenges arise when designing the clever algorithm . First , can it be computationally efficient ? Second , what is the regret due to adapting to µ∗ ? We make the following contributions . First , we propose a Thompson sampling algorithm for our problem , which we call AdaTS . AdaTS maintains a distribution over the meta-parameter µ∗ , which concentrates over time and is marginalized out when interacting with individual bandit instances . Second , we propose computationally-efficient implementations of AdaTS for multi-armed bandits [ 29 , 4 ] , linear bandits [ 14 , 1 ] , and combinatorial semi-bandits [ 19 , 13 , 26 ] . These implementations are for specific reward distributions and conjugate task priors . Third , we bound the n-round Bayes regret of AdaTS in linear bandits and semi-bandits , and multi-armed bandits as a special case . The Bayes regret is defined by taking an expectation over all random quantities , including µ∗ ∼ Q . Our bounds show that not knowing µ∗ has a minimal impact on the regret as the number of tasks grows , of only Õ ( √ mn ) . This is in a sharp contrast to prior work [ 28 ] , where this is Õ ( √ mn2 ) . Finally , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . our experiments show that AdaTS quickly adapts to the unknown meta-parameter µ∗ , is robust to meta-prior misspecification , and performs well even in challenging classification problems . We present a general framework for learning to explore from similar past exploration problems . One potential application is cold-start personalization in recommender systems where users are tasks . The users have similar preferences , but neither the individual preferences nor their similarity is known in advance . Another application could be online regression with bandit feedback ( Appendix E.2 ) where individual regression problems are tasks . Similar examples in the tasks have similar mean responses , which are unknown in advance . 2 Setting . We first introduce our notation . The set { 1 , . . . , n } is denoted by [ n ] . The indicator 1 { E } denotes that event E occurs . The i-th entry of vector v is vi . If the vector or its index are already subindexed , we write v ( i ) . We use Õ for the big-O notation up to polylogarithmic factors . A diagonal matrix with entries v is denoted diag ( v ) . We use the terms “ arm ” and “ action ” interchangeably , depending on the context . Our setting was proposed in Kveton et al . [ 28 ] and is defined as follows . Each bandit problem instance has K arms . Each arm i ∈ [ K ] is defined by distribution pi ( · ; θ ) with parameter θ ∈ Θ . The parameter θ is shared among all arms . The mean of pi ( · ; θ ) is denoted by r ( i ; θ ) . The learning agent interacts with m instances , one at each of m tasks . At the beginning of task s ∈ [ m ] , an instance θs , ∗ ∈ Θ is sampled i.i.d . from a task prior P ( · ; µ∗ ) , which is parameterized by µ∗ . The agent interacts with θs , ∗ for n rounds . In round t ∈ [ n ] , it pulls one arm and observes a stochastic realization of its reward . We denote the pulled arm in round t of task s by As , t ∈ [ K ] , the realized rewards of all arms in round t of task s by Ys , t ∈ RK , and the reward of arm i ∈ [ K ] by Ys , t ( i ) ∼ pi ( · ; θs , ∗ ) . We assume that the realized rewards Ys , t are i.i.d . with respect to both s and t. A graphical model of our environment is drawn in Figure 1 . We define the distribution-specific parameters µq , Σq , Σ0 , and σ2 when we instantiate our framework . Our terminology is summarized in Appendix A . The n-round regret of an agent or algorithm over m tasks with task prior P ( · ; µ∗ ) is defined as R ( m , n ; µ∗ ) = m∑ s=1 E [ n∑ t=1 r ( As , ∗ ; θs , ∗ ) − r ( As , t ; θs , ∗ ) ∣∣∣∣∣µ∗ ] , ( 1 ) where As , ∗ = arg max i∈ [ K ] r ( i ; θs , ∗ ) is the optimal arm in the random problem instance θs , ∗ in task s ∈ [ m ] . The above expectation is over problem instances θs , ∗ ∼ P ( · ; µ∗ ) , their realized rewards , and also pulled arms . Note that µ∗ is fixed . Russo and Van Roy [ 36 ] showed that the Bayes regret , which matches the definition in ( 1 ) in any task , of Thompson sampling in a K-armed bandit with n rounds is Õ ( √ Kn ) . So , when TS is applied independently in each task , R ( m , n ; µ∗ ) = Õ ( m √ Kn ) . Our goal is to attain a comparable regret without knowing µ∗ . We frame this problem in a Bayesian fashion , where µ∗ ∼ Q before the learning agent interacts with the first task . The agent knows Q and we call it a meta-prior . Accordingly , we consider R ( m , n ) = E [ R ( m , n ; µ∗ ) ] as a metric and call it the Bayes regret . Our approach is motivated by hierarchical Bayesian models [ 20 ] , where the uncertainty in prior parameters , such as µ∗ , is represented by another distribution , such as Q . In these models , Q is called a hyper-prior and µ∗ is called a hyper-parameter . We attempt to learn µ∗ from sequential interactions with instances θs , ∗ ∼ P ( · ; µ∗ ) , which are also unknown . The agent can only observe their noisy realizations Ys , t . 3 Algorithm . Our algorithm is presented in this section . To describe it , we need to introduce several notions of history , the past interactions of the agent . We denote by Hs = ( As , t , Ys , t ( As , t ) ) nt=1 the history in Algorithm 1 AdaTS : Instance-adaptive exploration in Thompson sampling . 1 : Initialize meta-prior Q0 ← Q 2 : for s = 1 , . . . , m do 3 : Compute meta-posterior Qs ( Proposition 1 ) 4 : Compute uncertainty-adjusted task prior Ps ( Proposition 1 ) 5 : for t = 1 , . . . , n do 6 : Compute posterior of θ in task s , Ps , t ( θ ) ∝ Ls , t ( θ ) Ps ( θ ) 7 : Sample θ̃s , t ∼ Ps , t , pull arm As , t ← arg max i∈ [ K ] r ( i ; θ̃s , t ) , and observe Ys , t ( As , t ) task s and by H1 : s = H1 ⊕ · · · ⊕Hs a concatenated vector of all histories in the first s tasks . The history up to round t in task s is Hs , t = ( As , ` , Ys , ` ( As , ` ) ) t−1 ` =1 and all history up to round t in task s is H1 : s , t = H1 : s−1 ⊕Hs , t . We denote the conditional probability distribution given history H1 : s , t by Ps , t ( · ) = P ( · |H1 : s , t ) and the corresponding conditional expectation by Es , t [ · ] = E [ · |H1 : s , t ] . Our algorithm is a form of Thompson sampling [ 38 , 11 , 2 , 37 ] . TS pulls arms proportionally to being optimal with respect to the posterior . In particular , let Ls , t ( θ ) = ∏t−1 ` =1 pAs , ` ( Ys , ` ( As , ` ) ; θ ) be the likelihood of observations in task s up to round t. If the prior P ( · ; µ∗ ) was known , the posterior of instance θ in round t would be P TSs , t ( θ ) ∝ Ls , t ( θ ) P ( θ ; µ∗ ) . TS would sample θ̃t ∼ P TSs , t and pull arm At = arg max i∈ [ K ] r ( i ; θ̃t ) . We address the case of unknown µ∗ . The key idea in our method is to maintain a posterior density of µ∗ , which we call a meta-posterior . This density represents uncertainty in µ∗ given history . In task s , we denote it by Qs and define it such that P ( µ∗ ∈ B |H1 : s−1 ) = ∫ µ∈B Qs ( µ ) dκ1 ( µ ) holds for any set B , where κ1 is the reference measure for µ . We use this more general notation , as opposing to dµ , because µ can be both continuous and discrete . When solving task s , Qs is used to compute an uncertainty-adjusted task prior Ps , which is a posterior density of θs , ∗ given history . Formally , Ps is a density such that P ( θs , ∗ ∈ B |H1 : s−1 ) = ∫ θ∈B Ps ( θ ) dκ2 ( θ ) holds for any set B , where κ2 is the reference measure for θ . After computing Ps , we run TS with prior Ps to solve task s. To maintain Qs and Ps , we find it useful expressing them using a recursive update rule below . Proposition 1 . Let Ls ( θ ) = ∏n ` =1 pAs , ` ( Ys , ` ( As , ` ) ; θ ) be the likelihood of observations in task s. Then for any task s ∈ [ m ] , Ps ( θ ) = ∫ µ P ( θ ; µ ) Qs ( µ ) dκ1 ( µ ) , Qs ( µ ) = ∫ θ Ls−1 ( θ ) P ( θ ; µ ) dκ2 ( θ ) Qs−1 ( µ ) . The claim is proved in Appendix A . The proof uses the Bayes rule , where we carefully account for the fact that the observations are collected adaptively , the pulled arm in round t of task s depends on history H1 : s , t . The pseudocode of our algorithm is in Algorithm 1 . Since the algorithm adapts to the unknown task prior P ( · ; µ∗ ) , we call it AdaTS . AdaTS can be implemented efficiently when Ps is a conjugate prior for rewards , or a mixture of conjugate priors . We discuss several exact and efficient implementations starting from Section 3.1 . The design of AdaTS is motivated by MetaTS [ 28 ] , which also maintains a meta-posterior Qs . The difference is that MetaTS samples µ̃s ∼ Qs in task s to be optimistic with respect to the unknown µ∗ . Then it runs TS with prior P ( · ; µ̃s ) . While simple and intuitive , the sampling of µ̃s induces a high variance and leads to a conservative worst-case analysis . We improve MetaTS by avoiding the sampling step . This leads to tighter and more general regret bounds ( Section 4 ) , beyond multi-armed bandits ; while the practical performance also improves significantly ( Section 5 ) .

Summary: This paper studies a variant of Thompson Sampling for meta-learning. By maintaining the distribution of the parameter of the prior distribution, the proposed algorithm can adapt to the certain task it encountered with. The authors also provide Bayesian regret bound for this algorithm. Experiments are provided to support their theoretical claim showing their algorithm can outperform benchmark algorithms.