Bayesian decision-making under misspecified priors with applications to meta-learning

1 Introduction . Bayesian decision-making algorithms are widely popular , due to both strong empirical performance and the flexibility afforded by incorporating inductive biases and domain knowledge through priors . However , in practical applications , any chosen prior is at best an approximation of the true environment in which the algorithm is deployed . This raises a critical question : How sensitive are Bayesian decision-making algorithms to prior misspecification ? For decision-making problems with a very large horizon , it suffices that the misspecified prior places a vanishingly small probability mass on the ground truth environment ; this condition is referred to informally as a “ grain of truth. ” This is because , in the large-horizon limit , Bayesian algorithms ( like many non-Bayesian methods ) should converge to the optimal policy . ∗Massachusetts Institute of Technology , msimchow @ mit.edu †Columbia University ‡Microsoft Research NYC §Massachusetts Institute of Technology 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . But in many practical settings , decision-making takes place on shorter time scales . Consider a news recommendation website that , when presented with a new user , sequentially offers a selection of currently trending articles . Such a system may only have a few opportunities to make recommendations before the user decides to navigate away , leaving little time to correct for misspecified or underspecified prior knowledge . Such examples are described more broadly by the meta-learning paradigm , where a single learning agent must complete multiple disparate-though-related tasks . In meta-learning problems , and in short-horizon problems more broadly , the “ grain of truth ” argument paints a rather uninformative picture . Consequently , recent work has begun to explore sensitivity bounds in applications with shorter horizons [ LL16 , KKZ+21 ] . However , these recent works focus on particular classes of priors and/or reward models , as well as on the Thompson sampling algorithm specifically . Notably , this leaves open questions about the extent to which prior sensitivity is determined by properties of the Bayesian decision-making algorithm , the reward model , and the prior itself . 1.1 Our Contributions . Motivated by meta-learning problems with short task horizons , we establish general , distributionindependent , and worst-case optimal bounds on the sensitivity of Bayesian algorithms to prior misspecification . We focus on the Bayesian bandit setting , where a mean-vector “ environment ” µ is drawn from a distribution P , and rewards for each action are drawn in accordance with µ . We study the performance of Bayesian algorithms which operate according to a misspecified prior P ′ . Sensitivity of Thompson Sampling and Related Bayesian Bandit Algorithms . As a concrete example , we consider the expected reward obtained by Thompson sampling with misspecified prior P ′ under environments drawn from true prior P . When the mean rewards lie in the range [ 0 , 1 ] , as in the Bernoulli reward setting , we show that the difference in expected reward between Thompson sampling with P ′ and with P is at most twice the total variation distance between P and P ′ multiplied by the square of the horizon length . We prove a lower bound demonstrating that , for worst-case priors , this result is tight up to constants . Moreover , our upper bound holds for any two priors P and P ′ and suffers no dependence on the complexity of the decision space . We extend this result in two directions . First , we remove the boundedness requirement on the mean reward range , showing that so long as certain tail probability conditions on the prior means are satisfied , a similar result holds . Second , we generalize beyond Thompson sampling , bounding the prior sensitivity of a broad class of Bayesian bandit algorithms , which we term n-Monte Carlo algorithms . Our lower bounds extend to this class , verifying sharp dependence on the parameter n. Sample Complexity of Bayesian Meta-Learning . We apply our prior sensitivity results to the Bayesian bandit meta-learning setting , in which a meta-learner iteratively interacts on bandit instances that are sampled from an unknown prior distribution . Motivated by our sensitivity analysis we describe a generic algorithmic recipe for Bayesian meta-learning , in which the meta-learner explores for several episodes to estimate the prior and then exploits by instantiating a Bayesian decision-maker with the learned prior . We formally consider two instantiations of this setup : ( 1 ) the Beta-Bernoulli setting where the rewards are Bernoulli and the prior is a product of Beta distributions and ( 2 ) the Gaussian-Gaussian setting where the rewards are Gaussian and the prior is a Gaussian ( with arbitrary covariance structure ) over the means . We note that the Gaussian-Gaussian setting was recently studied in [ KKZ+21 ] but only for the diagonal covariance setting . Bayesian Decision-Making Beyond the Bandit Setting . A striking feature of our proof is that it makes no explicit reference to the structure of bandit decision-making . As a consequence , our results extend seamlessly to both contextual bandits and the most general Bayesian decision-making problem : Bayesian POMDPs . While our sensitivity bounds hold almost verbatim in these settings , we note that estimating the prior may be statistically much more challenging in these scenarios , so there is no free lunch . To facilitate readability of the paper , we defer all further discussion and formal results to Appendix E. Experimental results . We complement our meta-learning theory with synthetic experiments in multi-armed and contextual bandit settings . Our experiments show the benefits of ( a ) meta-learning broadly , ( b ) estimating higher-order moments of the prior distribution , and ( c ) using less myopic algorithms like the Knowledge Gradient [ RPF12 ] over Thompson sampling when faced with structured environments . 1.2 Related Work . Bayesian Decision-Making . Bayesian decision-making broadly refers to a class of algorithms that use Bayesian methods to estimate various problem parameters , and then derive decision/allocation rules from these estimates . The study of Bayesian decision-making began with the seminal work of Thompson [ Tho33 ] , who introduced the Thompson sampling algorithm for adaptive experiment design in clinical trials . Thompson sampling later gained popularity in the reinforcement learning community as a means to solve multi-armed bandit and tabular reinforcement learning problems [ Str00 , OVR17 ] , and has been extended in many directions [ AL17 , AL18 , GMM14 ] . Recent years have seen the proliferation of other Bayesian decision-making and learning algorithms , including Information Directed Sampling [ RVR16 ] , Top-Two Thompson Sampling [ Rus16 ] , and Knowledge Gradient [ RPF12 ] . Sensitivity Analysis and Frequentist Regret . The field of robust Bayesian analysis examines the sensitivity of Bayesian inference to prior and model misspecification ( c.f. , [ BMP+94 ] ) . These approaches typically do not consider decision-making , so they do not account for multi-step adaptive sampling inherent in our setting . More recent works study frequentist regret for Thompson sampling [ AG12 , KKM12 ] . These guarantees can be interpreted as controlling the sensitivity to arbitrary degrees of prior misspecification , but consequently , they do not provide a precise picture of how misspecification affects performance . Moreover , frequentist guarantees for Thompson sampling focus on relatively long learning horizons , so they are less relevant in the context of meta-learning with many short-horizon tasks . Short-Horizon Sensitivity . Most closely related to our paper are two previous works on sensitivity of Thompson sampling to small amounts of misspecification in short-horizon settings . [ LL16 ] study the sensitivity of Thompson sampling for two-armed bandits when the prior has finite support . More recently , [ KKZ+21 ] study meta-learning with Thompson sampling and derive sensitivity bounds for Thompson sampling in multi-armed bandits with Gaussian rewards and independent-across-arm Gaussian priors . In contrast to both of these works , the bounds presented in this work apply to arbitrary families of priors , more general decision-making problems , and to more general families of decision-making algorithms . Further , as illustrated in Remark 2 , our bounds are also tighter than those achieved by [ KKZ+21 ] when specialized to their precise setting . Finally , our lower bounds demonstrate that the square-horizon factor incurred in [ KKZ+21 ] is unavoidable for worst-case priors ( though perhaps not for their special case ) . [ BSLZ19 ] study sensitivity of general Bayesian algorithms in a dynamic pricing context . Their approach requires “ sufficiently random ” reward noise and applies only to algorithms with a nonadaptive initial exploration phase ( unlike true Thompson Sampling ) ; under these conditions , they show that the trajectories under a well-specified and misspecified Bayesian decision maker can be coupled so that the two algorithms maintain the same posteriors with good probability . In contrast , our analysis applies only to Bayesian decision-making algorithms which have sufficient “ internal randomness ” ( e.g. , Thompson sampling , and more generally , the n-Monte Carlo algorithms ) . Our approach obviates assumptions about reward noise and initial exploration at the expense of slightly restricting the class of algorithms to which our guarantees apply . Meta-learning and Meta-RL . Meta-learning is a classical learning paradigm in which a learner faces many distinct-but-related tasks [ Thr96 , Thr98 , Bax98 , Bax00 , HYC01 ] . While the classical work primarily considered supervised learning tasks , recent , predominantly empirical work has focused on meta-reinforcement learning ( Meta-RL ) , where each task is itself a decision-making problem ( c.f. , [ WKNT+17 , DSC+16 ] ) . This includes some Bayesian approaches [ HGH+20 ] . While there have been some theoretical results on Meta-RL [ ALB13 , CLP20 , YHLD21 , HCJ+21 ] , apart from [ KKZ+21 ] we are not aware of other theoretical treatments with a Bayesian flavor . Learning under model misspecification . This work studies a specific notion of misspecification : running Bayesian decision-making algorithms with inexact approximations of a true underlying prior . Numerous other types of mispecification have been considered by the learning theory community more broadly , although typically in the absence of meta-learning and in frequentist settings . These include models where rewards may be changing with time [ GM11 , BGZ14 , CSLZ18 , WL21 ] or adversarially corrupted [ LMPL18 , GKT19 , ZS21 ] , or where a simple function class ( e.g. , linear models ) is used to approximate a more complex reward function [ DKWY20 , VRD19 , LSW20 , FGMZ20 ] . The Bayesian analog of these models is that the likelihood is misspecified , which is quite different from the prior misspecification considered here . Translating these notions of misspecification to Bayesian decision-making and unifying these lines of work remains an exciting direction for future research . 2 Setting and Notation . Throughout , we use bold v to denote vectors and non-bold va to denote scalars . When the vector vh has a subscript , vh , a denotes its coordinates . Bayesian Bandit Learning under Misspecification . A Bayesian bandit learning instance is specified by ( a ) an abstract action space A , ( b ) a parametric family of priors Pθ indexed by parameters θ ∈ Θ over mean vectors µ ∈ RA with coordinates µa , and ( c ) a function D : RA → ∆ ( RA ) mapping mean vectors µ to distributions over reward vectors r ∈ RA such that the expected reward under D ( µ ) is µ , i.e. , Er∼D ( µ ) [ r ] = µ.5 Note that this general setup allows the prior Pθ to encode complex dependencies between the mean rewards µa of actions a ∈ A . We consider an episodic bandit protocol with horizon H . First , µ ∼ Pθ is drawn from the prior . Then , at each time step h = 1 , 2 , . . . , H , the learner ’ s policy , specified by an algorithm alg , selects an action ah ∈ A . Simultaneously , a reward vector rh is drawn independently from D ( µ ) , and the learner observes reward rh = rh , ah . The choice of action ah may depend on the partial trajectory τh−1 = ( a1 , r1 , . . . , ah−1 , rh−1 ) . We let Pθ , alg denote the joint law over µ and the full trajectory τH , while expectations are denoted Eθ , alg . We abbreviate the full trajectory τ = τH . We denote the cumulative reward R ( θ , alg ) : = Eθ , alg [ ∑H h=1 rh ] = Eθ , alg [ ∑H h=1 µah ] . Bayesian Learning Algorithms . We study a class of algorithms alg ( θ ) also parameterized by θ ∈ Θ . For concreteness , the reader may think of alg ( θ ) as corresponding to Thompson sampling , where the learner internally computes posteriors using Pθ as its prior . More general classes of Bayesian algorithms are defined in Section 3.1 . We are interested in the consequences of misspecification ; that is , interacting with µ ∼ Pθ , but executing alg ( θ′ ) for some other θ′ ̸= θ . Note that our notation for the induced law on the trajectory is Pθ , alg ( θ′ ) . Episodic Bayesian Meta-Learning . We apply the above framework to the problem of Bayesian meta-learning . Let θ⋆ ∈ Θ be a ground-truth parameter . At each episode t = 1 , 2 , . . . , T , a mean parameter µ ( t ) is drawn i.i.d . from Pθ⋆ . Simultaneously , the learner commits to a ( potentially nonBayesian ) exploration strategy explore ( t ) and collects the induced trajectory τ ( t ) . At the end of T episodes , the learner selects a parameter θ̂ ∈ Θ as a function of τ ( 1 ) , . . . , τ ( T ) . The learner ’ s performance is evaluated on the expected reward of the plug-in algorithm on θ̂ : R ( θ⋆ , alg ( θ̂ ) ) . Further notation . Given two probability distributions P and Q over the same probability space ( Ω , F ) , we denote their total variation TV ( P ∥ Q ) : = supE∈F |P [ E ] −Q [ E ] | and Kullback-Leibler divergence KL ( P ∥ Q ) . If P is a joint distribution of random variables ( X , Y , Z , . . . ) , P ( X ) denotes the marginal of X under P , and P ( Y |X ) the conditional distribution ( as a function of random variable X ) . We define the diameter of a mean vector as diam ( µ ) : = supa∈A µa− infa∈A µa , which is a random variable when µ is drawn from Pθ . Throughout , log ( · ) denotes the natural logarithm . Given a space X , we let ∆ ( X ) be the set of probability distributions on X ; see Appendix B.1 for measure-theoretic considerations . 5In fact , our analysis extends to more general cases where the reward distribution is parameterized by more than just the mean vectors , but we restrict ourselves to the current setting for ease of exposition .

Summary: This paper focus on the performance of the Thompson sampling algorithm under the misspecification of the prior distribution. The authors propose a general family of n-Monte Carlo algorithm, which contains the canonical TS algorithm as a special case. The authors then proceed to show that the regret difference of a misspecified prior up to horizon H of an n-MC algorithm is of order H^2*eps, where eps is the amount of misspecification measured by the total variation distance. The results are applied to the meta-learning setting, in which one also aims to learn the prior. The paper concludes by a synthetic numerical experiment to show the benefit of meta-learning.