Bayesian decision-making under misspecified priors with applications to meta-learning

1 Introduction . Bayesian decision-making algorithms are widely popular , due to both strong empirical performance and the flexibility afforded by incorporating inductive biases and domain knowledge through priors . However , in practical applications , any chosen prior is at best an approximation of the true environment in which the algorithm is deployed . This raises a critical question : How sensitive are Bayesian decision-making algorithms to prior misspecification ? For decision-making problems with a very large horizon , it suffices that the misspecified prior places a vanishingly small probability mass on the ground truth environment ; this condition is referred to informally as a “ grain of truth. ” This is because , in the large-horizon limit , Bayesian algorithms ( like many non-Bayesian methods ) should converge to the optimal policy . ∗Massachusetts Institute of Technology , msimchow @ mit.edu †Columbia University ‡Microsoft Research NYC §Massachusetts Institute of Technology 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . But in many practical settings , decision-making takes place on shorter time scales . Consider a news recommendation website that , when presented with a new user , sequentially offers a selection of currently trending articles . Such a system may only have a few opportunities to make recommendations before the user decides to navigate away , leaving little time to correct for misspecified or underspecified prior knowledge . Such examples are described more broadly by the meta-learning paradigm , where a single learning agent must complete multiple disparate-though-related tasks . In meta-learning problems , and in short-horizon problems more broadly , the “ grain of truth ” argument paints a rather uninformative picture . Consequently , recent work has begun to explore sensitivity bounds in applications with shorter horizons [ LL16 , KKZ+21 ] . However , these recent works focus on particular classes of priors and/or reward models , as well as on the Thompson sampling algorithm specifically . Notably , this leaves open questions about the extent to which prior sensitivity is determined by properties of the Bayesian decision-making algorithm , the reward model , and the prior itself . 1.1 Our Contributions . Motivated by meta-learning problems with short task horizons , we establish general , distributionindependent , and worst-case optimal bounds on the sensitivity of Bayesian algorithms to prior misspecification . We focus on the Bayesian bandit setting , where a mean-vector “ environment ” µ is drawn from a distribution P , and rewards for each action are drawn in accordance with µ . We study the performance of Bayesian algorithms which operate according to a misspecified prior P ′ . Sensitivity of Thompson Sampling and Related Bayesian Bandit Algorithms . As a concrete example , we consider the expected reward obtained by Thompson sampling with misspecified prior P ′ under environments drawn from true prior P . When the mean rewards lie in the range [ 0 , 1 ] , as in the Bernoulli reward setting , we show that the difference in expected reward between Thompson sampling with P ′ and with P is at most twice the total variation distance between P and P ′ multiplied by the square of the horizon length . We prove a lower bound demonstrating that , for worst-case priors , this result is tight up to constants . Moreover , our upper bound holds for any two priors P and P ′ and suffers no dependence on the complexity of the decision space . We extend this result in two directions . First , we remove the boundedness requirement on the mean reward range , showing that so long as certain tail probability conditions on the prior means are satisfied , a similar result holds . Second , we generalize beyond Thompson sampling , bounding the prior sensitivity of a broad class of Bayesian bandit algorithms , which we term n-Monte Carlo algorithms . Our lower bounds extend to this class , verifying sharp dependence on the parameter n. Sample Complexity of Bayesian Meta-Learning . We apply our prior sensitivity results to the Bayesian bandit meta-learning setting , in which a meta-learner iteratively interacts on bandit instances that are sampled from an unknown prior distribution . Motivated by our sensitivity analysis we describe a generic algorithmic recipe for Bayesian meta-learning , in which the meta-learner explores for several episodes to estimate the prior and then exploits by instantiating a Bayesian decision-maker with the learned prior . We formally consider two instantiations of this setup : ( 1 ) the Beta-Bernoulli setting where the rewards are Bernoulli and the prior is a product of Beta distributions and ( 2 ) the Gaussian-Gaussian setting where the rewards are Gaussian and the prior is a Gaussian ( with arbitrary covariance structure ) over the means . We note that the Gaussian-Gaussian setting was recently studied in [ KKZ+21 ] but only for the diagonal covariance setting . Bayesian Decision-Making Beyond the Bandit Setting . A striking feature of our proof is that it makes no explicit reference to the structure of bandit decision-making . As a consequence , our results extend seamlessly to both contextual bandits and the most general Bayesian decision-making problem : Bayesian POMDPs . While our sensitivity bounds hold almost verbatim in these settings , we note that estimating the prior may be statistically much more challenging in these scenarios , so there is no free lunch . To facilitate readability of the paper , we defer all further discussion and formal results to Appendix E. Experimental results . We complement our meta-learning theory with synthetic experiments in multi-armed and contextual bandit settings . Our experiments show the benefits of ( a ) meta-learning broadly , ( b ) estimating higher-order moments of the prior distribution , and ( c ) using less myopic algorithms like the Knowledge Gradient [ RPF12 ] over Thompson sampling when faced with structured environments . 1.2 Related Work . Bayesian Decision-Making . Bayesian decision-making broadly refers to a class of algorithms that use Bayesian methods to estimate various problem parameters , and then derive decision/allocation rules from these estimates . The study of Bayesian decision-making began with the seminal work of Thompson [ Tho33 ] , who introduced the Thompson sampling algorithm for adaptive experiment design in clinical trials . Thompson sampling later gained popularity in the reinforcement learning community as a means to solve multi-armed bandit and tabular reinforcement learning problems [ Str00 , OVR17 ] , and has been extended in many directions [ AL17 , AL18 , GMM14 ] . Recent years have seen the proliferation of other Bayesian decision-making and learning algorithms , including Information Directed Sampling [ RVR16 ] , Top-Two Thompson Sampling [ Rus16 ] , and Knowledge Gradient [ RPF12 ] . Sensitivity Analysis and Frequentist Regret . The field of robust Bayesian analysis examines the sensitivity of Bayesian inference to prior and model misspecification ( c.f. , [ BMP+94 ] ) . These approaches typically do not consider decision-making , so they do not account for multi-step adaptive sampling inherent in our setting . More recent works study frequentist regret for Thompson sampling [ AG12 , KKM12 ] . These guarantees can be interpreted as controlling the sensitivity to arbitrary degrees of prior misspecification , but consequently , they do not provide a precise picture of how misspecification affects performance . Moreover , frequentist guarantees for Thompson sampling focus on relatively long learning horizons , so they are less relevant in the context of meta-learning with many short-horizon tasks . Short-Horizon Sensitivity . Most closely related to our paper are two previous works on sensitivity of Thompson sampling to small amounts of misspecification in short-horizon settings . [ LL16 ] study the sensitivity of Thompson sampling for two-armed bandits when the prior has finite support . More recently , [ KKZ+21 ] study meta-learning with Thompson sampling and derive sensitivity bounds for Thompson sampling in multi-armed bandits with Gaussian rewards and independent-across-arm Gaussian priors . In contrast to both of these works , the bounds presented in this work apply to arbitrary families of priors , more general decision-making problems , and to more general families of decision-making algorithms . Further , as illustrated in Remark 2 , our bounds are also tighter than those achieved by [ KKZ+21 ] when specialized to their precise setting . Finally , our lower bounds demonstrate that the square-horizon factor incurred in [ KKZ+21 ] is unavoidable for worst-case priors ( though perhaps not for their special case ) . [ BSLZ19 ] study sensitivity of general Bayesian algorithms in a dynamic pricing context . Their approach requires “ sufficiently random ” reward noise and applies only to algorithms with a nonadaptive initial exploration phase ( unlike true Thompson Sampling ) ; under these conditions , they show that the trajectories under a well-specified and misspecified Bayesian decision maker can be coupled so that the two algorithms maintain the same posteriors with good probability . In contrast , our analysis applies only to Bayesian decision-making algorithms which have sufficient “ internal randomness ” ( e.g. , Thompson sampling , and more generally , the n-Monte Carlo algorithms ) . Our approach obviates assumptions about reward noise and initial exploration at the expense of slightly restricting the class of algorithms to which our guarantees apply . Meta-learning and Meta-RL . Meta-learning is a classical learning paradigm in which a learner faces many distinct-but-related tasks [ Thr96 , Thr98 , Bax98 , Bax00 , HYC01 ] . While the classical work primarily considered supervised learning tasks , recent , predominantly empirical work has focused on meta-reinforcement learning ( Meta-RL ) , where each task is itself a decision-making problem ( c.f. , [ WKNT+17 , DSC+16 ] ) . This includes some Bayesian approaches [ HGH+20 ] . While there have been some theoretical results on Meta-RL [ ALB13 , CLP20 , YHLD21 , HCJ+21 ] , apart from [ KKZ+21 ] we are not aware of other theoretical treatments with a Bayesian flavor . Learning under model misspecification . This work studies a specific notion of misspecification : running Bayesian decision-making algorithms with inexact approximations of a true underlying prior . Numerous other types of mispecification have been considered by the learning theory community more broadly , although typically in the absence of meta-learning and in frequentist settings . These include models where rewards may be changing with time [ GM11 , BGZ14 , CSLZ18 , WL21 ] or adversarially corrupted [ LMPL18 , GKT19 , ZS21 ] , or where a simple function class ( e.g. , linear models ) is used to approximate a more complex reward function [ DKWY20 , VRD19 , LSW20 , FGMZ20 ] . The Bayesian analog of these models is that the likelihood is misspecified , which is quite different from the prior misspecification considered here . Translating these notions of misspecification to Bayesian decision-making and unifying these lines of work remains an exciting direction for future research . 2 Setting and Notation . Throughout , we use bold v to denote vectors and non-bold va to denote scalars . When the vector vh has a subscript , vh , a denotes its coordinates . Bayesian Bandit Learning under Misspecification . A Bayesian bandit learning instance is specified by ( a ) an abstract action space A , ( b ) a parametric family of priors Pθ indexed by parameters θ ∈ Θ over mean vectors µ ∈ RA with coordinates µa , and ( c ) a function D : RA → ∆ ( RA ) mapping mean vectors µ to distributions over reward vectors r ∈ RA such that the expected reward under D ( µ ) is µ , i.e. , Er∼D ( µ ) [ r ] = µ.5 Note that this general setup allows the prior Pθ to encode complex dependencies between the mean rewards µa of actions a ∈ A . We consider an episodic bandit protocol with horizon H . First , µ ∼ Pθ is drawn from the prior . Then , at each time step h = 1 , 2 , . . . , H , the learner ’ s policy , specified by an algorithm alg , selects an action ah ∈ A . Simultaneously , a reward vector rh is drawn independently from D ( µ ) , and the learner observes reward rh = rh , ah . The choice of action ah may depend on the partial trajectory τh−1 = ( a1 , r1 , . . . , ah−1 , rh−1 ) . We let Pθ , alg denote the joint law over µ and the full trajectory τH , while expectations are denoted Eθ , alg . We abbreviate the full trajectory τ = τH . We denote the cumulative reward R ( θ , alg ) : = Eθ , alg [ ∑H h=1 rh ] = Eθ , alg [ ∑H h=1 µah ] . Bayesian Learning Algorithms . We study a class of algorithms alg ( θ ) also parameterized by θ ∈ Θ . For concreteness , the reader may think of alg ( θ ) as corresponding to Thompson sampling , where the learner internally computes posteriors using Pθ as its prior . More general classes of Bayesian algorithms are defined in Section 3.1 . We are interested in the consequences of misspecification ; that is , interacting with µ ∼ Pθ , but executing alg ( θ′ ) for some other θ′ ̸= θ . Note that our notation for the induced law on the trajectory is Pθ , alg ( θ′ ) . Episodic Bayesian Meta-Learning . We apply the above framework to the problem of Bayesian meta-learning . Let θ⋆ ∈ Θ be a ground-truth parameter . At each episode t = 1 , 2 , . . . , T , a mean parameter µ ( t ) is drawn i.i.d . from Pθ⋆ . Simultaneously , the learner commits to a ( potentially nonBayesian ) exploration strategy explore ( t ) and collects the induced trajectory τ ( t ) . At the end of T episodes , the learner selects a parameter θ̂ ∈ Θ as a function of τ ( 1 ) , . . . , τ ( T ) . The learner ’ s performance is evaluated on the expected reward of the plug-in algorithm on θ̂ : R ( θ⋆ , alg ( θ̂ ) ) . Further notation . Given two probability distributions P and Q over the same probability space ( Ω , F ) , we denote their total variation TV ( P ∥ Q ) : = supE∈F |P [ E ] −Q [ E ] | and Kullback-Leibler divergence KL ( P ∥ Q ) . If P is a joint distribution of random variables ( X , Y , Z , . . . ) , P ( X ) denotes the marginal of X under P , and P ( Y |X ) the conditional distribution ( as a function of random variable X ) . We define the diameter of a mean vector as diam ( µ ) : = supa∈A µa− infa∈A µa , which is a random variable when µ is drawn from Pθ . Throughout , log ( · ) denotes the natural logarithm . Given a space X , we let ∆ ( X ) be the set of probability distributions on X ; see Appendix B.1 for measure-theoretic considerations . 5In fact , our analysis extends to more general cases where the reward distribution is parameterized by more than just the mean vectors , but we restrict ourselves to the current setting for ease of exposition .

Summary: This article presents some very nice results on posterior-sampling-based bandit algorithms with mis-specified priors. The algorithms are those like Thompson sampling, in which the actions selected are functions of draws from the posterior distribution to date (an approximation of knowledge gradient is also analysed as part of a wider class of posterior-sample-based decision algorithms, and this is shown to be an improvement on TS). Results apply both to standard bandit settings and to meta-learning settings in which a series of short episodes are attempted. Theory and experiments are impressive.