Metadata-based Multi-Task Bandits with Bayesian Hierarchical Models

1 Introduction . The multi-armed bandit ( MAB ) is a popular framework for sequential decision making problems , where the agent will sequentially choose among a few arms and receive a random reward for the arm [ 37 ] . Since the mean rewards for the arms are not known a priori and must be learned through partial feedbacks , the agent is faced with the well-known exploration-exploitation trade-off . MAB is receiving increasing attention and has been widely applied to areas such as clinical trials [ 19 ] , finance [ 54 ] , recommendation systems [ 70 ] , among others . How to explore efficiently is a central problem in MAB . In many modern applications , we usually have a large number of separate but related MAB tasks . For example , in e-commerce , the company needs to find the optimal display mode for each of many products , and in medical applications , each patient needs to individually undergo a series of treatment periods to find the personalized optimal treatment . These tasks typically share similar problem structures , but may have different reward distributions . Intuitively , appropriate information sharing can largely speed up our learning process and reduce the regret , while a naive pooling may cause a linear regret due to the bias . This paper is concerned with the following question : given a large number of MAB tasks , how do we efficiently share information among them ? The central task is to capture task relations in a principled way [ 68 ] . While a few approaches have been proposed ( see Section 2 ) , most of them only utilize the action-reward pairs observed for each task . To our knowledge , none of the existing works can leverage another valuable information source , i.e. , the metadata of each task , which contains some static task-specific features . Such metadata is commonly available in real applications [ 66 , 68 , 57 ] , such as the demographic features of each customer or patient , or basic information of each web page or product . Although the metadata can hardly be directly utilized in a single MAB task , it usually contains intrinsic information about each task , and hence can guide us to learn task relations and efficiently share knowledge in the multi-task setup . Specifically , suppose task i has a feature vector xi ( i.e. , metadata ) and its expected reward vector for all arms is ri . We consider a general formulation that ri is sampled from the conditional distribution P ( ri|xi ) . When P ( ri|xi ) is 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . informative , the metadata , being appropriately utilized , can guide our exploration and hence reduce the regret . Several specific cases of P ( ri|xi ) are illustrated in Figure 1 . As one of our motivating examples , consider that a company needs to learn about the personal preference of many users over several options ( e.g. , message templates ) , via sequential interactions with each user . Suppose some user features ( e.g. , gender , age , geography ) are informative in predicting the preference . Then , even before any interactions with a specific user , we can predict her preference with high probability , based on her features and our experience with other users . Such knowledge can be concisely encoded as P ( ri|xi ) . However , ri typically can not be fully determined by xi . Given the possible heterogeneity of ri conditional on xi , the true reward distribution ( and optimal arm ) of this specific user still needs to be learned and confirmed via interactions with herself . Therefore , we must carefully utilize these features in a multi-task algorithm . Indeed , we conjecture this is one way how humans adapt to tasks effectively , by constructing a rough first impression according to features , and then adjusting the impression gradually in the following interactions . To judiciously share information and minimize cumulative regrets , we design a multi-task bandit algorithm . Recall that , when P ( ri|xi ) is known a priori , a Thompson sampling ( TS ) algorithm [ 53 ] with P ( ri|xi ) as the prior of ri for each task i is known to be ( nearly ) optimal in Bayes regret , for a wide range of problems [ 37 ] . However , such knowledge typically is absent , and many concerns have been raised regarding that TS is sensitive to the prior [ 42 , 25 , 7 , 23 ] . In this work , we aim to aggregate data from all tasks to construct an informative prior for each task . Specifically , we propose to characterize the task relations via a Bayesian hierarchical model , upon which the Multi-Task TS ( MTTS ) algorithm is designed to continuously learn P ( ri|xi ) by aggregating data . With such a design , MTTS yields a similar low regret to the oracle TS algorithm which knows P ( ri|xi ) a priori . In particular , the metadata xi allows us to efficiently capture task relations and share information . The benefits can be seen clearly from our experiments and analysis . Contribution . Our contributions can be summarized as follows . First of all , motivated by the usefulness of the commonly available metadata , we introduce and formalize the metadata-based multi-task MAB problem . The related notion of regret ( multi-task regret ) is defined , which serves as an appropriate metric for this problem . Second , we propose to address this problem with a unified hierarchical Bayesian framework , which characterizes the task relations in a principled way . We design a TS-type algorithm , named MTTS , to continuously learn the task distribution while minimizing the cumulative regrets . The framework is general to accommodate various reward distributions and sequences of interactions , and interpretable as one can study how the information from other tasks affects one specific task and how its metadata determines its prior . Two concrete examples for Gaussian bandits and Bernoulli bandits are carefully analyzed , and efficient implementations are discussed . Our formulation and framework open a door to connect the multi-task bandit problem with the rich literature on Bayesian hierarchical models . Third , we theoretically demonstrate the benefits of metadata-based information sharing , by deriving the regret bounds for MTTS and several baselines under a Gaussian bandit setting . Specifically , the average multi-task regret of MTTS decreases as the number of tasks grows , while the other baselines suffer a linear regret . Lastly , systematic simulation experiments are conducted to investigate the performance of MTTS under different conditions , which provides meaningful insights . 2 Related Work . First of all , our setup subsumes the meta MAB problem [ 28 , 10 , 34 , 35 ] as a special case , where xi degenerates to a constant . Meta MAB assumes the tasks arrive sequentially and are drawn from one metadata-agnostic distribution . Among these works , meta TS [ 34 ] is the most related one , which assumes the mean reward vectors { ri } are i.i.d . sampled from some distribution f ( r ; θ ) , and pool data to learn θ . The other works either pool data from accomplished tasks to optimize algorithm hyper-parameters [ 28 ] or directly optimize the policy via policy gradient [ 10 , 35 ] . Compared with meta MAB , our setting is more general as we allow incorporating metadata to learn and capture task relations , and also allow the interactions with different tasks to arrive in arbitrary orders . The former is particularly important when there exists significant heterogeneity among tasks ( e.g. , setting ( c ) in Figure 1 ) , where a naive pooling may lose most information ; the latter allows broader applications . The results developed in this paper are equally applicable to meta MAB . In addition to meta MAB , there are several other topics concerned with multiple bandit tasks . We review the most related papers in this section , and provide a comprehensive comparison with additional related works and concepts in Appendix B . First of all , there is a large number of works on multi-agent bandits [ 30 , 12 , 62 , 17 ] studying that multiple agents interact with the same task concurrently . In contrast , we consider there is inter-task heterogeneity . Given such heterogeneity , certain approaches to capture task relations are then required : when there exist useful network structures between tasks , [ 13 ] and [ 61 ] propose to utilize this knowledge to capture task relations ; [ 6 ] assumes there is a finite number of task instances , considers the tasks arrive sequentially , and aims to transfer knowledge about this set of instances ; [ 64 ] assumes there exist pairwise upper bounds on the difference in reward distributions ; clustering of bandits [ 22 , 40 ] and latent bandits [ 26 , 27 ] assume there is a perfect clustered structure , that tasks in the same cluster share the same reward distribution . In contrast to all these setups , to our knowledge , this is the first work that can utilize metadata , an important information source that is commonly available . Practitioners are typically familiar with feature-based methods and our work adds one useful tool to the multi-task bandits toolbox . Besides , our modelling-based approach provides nice interpretability and is flexible to accommodate arbitrary order of interactions . Lastly , we learn task relations from data automatically , with arguably fewer structural assumptions . In the presence of task-specific features , another approach to pool information is to adapt contextual MAB ( CMAB ) algorithms [ 43 , 69 , 51 , 4 , 58 , 36 , 38 ] , by neglecting the task identities and regarding the metadata as ” contexts ” . Specifically , one can assume a deterministic relationship ri = f ( xi ; θ ) for some function f , and then apply CMAB algorithms to learn θ . However , such a solution heavily relies on the realizability assumption [ 21 ] , that ri = f ( xi ; θ ) holds exactly with no variation or error . When this assumption is violated , a linear regret is generally unavoidable . This limitation attracts increasing attention [ 21 , 33 , 32 ] . In our setup , the agent will interact with each task multiple times and the metadata is fixed , which naturally enables us to relax this restrictive assumption by regarding ri as a random variable sampled from P ( ri|xi ) . To our knowledge , this is the first work that can handle the realizability issue via utilizing the repeated-measurement structure . Finally , there exists a rich literature on multi-task supervised learning ( see [ 67 ] for a survey ) , where hierarchical models [ 44 , 55 , 63 ] are a popular choice , [ 49 , 3 , 46 ] studied aggregating historical data to construct informative priors , and [ 68 ] considered leveraging metadata to cluster tasks . Besides , this work is certainly related to single-task bandits . In particular , Thompson sampling is one of the most popular approaches with superior performance [ 1 , 53 , 60 , 34 , 2 , 18 ] . Its popularity is partially due to its flexibility to incorporate prior knowledge [ 14 ] . We provide a data-driven approach to construct such a prior . Our work naturally builds on these two areas to provide an efficient solution to the metadata-based multi-task MAB problem .

Summary: The paper considers the multi-task bandit setting with a contextual information related to the task. The setting is defined in a fully Bayesian way, pathing the way to a Thompson Sampling (TS) bandit algorithm to handle this setting. A worst-case regret bound with respect to the parameters of the task-generator is proven and experiments on synthetic data show that the proposed model has smaller regret than baselines independently of the entropy of the task with respect to the contextual information.