Metadata-based Multi-Task Bandits with Bayesian Hierarchical Models

1 Introduction . The multi-armed bandit ( MAB ) is a popular framework for sequential decision making problems , where the agent will sequentially choose among a few arms and receive a random reward for the arm [ 37 ] . Since the mean rewards for the arms are not known a priori and must be learned through partial feedbacks , the agent is faced with the well-known exploration-exploitation trade-off . MAB is receiving increasing attention and has been widely applied to areas such as clinical trials [ 19 ] , finance [ 54 ] , recommendation systems [ 70 ] , among others . How to explore efficiently is a central problem in MAB . In many modern applications , we usually have a large number of separate but related MAB tasks . For example , in e-commerce , the company needs to find the optimal display mode for each of many products , and in medical applications , each patient needs to individually undergo a series of treatment periods to find the personalized optimal treatment . These tasks typically share similar problem structures , but may have different reward distributions . Intuitively , appropriate information sharing can largely speed up our learning process and reduce the regret , while a naive pooling may cause a linear regret due to the bias . This paper is concerned with the following question : given a large number of MAB tasks , how do we efficiently share information among them ? The central task is to capture task relations in a principled way [ 68 ] . While a few approaches have been proposed ( see Section 2 ) , most of them only utilize the action-reward pairs observed for each task . To our knowledge , none of the existing works can leverage another valuable information source , i.e. , the metadata of each task , which contains some static task-specific features . Such metadata is commonly available in real applications [ 66 , 68 , 57 ] , such as the demographic features of each customer or patient , or basic information of each web page or product . Although the metadata can hardly be directly utilized in a single MAB task , it usually contains intrinsic information about each task , and hence can guide us to learn task relations and efficiently share knowledge in the multi-task setup . Specifically , suppose task i has a feature vector xi ( i.e. , metadata ) and its expected reward vector for all arms is ri . We consider a general formulation that ri is sampled from the conditional distribution P ( ri|xi ) . When P ( ri|xi ) is 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . informative , the metadata , being appropriately utilized , can guide our exploration and hence reduce the regret . Several specific cases of P ( ri|xi ) are illustrated in Figure 1 . As one of our motivating examples , consider that a company needs to learn about the personal preference of many users over several options ( e.g. , message templates ) , via sequential interactions with each user . Suppose some user features ( e.g. , gender , age , geography ) are informative in predicting the preference . Then , even before any interactions with a specific user , we can predict her preference with high probability , based on her features and our experience with other users . Such knowledge can be concisely encoded as P ( ri|xi ) . However , ri typically can not be fully determined by xi . Given the possible heterogeneity of ri conditional on xi , the true reward distribution ( and optimal arm ) of this specific user still needs to be learned and confirmed via interactions with herself . Therefore , we must carefully utilize these features in a multi-task algorithm . Indeed , we conjecture this is one way how humans adapt to tasks effectively , by constructing a rough first impression according to features , and then adjusting the impression gradually in the following interactions . To judiciously share information and minimize cumulative regrets , we design a multi-task bandit algorithm . Recall that , when P ( ri|xi ) is known a priori , a Thompson sampling ( TS ) algorithm [ 53 ] with P ( ri|xi ) as the prior of ri for each task i is known to be ( nearly ) optimal in Bayes regret , for a wide range of problems [ 37 ] . However , such knowledge typically is absent , and many concerns have been raised regarding that TS is sensitive to the prior [ 42 , 25 , 7 , 23 ] . In this work , we aim to aggregate data from all tasks to construct an informative prior for each task . Specifically , we propose to characterize the task relations via a Bayesian hierarchical model , upon which the Multi-Task TS ( MTTS ) algorithm is designed to continuously learn P ( ri|xi ) by aggregating data . With such a design , MTTS yields a similar low regret to the oracle TS algorithm which knows P ( ri|xi ) a priori . In particular , the metadata xi allows us to efficiently capture task relations and share information . The benefits can be seen clearly from our experiments and analysis . Contribution . Our contributions can be summarized as follows . First of all , motivated by the usefulness of the commonly available metadata , we introduce and formalize the metadata-based multi-task MAB problem . The related notion of regret ( multi-task regret ) is defined , which serves as an appropriate metric for this problem . Second , we propose to address this problem with a unified hierarchical Bayesian framework , which characterizes the task relations in a principled way . We design a TS-type algorithm , named MTTS , to continuously learn the task distribution while minimizing the cumulative regrets . The framework is general to accommodate various reward distributions and sequences of interactions , and interpretable as one can study how the information from other tasks affects one specific task and how its metadata determines its prior . Two concrete examples for Gaussian bandits and Bernoulli bandits are carefully analyzed , and efficient implementations are discussed . Our formulation and framework open a door to connect the multi-task bandit problem with the rich literature on Bayesian hierarchical models . Third , we theoretically demonstrate the benefits of metadata-based information sharing , by deriving the regret bounds for MTTS and several baselines under a Gaussian bandit setting . Specifically , the average multi-task regret of MTTS decreases as the number of tasks grows , while the other baselines suffer a linear regret . Lastly , systematic simulation experiments are conducted to investigate the performance of MTTS under different conditions , which provides meaningful insights . 2 Related Work . First of all , our setup subsumes the meta MAB problem [ 28 , 10 , 34 , 35 ] as a special case , where xi degenerates to a constant . Meta MAB assumes the tasks arrive sequentially and are drawn from one metadata-agnostic distribution . Among these works , meta TS [ 34 ] is the most related one , which assumes the mean reward vectors { ri } are i.i.d . sampled from some distribution f ( r ; θ ) , and pool data to learn θ . The other works either pool data from accomplished tasks to optimize algorithm hyper-parameters [ 28 ] or directly optimize the policy via policy gradient [ 10 , 35 ] . Compared with meta MAB , our setting is more general as we allow incorporating metadata to learn and capture task relations , and also allow the interactions with different tasks to arrive in arbitrary orders . The former is particularly important when there exists significant heterogeneity among tasks ( e.g. , setting ( c ) in Figure 1 ) , where a naive pooling may lose most information ; the latter allows broader applications . The results developed in this paper are equally applicable to meta MAB . In addition to meta MAB , there are several other topics concerned with multiple bandit tasks . We review the most related papers in this section , and provide a comprehensive comparison with additional related works and concepts in Appendix B . First of all , there is a large number of works on multi-agent bandits [ 30 , 12 , 62 , 17 ] studying that multiple agents interact with the same task concurrently . In contrast , we consider there is inter-task heterogeneity . Given such heterogeneity , certain approaches to capture task relations are then required : when there exist useful network structures between tasks , [ 13 ] and [ 61 ] propose to utilize this knowledge to capture task relations ; [ 6 ] assumes there is a finite number of task instances , considers the tasks arrive sequentially , and aims to transfer knowledge about this set of instances ; [ 64 ] assumes there exist pairwise upper bounds on the difference in reward distributions ; clustering of bandits [ 22 , 40 ] and latent bandits [ 26 , 27 ] assume there is a perfect clustered structure , that tasks in the same cluster share the same reward distribution . In contrast to all these setups , to our knowledge , this is the first work that can utilize metadata , an important information source that is commonly available . Practitioners are typically familiar with feature-based methods and our work adds one useful tool to the multi-task bandits toolbox . Besides , our modelling-based approach provides nice interpretability and is flexible to accommodate arbitrary order of interactions . Lastly , we learn task relations from data automatically , with arguably fewer structural assumptions . In the presence of task-specific features , another approach to pool information is to adapt contextual MAB ( CMAB ) algorithms [ 43 , 69 , 51 , 4 , 58 , 36 , 38 ] , by neglecting the task identities and regarding the metadata as ” contexts ” . Specifically , one can assume a deterministic relationship ri = f ( xi ; θ ) for some function f , and then apply CMAB algorithms to learn θ . However , such a solution heavily relies on the realizability assumption [ 21 ] , that ri = f ( xi ; θ ) holds exactly with no variation or error . When this assumption is violated , a linear regret is generally unavoidable . This limitation attracts increasing attention [ 21 , 33 , 32 ] . In our setup , the agent will interact with each task multiple times and the metadata is fixed , which naturally enables us to relax this restrictive assumption by regarding ri as a random variable sampled from P ( ri|xi ) . To our knowledge , this is the first work that can handle the realizability issue via utilizing the repeated-measurement structure . Finally , there exists a rich literature on multi-task supervised learning ( see [ 67 ] for a survey ) , where hierarchical models [ 44 , 55 , 63 ] are a popular choice , [ 49 , 3 , 46 ] studied aggregating historical data to construct informative priors , and [ 68 ] considered leveraging metadata to cluster tasks . Besides , this work is certainly related to single-task bandits . In particular , Thompson sampling is one of the most popular approaches with superior performance [ 1 , 53 , 60 , 34 , 2 , 18 ] . Its popularity is partially due to its flexibility to incorporate prior knowledge [ 14 ] . We provide a data-driven approach to construct such a prior . Our work naturally builds on these two areas to provide an efficient solution to the metadata-based multi-task MAB problem .

Summary:  The paper studies the multi-task multi-armed bandits, where tasks are played sequentially and each task is governed by some unknown hidden parameter (metadata) which follows certain prior distribution.  1. The paper comes up with a Bayesian framework for the metadata-based multi-task multi-armed bandits.  2. The paper designs Thompson sampling-type algorithm, called MTTS, and demonstrates its supremacy over other algorithms, by proving regret bounds for the common Bernoulli / Gaussian prior cases.  3. The paper shows detailed experimental results to corroborate the effectiveness of the algorithm MTTS.