Meta-Learning for Contextual Bandit Exploration

We describe MÊLÉE , a meta-learning algorithm for learning a good exploration policy in the interactive contextual bandit setting . Here , an algorithm must take actions based on contexts , and learn based only on a reward signal from the action taken , thereby generating an exploration/exploitation trade-off . MÊLÉE addresses this trade-off by learning a good exploration strategy based on offline synthetic tasks , on which it can simulate the contextual bandit setting . Based on these simulations , MÊLÉE uses an imitation learning strategy to learn a good exploration policy that can then be applied to true contextual bandit tasks at test time . We compare MÊLÉE to seven strong baseline contextual bandit algorithms on a set of three hundred real-world datasets , on which it outperforms alternatives in most settings , especially when differences in rewards are large . Finally , we demonstrate the importance of having a rich feature representation for learning how to explore . 1 INTRODUCTION . In a contextual bandit problem , an agent attempts to optimize its behavior over a sequence of rounds based on limited feedback ( Kaelbling , 1994 ; Auer , 2003 ; Langford & Zhang , 2008 ) . In each round , the agent chooses an action based on a context ( features ) for that round , and observes a reward for that action but no others ( §2 ) . Contextual bandit problems arise in many real-world settings like online recommendations and personalized medicine . As in reinforcement learning , the agent must learn to balance exploitation ( taking actions that , based on past experience , it believes will lead to high instantaneous reward ) and exploration ( trying actions that it knows less about ) . In this paper , we present a meta-learning approach to automatically learn a good exploration mechanism from data . To achieve this , we use supervised learning data sets on which we can simulate contextual bandit tasks . Based on these simulations , our algorithm , MÊLÉE ( MEta LEarner for Exploration ) 1 , learns a good heuristic exploration strategy that should ideally generalize to future contextual bandit problems . MÊLÉE contrasts with more classical approaches to exploration ( like -greedy or LinUCB ; see §4 ) , in which exploration strategies are constructed by hand and by expert algorithm designer . These approaches often achieve provably good exploration strategies in the worst case , but are potentially overly pessimistic and are sometimes computationally intractable . At training time ( §2.3 ) , MÊLÉE simulates many contextual bandit problems from fully labeled data . Using this data , in each round , MÊLÉE is able to counterfactually simulate what would happen under all possible action choices . We can then use this information to compute regret estimates for each action , which can be optimized using the AggreVaTe imitation learning algorithm ( Ross & Bagnell , 2014 ) . Our imitation learning strategy mirrors that of the meta-learning approach of Bachman et al . ( 2017 ) in the active learning setting . We present a simplified , stylized analysis of the behavior of MÊLÉE to ensure that our cost function encourages good behavior ( § 2.4 ) . Empirically , we use MÊLÉE to train an exploration policy on only synthetic datasets and evaluate the resulting bandit performance across three hundred ( simulated ) contextual bandit tasks ( §3.2 ) , comparing to a number of alternative exploration algorithms , and showing the efficacy of our approach ( §3.4 ) . 1Code release : the code is available online https : //www.dropbox.com/sh/dc3v8po5cbu8zaw/ AACu1f_4c4wIZxD1e7W0KVZ0a ? dl=0 2 META-LEARNING FOR CONTEXTUAL BANDITS . Contextual bandits is a model of interaction in which an agent chooses actions ( based on contexts ) and receives immediate rewards for that action alone . For example , in a simplified news personalization setting , at each time step t , a user arrives and the system must choose a news article to display to them . Each possible news article corresponds to an action a , and the user corresponds to a context xt . After the system chooses an article at to display , it can observe , for instance , the amount of time that the user spends reading that article , which it can use as a reward rt ( at ) . The goal of the system is to choose articles to display that maximize the cumulative sum of rewards , but it has to do this without ever being able to know what the reward would have been had it shown a different article a′t . Formally , we largely follow the setup and notation of Agarwal et al . ( 2014 ) . Let X be an input space of contexts ( users ) and [ K ] = { 1 , . . . , K } be a finite action space ( articles ) . We consider the statistical setting in which there exists a fixed but unknown distribution D over pairs ( x , r ) ∈ X× [ 0 , 1 ] K , where r is a vector of rewards ( for convenience , we assume all rewards are bounded in [ 0 , 1 ] ) . In this setting , the world operates iteratively over rounds t = 1 , 2 , . . . . At each round t : 1 . The world draws ( xt , rt ) ∼ D and reveals context xt . 2 . The agent ( randomly ) chooses action at ∈ [ K ] based on xt , and observes reward rt ( at ) . The goal of an algorithm is to maximize the cumulative sum of rewards over time . Typically the primary quantity considered is the average regret of a sequence of actions a1 , . . . , aT to the behavior of the best possible function in a prespecified class F : Reg ( a1 , . . . , aT ) = max f∈F 1 T T∑ t=1 [ rt ( f ( xt ) ) − rt ( at ) ] ( 1 ) An agent is call no-regret if its average regret is zero in the limit of large T . 2.1 POLICY OPTIMIZATION OVER FIXED HISTORIES . To produce a good agent for interacting with the world , we assume access to a function class F and to an oracle policy optimizer for that function class . For example , F may be a set of single layer neural networks mapping user features ( e.g. , IP , browser , etc . ) x ∈ X to predicted rewards for actions ( articles ) a ∈ [ K ] , where K is the total number of actions . Formally , the observable record of interaction resulting from round t is the tuple ( xt , at , rt ( at ) , pt ( at ) ) ∈ X× [ K ] × [ 0 , 1 ] × [ 0 , 1 ] , where pt ( at ) is the probability that the agent chose action at , and the full history of interaction is ht = 〈 ( xi , ai , ri ( ai ) , pi ( ai ) ) 〉ti=1 . The oracle policy optimizer , POLOPT , takes as input a history of user interactions with the news recommendation system and outputs an f ∈ F with low expected regret . A standard example of a policy optimizer is to combine inverse propensity scaling ( IPS ) with a regression algorithm ( Dudik et al. , 2011 ) . Here , given a history h , each tuple ( x , a , r , p ) in that history is mapped to a multiple-output regression example . The input for this regression example is the same x ; the output is a vector of K costs , all of which are zero except the ath component , which takes value r/p . For example , if the agent chose to show to user x article 3 , made that decision with 80 % probability , and received a reward of 0.6 , then the corresponding output vector would be 〈0 , 0 , 0.75 , 0 , . . . , 0〉 . This mapping is done for all tuples in the history , and then a supervised learning algorithm on the function class F is used to produce a low-regret regressor f . This is the function returned by the policy optimizer . IPS has this nice property that it is an unbiased estimator ; unfortunately , it tends to have large variance especially when some probabilities p are small . In addition to IPS , there are several standard policy optimizers that mostly attempt to reduce variance while remaining unbiased : the direct method ( which estimates the reward function from given data and uses this estimate in place of actual reward ) , the double-robust estimator , and multitask regression . In our experiments , we use the direct method because we found it best on average , but in principle any could be used . 2.2 TEST TIME BEHAVIOR OF MÊLÉE . In order to have an effective approach to the contextual bandit problem , one must be able to both optimize a policy based on historic data and make decisions about how to explore . After all , in order for the example news recommendation system to learn whether a particular user is interested in news articles on some topic is to try showing such articles to see how the user responds ( or to generalize from related articles or users ) . The exploration/exploitation dilemma is fundamentally about longterm payoffs : is it worth trying something potentially suboptimal now in order to learn how to behave better in the future ? A particularly simple and effective form of exploration is -greedy : given a function f output by POLOPT , act according to f ( x ) with probability ( 1− ) and act uniformly at random with probability . Intuitively , one would hope to improve on such a strategy by taking more ( any ! ) information into account ; for instance , basing the probability of exploration on f ’ s uncertainty . Our goal in this paper is to learn how to explore from experience . The training procedure for MÊLÉE will use offline supervised learning problems to learn an exploration policy π , which takes two inputs : a function f ∈ F and a context x , and outputs an action . In our example , f will be the output of the policy optimizer on all historic data , and x will be the current user . This is used to produce an agent which interacts with the world , maintaining an initially empty history buffer h , as : 1 . The world draws ( xt , rt ) ∼ D and reveals context xt . 2 . The agent computes ft ← POLOPT ( h ) and a greedy action ãt = π ( ft , xt ) . 3 . The agent plays at = ãt with probability ( 1− µ ) , and at uniformly at random otherwise . 4 . The agent observes rt ( at ) and appends ( xt , at , rt ( at ) , pt ) to the history h , where pt = µ/K if at 6= ãt ; and pt = 1− µ+ µ/K if at = ãt . Here , ft is the function optimized on the historical data , and π uses it and xt to choose an action . Intuitively , π might choose to use the prediction ft ( xt ) most of the time , unless ft is quite uncertain on this example , in which case π might choose to return the second ( or third ) most likely action according to ft . The agent then performs a small amount of additional µ-greedy-style exploration : most of the time it acts according to π but occasionally it explores some more . In practice ( §3 ) , we find that setting µ = 0 is optimal in aggregate , but non-zero µ is necessary for our theory ( §2.4 ) .

Summary: This paper investigates a meta-learning approach for the contextual bandit problem. The goal is to learn a generic exploration policy from datasets, and then to apply the exploration policy to contextual bandit tasks. The authors have adapted an algorithm proposed for imitation learning (Ross & Bagnell 2014) to their setting. Some theoretical guarantees straightforwardly extracted from (Ross & Bagnell 2014) and from (Kakade et al 2008) are presented. Experiments are done on 300 supervised datasets.