Learning Effective Exploration Strategies For Contextual Bandits

1 INTRODUCTION . In a contextual bandit problem , an agent attempts to optimize its behavior over a sequence of rounds based on limited feedback ( Kaelbling , 1994 ; Auer , 2003 ; Langford & Zhang , 2008 ) . In each round , the agent chooses an action based on a context ( features ) for that round , and observes a reward for that action but no others ( §2 ) . Contextual bandit problems arise in many real-world settings like online recommendations and personalized medicine . As in reinforcement learning , the agent must learn to balance exploitation ( taking actions that , based on past experience , it believes will lead to high instantaneous reward ) and exploration ( trying actions that it knows less about ) . In this paper , we present a meta-learning approach to automatically learn a good exploration mechanism from data . To achieve this , we use synthetic supervised learning data sets on which we can simulate contextual bandit tasks in an offline setting . Based on these simulations , our algorithm , MÊLÉE ( MEta LEarner for Exploration ) 1 , learns a good heuristic exploration strategy that should ideally generalize to future contextual bandit problems . MÊLÉE contrasts with more classical approaches to exploration ( like -greedy or LinUCB ; see §6 ) , in which exploration strategies are constructed by expert algorithm designers . These approaches often achieve provably good exploration strategies in the worst case , but are potentially overly pessimistic and are sometimes computationally intractable . At training time ( § 3.2 ) , MÊLÉE simulates many contextual bandit problems from fully labeled synthetic data . Using this data , in each round , MÊLÉE is able to counterfactually simulate what would happen under all possible action choices . We can then use this information to compute regret estimates for each action , which can be optimized using the AggreVaTe imitation learning algorithm ( Ross & Bagnell , 2014 ) . Our imitation learning strategy mirrors that of the meta-learning approach of Bachman et al . ( 2017 ) in the active learning setting . We present a simplified , stylized analysis of the behavior of MÊLÉE to ensure that our cost function encourages good behavior ( §4 ) . Empirically , we use MÊLÉE to train an exploration policy on only synthetic datasets and evaluate this policy on both a contextual bandit task based on a natural learning to rank dataset as well as three hundred simulated contextual bandit tasks ( §5.2 ) . We compare the trained policy to a number of alternative exploration algorithms , and show the efficacy of our approach ( §5.3 ) . 1Code release : the code is available online https : //www.dropbox.com/sh/ dc3v8po5cbu8zaw/AACu1f_4c4wIZxD1e7W0KVZ0a ? dl=0 2 PRELIMINARIES : CONTEXTUAL BANDITS AND POLICY OPTIMIZATION . Contextual bandits is a model of interaction in which an agent chooses actions ( based on contexts ) and receives immediate rewards for that action alone . For example , in a simplified news personalization setting , at each time step t , a user arrives and the system must choose a news article to display to them . Each possible news article corresponds to an action a , and the user corresponds to a context xt . After the system chooses an article at to display , it can observe , for instance , the amount of time that the user spends reading that article , which it can use as a reward rt ( at ) . Formally , we largely follow the setup and notation of Agarwal et al . ( 2014 ) . Let X be an input space of contexts ( users ) and [ K ] = { 1 , . . . , K } be a finite action space ( articles ) . We consider the statistical setting in which there exists a fixed but unknown distribution D over pairs ( x , r ) ∈ X× [ 0 , 1 ] K , where r is a vector of rewards ( for convenience , we assume all rewards are bounded in [ 0 , 1 ] ) . In this setting , the world operates iteratively over rounds t = 1 , 2 , . . . . Each round t : 1 . The world draws ( xt , rt ) ∼ D and reveals context xt . 2 . The agent ( randomly ) chooses action at ∈ [ K ] based on xt , and observes reward rt ( at ) . The goal of an algorithm is to maximize the cumulative sum of rewards over time . Typically the primary quantity considered is the average regret of a sequence of actions a1 , . . . , aT to the behavior of the best possible function in a prespecified class F : λ ( a1 , . . . , aT ) = max f∈F 1 T T∑ t=1 [ rt ( f ( xt ) ) − rt ( at ) ] ( 1 ) An agent is call no-regret if its average regret is zero in the limit of large T . To produce a good agent for interacting with the world , we assume access to a function class F and to an oracle policy optimizer for that function class . For example , F may be a set of single layer neural networks mapping user features x ∈ X to predicted rewards for actions a ∈ [ K ] . Formally , the observable record of interaction resulting from round t is the tuple ( xt , at , rt ( at ) , pt ( at ) ) ∈ X× [ K ] × [ 0 , 1 ] × [ 0 , 1 ] , where pt ( at ) is the probability that the agent chose action at , and the full history of interaction is ht = 〈 ( xi , ai , ri ( ai ) , pi ( ai ) ) 〉ti=1 . The oracle policy optimizer , POLOPT , takes as input a history of user interactions and outputs an f ∈ F with low expected regret . A standard example of a policy optimizer is to combine inverse propensity scaling ( IPS ) with a regression algorithm ( Dudik et al. , 2011 ) . Here , given a history h , each tuple ( x , a , r , p ) in that history is mapped to a multiple-output regression example . The input for this regression example is the same x ; the output is a vector of K costs , all of which are zero except the ath component , which takes value r/p . This mapping is done for all tuples in the history , and a supervised learning algorithm on the function class F is used to produce a low-regret regressor f . This is the function returned by the policy optimizer . IPS , and other estimators that have lower-variance than IPS ( such as the doubly-robust estimator ) , have the property of being unbiased . In experiments , we use the direct method ( Dudik et al. , 2011 ) largely for its simplicity , however , MÊLÉE is agnostic to the type of the estimator used by the policy optimizer . 3 APPROACH : LEARNING AND EFFECTIVE EXPLORATION STRATEGY . In order to have an effective approach to the contextual bandit problem , one must be able to both optimize a policy based on historic data and make decisions about how to explore . The exploration/exploitation dilemma is fundamentally about long-term payoffs : is it worth trying something potentially suboptimal now in order to learn how to behave better in the future ? A particularly simple and effective form of exploration is -greedy : given a function f output by POLOPT , act according to f ( x ) with probability ( 1− ) and act uniformly at random with probability . Intuitively , one would hope to improve on such a strategy by taking more ( any ! ) information into account ; for instance , basing the probability of exploration on f ’ s uncertainty . In this section , we describe MÊLÉE , first by describing how it operates at test time when applied to a new contextual bandit problem ( §3.1 ) , and then by describing how to train it using synthetic simulated contextual bandit problems ( §3.2 ) . 3.1 TEST TIME BEHAVIOR OF MÊLÉE . Our goal in this paper is to learn how to explore from experience . The training procedure for MÊLÉE will use offline supervised learning problems to learn an exploration policy π , which takes two inputs : a function f ∈ F and a context x , and outputs an action . In our example , f will be the output of the policy optimizer on all historic data , and x will be the current user . This is used to produce an agent which interacts with the world , maintaining an initially empty history buffer h , as : 1 . The world draws ( xt , rt ) ∼ D and reveals context xt . 2 . The agent computes ft ← POLOPT ( h ) and a greedy action ãt = π ( ft , xt ) . 3 . The agent plays at = ãt with probability ( 1− µ ) , and at uniformly at random otherwise . 4 . The agent observes rt ( at ) and appends ( xt , at , rt ( at ) , pt ) to the history h , where pt = µ/K if at 6= ãt ; and pt = 1− µ+ µ/K if at = ãt . Here , ft is the function optimized on the historical data , and π uses it and xt to choose an action . Intuitively , π might choose to use the prediction ft ( xt ) most of the time , unless ft is quite uncertain on this example , in which case π might choose to return the second ( or third ) most likely action according to ft . The agent then performs a small amount of additional µ-greedy-style exploration : most of the time it acts according to π but occasionally it explores some more . In practice ( §5 ) , we find that setting µ = 0 is optimal in aggregate , but non-zero µ is necessary for our theory ( §4 ) . Importantly , we wish to train π using one set of tasks ( for which we have fully supervised data on which to run simulations ) and apply it to wholly different tasks ( for which we only have bandit feedback ) . To achieve this , we allow π to depend representationally on ft in arbitrary ways : for instance , it might use features that capture ft ’ s uncertainty on the current example ( see §5.1 for details ) . We additionally allow π to depend in a task-independent manner on the history ( for instance , which actions have not yet been tried ) : it can use features of the actions , rewards and probabilities in the history but not depend directly on the contexts x . This is to ensure that π only learns to explore and not also to solve the underlying task-dependent classification problem . Because π needs to learn to be task independent , we found that if ft ’ s predictions were uncalibrated , it was very difficult for π to generalize well to unseen tasks . Therefore , we additionally allow π to depend on a very small amount of fully labeled data from the task at hand , which we use to allow π to calibrate ft ’ s predictions . In our experiments we use only 30 fully labeled examples , but alternative approaches to calibrating ft that do not require this data would be preferable .

Summary: This paper proposes a meta-learning algorithm to solve the problem of exploration in a contextual bandit task using prior knowledge. This is analogous to how exploration strategies have been learned from past data in the meta-learning for RL (for example, [1]). Their algorithm simulates contextual bandit problems from fully labeled data and uses it to learn an exploration policy that works well on tasks with bandit feedback. The training step for the exploration policy builds upon the AggreVaTe algorithm, where policy optimization is performed on the history augmented data by using a separate roll-out policy to estimate the advantage of a particular action for a particular context, from the point of view of regret minimization.  In terms of theory, they show that by using specific algorithms (example, Banditron) as the inner policy optimization procedure, a no-regret algorithm can be obtained.