Bayesian Exploration for Lifelong Reinforcement Learning

1 INTRODUCTION . Reinforcement-learning ( RL ) methods ( Sutton & Barto , 1998 ; Kaelbling et al. , 1996 ) have been successfully applied to solve challenging individual tasks such as learning robotic control ( Duan et al. , 2016 ) and playing expert-level Go ( Silver et al. , 2017 ) . However , in the real world , a robot usually experiences a collection of distinct tasks that arrive sequentially throughout its operational lifetime ; learning each new task from scratch is infeasible , but treating them all as a single task will fail . Therefore , recent research has focused on algorithms that enable agents to learn across multiple , sequentially posed tasks , leveraging past knowledge from previous tasks to accelerate the learning of new tasks . This problem setting is known as lifelong reinforcement learning ( Brunskill & Li , 2014 ; Wilson et al. , 2007b ; Isele et al. , 2016b ) . The key questions in lifelong RL research are : How can an algorithm exploit knowledge gained from past tasks to improve performance in new tasks ( forward transfer ) , and how can data from new tasks help the agent to perform better on previously learned tasks ( backward transfer ) ? To answer these two questions , first consider a simple problem , which is to find different items in different houses . Here , a single task corresponds to finding items in a specific house . Although items may be stored in different locations in different houses , there still exists some shared information that connects all houses . For instance , a toothbrush is more likely to be found in a bathroom than a kitchen , and a room without a window is more likely to be a bathroom than a living room . Such information can significantly accelerate the search for items in newly encountered houses . We propose that extracting the common structure existing in previously encountered tasks can help the agent quickly learn the dynamics of the new tasks . Specifically , this paper considers lifelong RL problems that can be modeled as hidden-parameter MDPs or HiP-MDPs ( Doshi-Velez & Konidaris , 2016 ; Killian et al. , 2017 ) , where variations among the true task dynamics can be described by a set of hidden parameters . We model two main categories of learning across multiple tasks : the world-model distribution , which describes the probability distribution over tasks , and the task-specific model , that defines the ( stochastic ) dynamics within a single task . To enable more accurate sequential knowledge transfer , we separate the learning process for these two quantities and maintain a hierarchical Bayesian posterior to approximate them . The world-model posterior is designed to manage the uncertainty in the world-model distribution , while the task-specific posterior handles the uncertainty from the data collected from only the current task . We propose a Bayesian exploration method for lifelong RL ( BLRL ) that learns a Bayesian worldmodel posterior that distills the common structure of previous tasks , and then uses it as a prior to learn a task-specific model in each subsequent task . For the discrete case , we derive an explicit performance bound that shows that the task-specific model requires fewer samples to become accurate as the world-model posterior approaches the true underlying world-model distribution . We further develop VBLRL , a more scalable version of BLRL that uses variational inference to approximate the world-model distribution and leverages Bayesian Neural Networks ( BNNs ) to build the hierarchical Bayesian posterior . Our experimental results on a set of challenging domains show that our algorithms achieve better forward and backward transfer performance than state-of-the-art lifelong RL algorithms within limited samples for each task . 2 BACKGROUND . RL is the problem of maximizing the long-term expected reward of an agent interacting with an environment ( Sutton & Barto , 1998 ) . We usually model the environment as a Markov Decision Process or MDP ( Puterman , 1994 ) , described by a five tuple : 〈S , A , R , T , γ〉 , where S is a finite set of states ; A is a finite set of actions ; R : S ×A 7→ [ 0 , 1 ] is a reward function , with a lower and upper bound 0 and 1 ; T : S ×A 7→ Pr ( S ) is a transition function , with T ( s′|s , a ) denoting the probability of arriving in state s′ ∈ S after executing action a ∈ A in state s ; and γ ∈ [ 0 , 1 ) is a discount factor , expressing the agent ’ s preference for delayed over immediate rewards . An MDP is a suitable model for the task facing a single agent . In the lifelong RL setting , the agent instead faces a series of tasks τ1 , ... , τn , each of which can be modeled as an MDP : 〈S ( i ) , A ( i ) , R ( i ) , T ( i ) , γ ( i ) 〉 . A key question is how these task MDPs are related ; we model the collection of tasks as a HiP-MDP ( Doshi-Velez & Konidaris , 2016 ; Killian et al. , 2017 ) , where a family of tasks is generated by varying a latent task parameter ω drawn for each task according to the world-model distribution PΩ . Each setting of ω specifies a unique MDP , but the agent neither observes ω nor has access to the function that generates the task family . Formally , then , the dynamics T ( s′|s , a ; ωi ) and reward function R ( r|s , a ; ωi ) for task i depend on ωi ∈ Ω , which is fixed for the duration of the task . For lifelong RL problems , the performance of a specific algorithm is usually evaluated based on both forward transfer and backward transfer results ( Lopez-Paz & Ranzato , 2017 ) : • Forward transfer : the influence that learning task t has on the performance in future task k t. • Backward transfer : the influence that learning task t has on the performance in earlier tasks k ≺ t. 3 BAYESIAN EXPLORATION FOR LIFELONG REINFORCEMENT LEARNING 𝜔𝑖 Ψ 𝑚𝑖 𝜏𝑗 𝑖 = 1 ⋯ 𝐾 𝑗 = 1 ⋯ 𝑅 Figure 1 shows our generative model in plate notation . Ψ is the parameter set that represents distribution PΩ . It functions as the world-model posterior that aims to capture the common structure across different tasks . The resulting MDP mi is created based on ωi , which is one hidden parameter sampled from Ψ . We can sample from our approximation of Ψ to create and solve possible MDPs . The proposed BLRL approach is formalized in Algorithm 3 in the appendix . Initially , before any MDPs are experienced , the world-model posterior qe ( ·|st , at ) is initialized to an uniformed prior . For each new task mi , we first initialize the task-specific posterior qmiθ ( ·|st , at ) with the parameter values from the current world-model posterior , and then , for each timestep , select actions using a Bayesian exploration algorithm based on sampling from this posterior ( Thompson , 1933 ; Asmuth et al. , 2009 ) . A set of sampled MDPs drawn from qmiθ is a concrete representation of the uncertainty within the current task . BLRL samples K models from the task-specific posterior whenever the number of transitions from a state–action pair has reached threshold B. Analogously to RMAX ( Brafman & Tennenholtz , 2003 ) and BOSS ( Asmuth et al. , 2009 ) , we call a state–action pair known whenever it has been observed Nst , at = B times . For each state–action pair , if it is known , we use the taskspecific posterior to sample the model . If it is unknown , we instead sample from the world-model posterior . These models are combined into a merged MDP m # i and BLRL solves m # i to get a policy π∗ m # i . This approach is adopted from BOSS ( best of sampled set ) to create optimism in the face of uncertainty , and thereby drive exploration . The new policy π∗ m # i will be used to interact with the environment until a new state–action pair reaches the sampling threshold . The collected transitions from this task will be used to update the task-specific posterior immediately , while the world-model posterior will be updated using transitions from all the previous tasks at a slower pace . For simple finite MDP problems in practice , we use the Dirichlet distribution ( the conjugate for the multinomial ) to represent the Bayesian posterior . Thus , the updating process for the posterior is straightforward to compute . Intuitively , BLRL is able to rapidly adapt to new tasks as long as the prior of the task-specific model ( that is , the world-model posterior ) is close to the true underlying model and captures the uncertainty of the common structure of a set of tasks . 3.1 SAMPLE COMPLEXITY ANALYSIS . We now provide a simple theoretical analysis of BLRL . First , we use the setting and results of Zhang ( 2006 ) to describe the properties of the Bayesian prior and how it relates to the sample complexity for the concentration of the Bayesian posterior . Lemma 1 . Let π ( ω ) denote the prior distribution on the parameter space Γ . We consider a set of transition-probability densities p ( ·|ω ) indexed by ω , and the true underlying density q . Define the prior-mass radius of the transition-probability densities as : dπ = inf { d : d ≥ − lnπ ( { p ∈ Γ : DKL ( q||p ) ≤ d } ) } . ( 1 ) Intuitively , this quantity measures the distance between the Bayesian prior we use to initialize the posterior and the true underlying distribution . Then , ∀ρ ∈ ( 0 , 1 ) and η ≥ 1 , let εn = ( 1 + 1 n ) ηdπ + ( η − ρ ) εupper , n ( ( η − 1 ) / ( η − ρ ) ) , ( 2 ) where εupper , n is the critical upper-bracketing radius ( Zhang , 2006 ) . The decay rate of εupper , n controls the consistency of the Bayesian posterior distribution ( Asmuth et al. , 2009 ) . Let ρ = 12 , we have for all t ≥ 0 and δ ∈ ( 0 , 1 ) , with probability at least 1− δ , πn ( { p ∈ Γ : ||p− q||21/2 ≥ 2εn + ( 4η − 2 ) t δ/4 } ∣∣∣X ) ≤ 1 1 + ent . ( 3 ) Proof ( sketch ) . The proof is similar to that of Corollary 5.2 of Zhang ( 2006 ) ( see Appendix A.5 ) . Instead of using the critical prior-mass radius επ , n to describe certain characteristics of the Bayesian prior , we define and use the prior-mass radius dπ , which is independent of the sample size n and measures the distance between the prior and true distribution . Similar to BOSS , for a new MDP m ∼M with hidden parameters ωm , we can define the Bayesian concentration sample complexity for the task-specific posterior : f ( s , a , 0 , δ0 , ρ0 ) , as the minimum number c such that , if c IID transitions from ( s , a ) are observed , then , with probability at least 1− δ0 , Prm∼posterior ( ||Tm ( s , a , ωm ) − Tm∗ ( s , a , ωm ) ||1 < 0 ) ≥ 1− ρ0 . ( 4 ) Lemma 2 . Assume the posterior is consistent ( that is , εupper , n = o ( 1 ) ) and set η = 2 , then the Bayesian concentration sample complexity for the task-specific posterior f ( s , a , , δ , ρ ) = O ( dπ+ln 1 ρ 2δ−dπ ) . Proof ( sketch ) . This bound can be derived by directly combining Lemma 1 and Equation 4 . The above lemma suggests an upper bound of the Bayesian concentration sample complexity using the prior-mass radius . We can further combine this result with PAC-MDP theory ( Strehl et al. , 2006 ) and derive the sample complexity of the algorithm for each new task . Theorem 1 . For each new task , set the sample size K = Θ ( S 2A δ ln SA δ ) and the parameters 0 = ( 1− γ ) 2 , δ0 = δSA , ρ0 = δ S2A2K , then , with probability at least 1− 4δ , V At ( st ) ≥ V ∗ ( st ) − 4 0 in all but Õ ( S 2A2dπ δ 3 ( 1−γ ) 6 ) steps , where Õ ( · ) suppresses logarithmic dependence . Proof ( sketch ) . The proof is based on the PAC-MDP theorem ( Strehl et al. , 2009 ) combined with the new bound for the Bayesian concentration sample complexity we derived in Lemma 2 . In general , we use the same process in BOSS to verify the three required properties of PAC-MDP : optimism , accuracy and learning complexity . For each new task , the main difference between BLRL and BOSS is that we use the world-model posterior to initialize the task-specific posterior , which results in a new sample complexity bound based on the prior-mass radius . The result formalizes the intuition that , if we put a larger prior mass at a density that is close to the true q such that dπ is small , the sample complexity of our algorithm will be lower . In the meantime , the sample complexity is bounded by polynomial functions of the relevant quantities , showing that our training strategy preserves the properties required by PAC-MDP algorithms ( Strehl et al. , 2009 ) .

Summary: The paper deals with the problem of lifelong RL, also referred to as meat-RL, where an agent attempts to solve a sequence of tasks in order to facilitate the solution of a novel task. The framework follows that of Baxter 2000 (albeit that paper deals with supervised learning), and has been widely studied in recent years.  The basic assumption is that the tasks are drawn from an underlying task-distribution, and each task (an MDP) is stochastically selected from  a task-specific distribution. The authors work with a Bayesian framework, assuming a hierarchical distribution of the two levels, and learn the two levels separately. This framework has the advantage of providing both estimates and uncertainty estimates. For the discrete case they present a sample complexity analysis, and suggest a variational approach to practical learning. Finally, experiments are provided supporting the utility of the approach.   The formal framework is that of hidden-parameter MDPs (HiP-MDPs) from Doshi-Velez 2016, and each MDP is modeled based on a transition and a reward model based on a hidden parameter. As more tasks are encountered the posterior over world models sharpens, and, being used to learn new tasks, is expected to facilitate learning. The learning of each new task is as in BOSS, and takes place by sampling from the learned MDP distribution, creating a mixed MDP, and using standard model-based approaches to solve these.