Bayesian Exploration for Lifelong Reinforcement Learning

1 INTRODUCTION . Reinforcement-learning ( RL ) methods ( Sutton & Barto , 1998 ; Kaelbling et al. , 1996 ) have been successfully applied to solve challenging individual tasks such as learning robotic control ( Duan et al. , 2016 ) and playing expert-level Go ( Silver et al. , 2017 ) . However , in the real world , a robot usually experiences a collection of distinct tasks that arrive sequentially throughout its operational lifetime ; learning each new task from scratch is infeasible , but treating them all as a single task will fail . Therefore , recent research has focused on algorithms that enable agents to learn across multiple , sequentially posed tasks , leveraging past knowledge from previous tasks to accelerate the learning of new tasks . This problem setting is known as lifelong reinforcement learning ( Brunskill & Li , 2014 ; Wilson et al. , 2007b ; Isele et al. , 2016b ) . The key questions in lifelong RL research are : How can an algorithm exploit knowledge gained from past tasks to improve performance in new tasks ( forward transfer ) , and how can data from new tasks help the agent to perform better on previously learned tasks ( backward transfer ) ? To answer these two questions , first consider a simple problem , which is to find different items in different houses . Here , a single task corresponds to finding items in a specific house . Although items may be stored in different locations in different houses , there still exists some shared information that connects all houses . For instance , a toothbrush is more likely to be found in a bathroom than a kitchen , and a room without a window is more likely to be a bathroom than a living room . Such information can significantly accelerate the search for items in newly encountered houses . We propose that extracting the common structure existing in previously encountered tasks can help the agent quickly learn the dynamics of the new tasks . Specifically , this paper considers lifelong RL problems that can be modeled as hidden-parameter MDPs or HiP-MDPs ( Doshi-Velez & Konidaris , 2016 ; Killian et al. , 2017 ) , where variations among the true task dynamics can be described by a set of hidden parameters . We model two main categories of learning across multiple tasks : the world-model distribution , which describes the probability distribution over tasks , and the task-specific model , that defines the ( stochastic ) dynamics within a single task . To enable more accurate sequential knowledge transfer , we separate the learning process for these two quantities and maintain a hierarchical Bayesian posterior to approximate them . The world-model posterior is designed to manage the uncertainty in the world-model distribution , while the task-specific posterior handles the uncertainty from the data collected from only the current task . We propose a Bayesian exploration method for lifelong RL ( BLRL ) that learns a Bayesian worldmodel posterior that distills the common structure of previous tasks , and then uses it as a prior to learn a task-specific model in each subsequent task . For the discrete case , we derive an explicit performance bound that shows that the task-specific model requires fewer samples to become accurate as the world-model posterior approaches the true underlying world-model distribution . We further develop VBLRL , a more scalable version of BLRL that uses variational inference to approximate the world-model distribution and leverages Bayesian Neural Networks ( BNNs ) to build the hierarchical Bayesian posterior . Our experimental results on a set of challenging domains show that our algorithms achieve better forward and backward transfer performance than state-of-the-art lifelong RL algorithms within limited samples for each task . 2 BACKGROUND . RL is the problem of maximizing the long-term expected reward of an agent interacting with an environment ( Sutton & Barto , 1998 ) . We usually model the environment as a Markov Decision Process or MDP ( Puterman , 1994 ) , described by a five tuple : 〈S , A , R , T , γ〉 , where S is a finite set of states ; A is a finite set of actions ; R : S ×A 7→ [ 0 , 1 ] is a reward function , with a lower and upper bound 0 and 1 ; T : S ×A 7→ Pr ( S ) is a transition function , with T ( s′|s , a ) denoting the probability of arriving in state s′ ∈ S after executing action a ∈ A in state s ; and γ ∈ [ 0 , 1 ) is a discount factor , expressing the agent ’ s preference for delayed over immediate rewards . An MDP is a suitable model for the task facing a single agent . In the lifelong RL setting , the agent instead faces a series of tasks τ1 , ... , τn , each of which can be modeled as an MDP : 〈S ( i ) , A ( i ) , R ( i ) , T ( i ) , γ ( i ) 〉 . A key question is how these task MDPs are related ; we model the collection of tasks as a HiP-MDP ( Doshi-Velez & Konidaris , 2016 ; Killian et al. , 2017 ) , where a family of tasks is generated by varying a latent task parameter ω drawn for each task according to the world-model distribution PΩ . Each setting of ω specifies a unique MDP , but the agent neither observes ω nor has access to the function that generates the task family . Formally , then , the dynamics T ( s′|s , a ; ωi ) and reward function R ( r|s , a ; ωi ) for task i depend on ωi ∈ Ω , which is fixed for the duration of the task . For lifelong RL problems , the performance of a specific algorithm is usually evaluated based on both forward transfer and backward transfer results ( Lopez-Paz & Ranzato , 2017 ) : • Forward transfer : the influence that learning task t has on the performance in future task k t. • Backward transfer : the influence that learning task t has on the performance in earlier tasks k ≺ t. 3 BAYESIAN EXPLORATION FOR LIFELONG REINFORCEMENT LEARNING 𝜔𝑖 Ψ 𝑚𝑖 𝜏𝑗 𝑖 = 1 ⋯ 𝐾 𝑗 = 1 ⋯ 𝑅 Figure 1 shows our generative model in plate notation . Ψ is the parameter set that represents distribution PΩ . It functions as the world-model posterior that aims to capture the common structure across different tasks . The resulting MDP mi is created based on ωi , which is one hidden parameter sampled from Ψ . We can sample from our approximation of Ψ to create and solve possible MDPs . The proposed BLRL approach is formalized in Algorithm 3 in the appendix . Initially , before any MDPs are experienced , the world-model posterior qe ( ·|st , at ) is initialized to an uniformed prior . For each new task mi , we first initialize the task-specific posterior qmiθ ( ·|st , at ) with the parameter values from the current world-model posterior , and then , for each timestep , select actions using a Bayesian exploration algorithm based on sampling from this posterior ( Thompson , 1933 ; Asmuth et al. , 2009 ) . A set of sampled MDPs drawn from qmiθ is a concrete representation of the uncertainty within the current task . BLRL samples K models from the task-specific posterior whenever the number of transitions from a state–action pair has reached threshold B. Analogously to RMAX ( Brafman & Tennenholtz , 2003 ) and BOSS ( Asmuth et al. , 2009 ) , we call a state–action pair known whenever it has been observed Nst , at = B times . For each state–action pair , if it is known , we use the taskspecific posterior to sample the model . If it is unknown , we instead sample from the world-model posterior . These models are combined into a merged MDP m # i and BLRL solves m # i to get a policy π∗ m # i . This approach is adopted from BOSS ( best of sampled set ) to create optimism in the face of uncertainty , and thereby drive exploration . The new policy π∗ m # i will be used to interact with the environment until a new state–action pair reaches the sampling threshold . The collected transitions from this task will be used to update the task-specific posterior immediately , while the world-model posterior will be updated using transitions from all the previous tasks at a slower pace . For simple finite MDP problems in practice , we use the Dirichlet distribution ( the conjugate for the multinomial ) to represent the Bayesian posterior . Thus , the updating process for the posterior is straightforward to compute . Intuitively , BLRL is able to rapidly adapt to new tasks as long as the prior of the task-specific model ( that is , the world-model posterior ) is close to the true underlying model and captures the uncertainty of the common structure of a set of tasks . 3.1 SAMPLE COMPLEXITY ANALYSIS . We now provide a simple theoretical analysis of BLRL . First , we use the setting and results of Zhang ( 2006 ) to describe the properties of the Bayesian prior and how it relates to the sample complexity for the concentration of the Bayesian posterior . Lemma 1 . Let π ( ω ) denote the prior distribution on the parameter space Γ . We consider a set of transition-probability densities p ( ·|ω ) indexed by ω , and the true underlying density q . Define the prior-mass radius of the transition-probability densities as : dπ = inf { d : d ≥ − lnπ ( { p ∈ Γ : DKL ( q||p ) ≤ d } ) } . ( 1 ) Intuitively , this quantity measures the distance between the Bayesian prior we use to initialize the posterior and the true underlying distribution . Then , ∀ρ ∈ ( 0 , 1 ) and η ≥ 1 , let εn = ( 1 + 1 n ) ηdπ + ( η − ρ ) εupper , n ( ( η − 1 ) / ( η − ρ ) ) , ( 2 ) where εupper , n is the critical upper-bracketing radius ( Zhang , 2006 ) . The decay rate of εupper , n controls the consistency of the Bayesian posterior distribution ( Asmuth et al. , 2009 ) . Let ρ = 12 , we have for all t ≥ 0 and δ ∈ ( 0 , 1 ) , with probability at least 1− δ , πn ( { p ∈ Γ : ||p− q||21/2 ≥ 2εn + ( 4η − 2 ) t δ/4 } ∣∣∣X ) ≤ 1 1 + ent . ( 3 ) Proof ( sketch ) . The proof is similar to that of Corollary 5.2 of Zhang ( 2006 ) ( see Appendix A.5 ) . Instead of using the critical prior-mass radius επ , n to describe certain characteristics of the Bayesian prior , we define and use the prior-mass radius dπ , which is independent of the sample size n and measures the distance between the prior and true distribution . Similar to BOSS , for a new MDP m ∼M with hidden parameters ωm , we can define the Bayesian concentration sample complexity for the task-specific posterior : f ( s , a , 0 , δ0 , ρ0 ) , as the minimum number c such that , if c IID transitions from ( s , a ) are observed , then , with probability at least 1− δ0 , Prm∼posterior ( ||Tm ( s , a , ωm ) − Tm∗ ( s , a , ωm ) ||1 < 0 ) ≥ 1− ρ0 . ( 4 ) Lemma 2 . Assume the posterior is consistent ( that is , εupper , n = o ( 1 ) ) and set η = 2 , then the Bayesian concentration sample complexity for the task-specific posterior f ( s , a , , δ , ρ ) = O ( dπ+ln 1 ρ 2δ−dπ ) . Proof ( sketch ) . This bound can be derived by directly combining Lemma 1 and Equation 4 . The above lemma suggests an upper bound of the Bayesian concentration sample complexity using the prior-mass radius . We can further combine this result with PAC-MDP theory ( Strehl et al. , 2006 ) and derive the sample complexity of the algorithm for each new task . Theorem 1 . For each new task , set the sample size K = Θ ( S 2A δ ln SA δ ) and the parameters 0 = ( 1− γ ) 2 , δ0 = δSA , ρ0 = δ S2A2K , then , with probability at least 1− 4δ , V At ( st ) ≥ V ∗ ( st ) − 4 0 in all but Õ ( S 2A2dπ δ 3 ( 1−γ ) 6 ) steps , where Õ ( · ) suppresses logarithmic dependence . Proof ( sketch ) . The proof is based on the PAC-MDP theorem ( Strehl et al. , 2009 ) combined with the new bound for the Bayesian concentration sample complexity we derived in Lemma 2 . In general , we use the same process in BOSS to verify the three required properties of PAC-MDP : optimism , accuracy and learning complexity . For each new task , the main difference between BLRL and BOSS is that we use the world-model posterior to initialize the task-specific posterior , which results in a new sample complexity bound based on the prior-mass radius . The result formalizes the intuition that , if we put a larger prior mass at a density that is close to the true q such that dπ is small , the sample complexity of our algorithm will be lower . In the meantime , the sample complexity is bounded by polynomial functions of the relevant quantities , showing that our training strategy preserves the properties required by PAC-MDP algorithms ( Strehl et al. , 2009 ) .

Summary: The authors proposes a Hierarchical Bayesian approach for lifelong RL. The global world-model posterior models the world model shared across tasks and the task-specific model learns the dynamics within a specific task. The task-specific model achieves forward transfer by initializing from the global world model. The authors use mean-field variational approximation to scale the proposed model. Also, the authors introduce sample complexity analysis. The method is evaluated on two toy tasks (grid-world and box jumping) and one on MuJoCo simulator and showed superior performance to the previous works.