Reset-Free Lifelong Learning with Skill-Space Planning

1 INTRODUCTION . Intelligent agents , such as humans , continuously interact with the real world and make decisions to maximize their utility over the course of their lifetime . This is broadly the goal of lifelong reinforcement learning ( RL ) , which seeks to automatically learn artificial agents that can mimic the continuous learning capabilities of real-world agents . This goal is challenging for current RL algorithms as real-world environments can be non-stationary , requiring the agents to continuously adapt to changing goals and dynamics in robust fashions . In contrast to much of prior work in lifelong RL , our focus is on developing RL algorithms that can operate in non-episodic or “ reset-free ” settings and learn from both online and offline interactions . This setup approximates real-world learning where we might have plentiful logs of offline data but resets to a fixed start distribution are not viable and our goals and environment change . Performing well in this setting is key for developing autonomous agents that can learn without laborious human supervision in non-stationary , high-stakes scenarios . However , the performance of standard RL algorithms drops significantly in non-episodic settings . To illustrate this issue , we first pre-train agents to convergence in the episodic Hopper environment ( Brockman et al. , 2016 ) with state-of-the-art model-free and model-based RL algorithms : Soft Actor Critic ( SAC ) ( Haarnoja et al. , 2018 ) and Model-Based Policy Optimization ( MBPO ) ( Janner et al. , 2019 ) , respectively . These agents are then trained further in a reset-free setting , representing a real-world scenario where agents seek to improve generalization via continuing to adapt at a test time where resets are more expensive . The learning curves are shown in Figure 1 . In spite of near-perfect initialization , all agents proceed to fail catastrophically , suggesting that current gradientbased RL methods are inherently unstable in non-episodic settings . 1Project website and materials : https : //sites.google.com/berkeley.edu/reset-free-lifelong-learning This illustrative experiment complements prior work highlighting other failures of RL algorithms in non-stationary and non-episodic environments : Co-Reyes et al . ( 2020 ) find current RL algorithms fail to learn in a simple gridworld environment without resets and Lu et al . ( 2019 ) find modelfree RL algorithms struggle to learn and adapt to nonstationarity even with access to the ground truth dynamics model . We can attribute these failures to RL algorithms succumbing to sink states . Intuitively , these are states from which agents struggle to escape , have low rewards , and suggest a catastrophic halting of learning progress ( Lyapunov , 1992 ) . For example , an upright walking agent may fall over and fail to return to a standing position , possibly because of underactuated joints . A less obvious notion of sink state we use is that the agent simply fails to escape from it due to low learning signal , which is almost equally undesirable . A lifelong agent must seek to avoid such disabling sink states , especially in the absence of resets . We introduce Lifelong Skill Planning ( LiSP ) , an algorithmic framework for reset-free , lifelong RL that uses long-horizon , decision-time planning in an abstract space of skills to overcome the above challenges . LiSP employs a synergistic combination of model-free policy networks and model-based planning , wherein we use a policy to execute certain skills , planning directly in the skill space . This combination offers two benefits : ( 1 ) skills constrain the search space to aid the planner in finding solutions to long-horizon problems and ( 2 ) skills mitigate errors in the dynamics model by constraining the distribution of behaviors . We demonstrate that agents learned via LiSP can effectively plan for longer horizons than prior work , enabling better long-term reasoning and adaptation . Another key component of the LiSP framework is the flexibility to learn skills from both online and offline interactions . For online learning , we extend Dynamics-Aware Discovery of Skills ( DADS ) , an algorithm for unsupervised skill discovery ( Sharma et al. , 2019 ) , with a skill-practice proposal distribution and a primitive dynamics model for generating rollouts for training . We demonstrate that the use of this proposal distribution significantly amplifies the signal for learning skills in resetfree settings . For offline learning from logged interactions , we employ a similar approach as above but with a modification of the reward function to correspond to the extent of disagreement amongst the models in a probabilistic ensemble ( Kidambi et al. , 2020 ) . Our key contributions can be summarized as follows : • We identify skills as a key ingredient for overcoming the challenges to achieve effective lifelong RL in reset-free environments . • We propose Lifelong Skill Planning ( LiSP ) , an algorithmic framework for reset-free lifelong RL with two novel components : ( a ) a skill learning module that can learn from both online and offline interactions , and ( b ) a long-horizon , skill-space planning algorithm . • We propose new challenging benchmarks for reset-free , lifelong RL by extending gridworld and MuJoCo OpenAI Gym benchmarks ( Brockman et al. , 2016 ) . We demonstrate the effectiveness of LiSP over prior approaches on these benchmarks in a variety of nonstationary , multi-task settings , involving both online and offline interactions . 2 BACKGROUND . Problem Setup . We represent the lifelong environment as a sequence of Markov decision processes ( MDPs ) . The lifelong MDPM is the concatenation of several MDPs ( Mi , Ti ) , where Ti denotes the length of time for which the dynamics of Mi are activated . Without loss of generality , we assume the sum of the Ti ( i.e. , the total environment time ) is greater than the agent ’ s lifetime . The properties of the MDPMi are defined by the tuple ( S , A , Pi , ri , γ ) , where S is the state space , A is the action space , Pi : S ×A×S → R are the transition dynamics , ri : S ×A → R is the reward function , and γ ∈ [ 0 , 1 ) is the discount factor . Consistent with prior work , we assume ri is always known to the agent specifying the task ; it is also easy to learn for settings where it is not known . We use P and r as shorthand to refer to the currentMi with respect to the agent . The agent is denoted by a policy π : S → A and seeks to maximize its expected return starting from the current state s0 : argmaxπ Est+1∼P , at∼π [ ∑∞ t=0 γ tr ( st , at ) ] . The policy π may be implemented as a parameterized function or an action-generating procedure . We expect the agent to optimize for the currentMi , rather than trying to predict the future dynamics ; e.g. , a robot may be moved to an arbitrary new MDP and expected to perform well , without anticipating this change in advance . Skill Discovery . Traditional single-task RL learns a single parametrized policy πθ ( ·|s ) . For an agent to succeed at multiple tasks , we can increase the flexibility of the agent by introducing a set of latent skills z ∈ [ −1 , 1 ] dim ( z ) and learning a skill conditional policy πθ ( ·|s , z ) . As in standard latent variable modeling , we assume a fixed , simple prior over the skills p ( z ) , e.g. , uniform . The learning objective of the skill policy is to maximize some intrinsic notion of reward . We denote the intrinsic reward as r̃ ( s , z , s′ ) to distinguish it from the task-specific reward defined previously . One such intrinsic reward , proposed in DADS ( Sharma et al. , 2019 ) , can be derived from a variational approximation to the mutual information between the skills and next states I ( s′ ; z|s ) as : r̃ ( s , z , s′ ) = log qν ( s ′|s , z ) 1 L ∑L i=1 qν ( s ′|s , zi ) where zi ∼ p ( z ) . ( 1 ) Here qν ( s′|s , z ) is a tractable variational approximation for the intractable posterior p ( s′|s , z ) . Intuitively , this r̃ learns predictable ( via the numerator ) and distinguishable ( via the denominator ) skills . Due to the mutual information objective , DADS also learns skills with high empowerment , which is useful for constraining the space of options for planning ; we discuss this in Appendix C. Model-Based Planning . Whereas RL methods act in the environment according to a parameterized policy , model-based planning methods learn a dynamics model fφ ( st+1|st , at ) to approximate P and use Model Predictive Control ( MPC ) to generate an action via search over the model ( Nagabandi et al. , 2018b ; Chua et al. , 2018 ) . At every timestep , MPC selects the policy π that maximizes the predicted H-horizon expected return from the current state s0 for a specified reward function r : πMPC = argmax π Eat∼π , st+1∼fφ [ H−1∑ t=0 γtr ( st , at ) ] . ( 2 ) We use Model Path Predictive Integral ( MPPI ) ( Williams et al. , 2015 ) as our optimizer . MPPI is a gradient-free optimization method that : ( 1 ) samples policies according to Gaussian noise on the optimization parameters , ( 2 ) estimates the policy returns , and ( 3 ) reweighs policies according to a Boltzmann distribution on the predicted returns . For more details , see Nagabandi et al . ( 2020 ) . For all dynamics models used in this work , we use a probabilistic ensemble ofN models { fφi } N−1i=0 , where each model predicts the mean and variance of the next state . For MPC planning , the returns are estimated via trajectory sampling ( Chua et al. , 2018 ) , where each policy is evaluated on each individual model for the entire H-length rollout and the returns are averaged . For policy optimization , each transition is generated by sampling from a member of the ensemble uniformly at random . 3 LIFELONG SKILL-SPACE PLANNING . In this section , we present Lifelong Skill Planning ( LiSP ) , our proposed approach for reset-free lifelong RL . We provide an outline of LiSP in Algorithm 1 . The agent initially learns a dynamics model fφ and a skill policy πθ from any available offline data . Thereafter , the agent continuously updates the model and policy based on online interactions in the reset-free lifelong environment . The agent uses skill-space planning to act in the environment and avoid sink states . In particular , LiSP learns the following distributions as neural networks : • A primitive dynamics model fφ used for both planning and policy optimization • A low-level skill policy πθ trained from generated model rollouts on intrinsic rewards • A discriminator qν for learning the intrinsic reward using Sharma et al . ( 2019 ) • A skill-practice distribution pψ to generate a curriculum for training skills 3.1 MODEL-BASED SKILL DISCOVERY . Our goal is to learn a skill-conditioned policy πθ ( a|s , z ) . In order to minimize interactions with the environment , we first learn a model fφ to generate synthetic rollouts for policy training . Since there is no start state distribution , the initialization of these rollouts is an important design choice . Algorithm 1 : Lifelong Skill Planning ( LiSP ) Initialize true replay buffer D , generated replay buffer D̂ , dynamics model ensemble { fφi } N−1i=0 , policy πθ , discriminator qν , and skill-practice distribution pψ if performing offline pretraining then Learn dynamics model fφ and train policy πθ with UpdatePolicy until convergence while agent is alive at current state s do Update dynamics model to maximize the log probability of transitions of D Update policy models with UpdatePolicy ( D , D̂ , fφ , πθ , qν , pψ ) Execute action from GetAction ( s , fφ , πθ ) and add environment transition to D We propose to learn a skill-practice distribution pψ ( z|s ) to define which skills to use at a particular state . pψ acts as a “ teacher ” for πθ , automatically generating a curriculum for skill learning . We include visualizations of the learned skills on 2D gridworld environments in Appendix E.2 . To actually learn the policy , we use the model to generate short rollouts , optimizing πθ with SAC , similar to model-based policy learning works that find long rollouts to destabilize learning due to compounding model errors ( Janner et al. , 2019 ) . To initialize the rollout , we sample a state from the replay buffer D and a skill to practice via pψ . The next state is predicted by the dynamics model fφ , where the model used to make the prediction is uniformly sampled from the ensemble . The transition is added to a generated replay buffer D̂ ; gradients do not propagate through fφ . Given minibatches sampled from D̂ , both πθ and pψ are independently trained using SAC to optimize the intrinsic reward r̃adjusted . Intuitively pψ only selects skills which are most useful from the current state , instead of arbitrary skills . This is summarized in Algorithm 2 . Algorithm 2 : Learning Latent Skills Hyperparameters : number of rollouts M , disagreement threshold αthres Function UpdatePolicy ( replay buffer D , generated replay buffer D̂ , dynamics model fφ , policy πθ , discriminator qν , skill-practice distribution pψ ) : for i = 1 to M do Sample si0 uniformly from D and latent zi from skill-practice pψ ( ·|si0 ) Generate si1 : = fφ ( ·|si0 , πθ ( ·|si0 , zi ) ) and add transition ( si0 , ai , zi , si1 ) to D̂ Update discriminator qν on { si0 , ai , zi , si1 } Mi=1 to maximize log qν ( s1|s0 , z ) Calculate intrinsic rewards r̃adjusted for D̂ with qν , αthres using Equations 1 and 3 Update πθ , qν , pψ using SAC with minibatches from D̂

Summary: This paper presents a lifelong reinforcement learning framework in a non-stationary environment with non-episodic interactions. The proposed approach is to 1) learn "skills" - a world model - to maximize the intrinsic rewards using both online and offline data, and to 2) make best plans based on the learned world model. This approach is evaluated with Hopper and Ant tasks.