Reset-Free Lifelong Learning with Skill-Space Planning

1 INTRODUCTION . Intelligent agents , such as humans , continuously interact with the real world and make decisions to maximize their utility over the course of their lifetime . This is broadly the goal of lifelong reinforcement learning ( RL ) , which seeks to automatically learn artificial agents that can mimic the continuous learning capabilities of real-world agents . This goal is challenging for current RL algorithms as real-world environments can be non-stationary , requiring the agents to continuously adapt to changing goals and dynamics in robust fashions . In contrast to much of prior work in lifelong RL , our focus is on developing RL algorithms that can operate in non-episodic or “ reset-free ” settings and learn from both online and offline interactions . This setup approximates real-world learning where we might have plentiful logs of offline data but resets to a fixed start distribution are not viable and our goals and environment change . Performing well in this setting is key for developing autonomous agents that can learn without laborious human supervision in non-stationary , high-stakes scenarios . However , the performance of standard RL algorithms drops significantly in non-episodic settings . To illustrate this issue , we first pre-train agents to convergence in the episodic Hopper environment ( Brockman et al. , 2016 ) with state-of-the-art model-free and model-based RL algorithms : Soft Actor Critic ( SAC ) ( Haarnoja et al. , 2018 ) and Model-Based Policy Optimization ( MBPO ) ( Janner et al. , 2019 ) , respectively . These agents are then trained further in a reset-free setting , representing a real-world scenario where agents seek to improve generalization via continuing to adapt at a test time where resets are more expensive . The learning curves are shown in Figure 1 . In spite of near-perfect initialization , all agents proceed to fail catastrophically , suggesting that current gradientbased RL methods are inherently unstable in non-episodic settings . 1Project website and materials : https : //sites.google.com/berkeley.edu/reset-free-lifelong-learning This illustrative experiment complements prior work highlighting other failures of RL algorithms in non-stationary and non-episodic environments : Co-Reyes et al . ( 2020 ) find current RL algorithms fail to learn in a simple gridworld environment without resets and Lu et al . ( 2019 ) find modelfree RL algorithms struggle to learn and adapt to nonstationarity even with access to the ground truth dynamics model . We can attribute these failures to RL algorithms succumbing to sink states . Intuitively , these are states from which agents struggle to escape , have low rewards , and suggest a catastrophic halting of learning progress ( Lyapunov , 1992 ) . For example , an upright walking agent may fall over and fail to return to a standing position , possibly because of underactuated joints . A less obvious notion of sink state we use is that the agent simply fails to escape from it due to low learning signal , which is almost equally undesirable . A lifelong agent must seek to avoid such disabling sink states , especially in the absence of resets . We introduce Lifelong Skill Planning ( LiSP ) , an algorithmic framework for reset-free , lifelong RL that uses long-horizon , decision-time planning in an abstract space of skills to overcome the above challenges . LiSP employs a synergistic combination of model-free policy networks and model-based planning , wherein we use a policy to execute certain skills , planning directly in the skill space . This combination offers two benefits : ( 1 ) skills constrain the search space to aid the planner in finding solutions to long-horizon problems and ( 2 ) skills mitigate errors in the dynamics model by constraining the distribution of behaviors . We demonstrate that agents learned via LiSP can effectively plan for longer horizons than prior work , enabling better long-term reasoning and adaptation . Another key component of the LiSP framework is the flexibility to learn skills from both online and offline interactions . For online learning , we extend Dynamics-Aware Discovery of Skills ( DADS ) , an algorithm for unsupervised skill discovery ( Sharma et al. , 2019 ) , with a skill-practice proposal distribution and a primitive dynamics model for generating rollouts for training . We demonstrate that the use of this proposal distribution significantly amplifies the signal for learning skills in resetfree settings . For offline learning from logged interactions , we employ a similar approach as above but with a modification of the reward function to correspond to the extent of disagreement amongst the models in a probabilistic ensemble ( Kidambi et al. , 2020 ) . Our key contributions can be summarized as follows : • We identify skills as a key ingredient for overcoming the challenges to achieve effective lifelong RL in reset-free environments . • We propose Lifelong Skill Planning ( LiSP ) , an algorithmic framework for reset-free lifelong RL with two novel components : ( a ) a skill learning module that can learn from both online and offline interactions , and ( b ) a long-horizon , skill-space planning algorithm . • We propose new challenging benchmarks for reset-free , lifelong RL by extending gridworld and MuJoCo OpenAI Gym benchmarks ( Brockman et al. , 2016 ) . We demonstrate the effectiveness of LiSP over prior approaches on these benchmarks in a variety of nonstationary , multi-task settings , involving both online and offline interactions . 2 BACKGROUND . Problem Setup . We represent the lifelong environment as a sequence of Markov decision processes ( MDPs ) . The lifelong MDPM is the concatenation of several MDPs ( Mi , Ti ) , where Ti denotes the length of time for which the dynamics of Mi are activated . Without loss of generality , we assume the sum of the Ti ( i.e. , the total environment time ) is greater than the agent ’ s lifetime . The properties of the MDPMi are defined by the tuple ( S , A , Pi , ri , γ ) , where S is the state space , A is the action space , Pi : S ×A×S → R are the transition dynamics , ri : S ×A → R is the reward function , and γ ∈ [ 0 , 1 ) is the discount factor . Consistent with prior work , we assume ri is always known to the agent specifying the task ; it is also easy to learn for settings where it is not known . We use P and r as shorthand to refer to the currentMi with respect to the agent . The agent is denoted by a policy π : S → A and seeks to maximize its expected return starting from the current state s0 : argmaxπ Est+1∼P , at∼π [ ∑∞ t=0 γ tr ( st , at ) ] . The policy π may be implemented as a parameterized function or an action-generating procedure . We expect the agent to optimize for the currentMi , rather than trying to predict the future dynamics ; e.g. , a robot may be moved to an arbitrary new MDP and expected to perform well , without anticipating this change in advance . Skill Discovery . Traditional single-task RL learns a single parametrized policy πθ ( ·|s ) . For an agent to succeed at multiple tasks , we can increase the flexibility of the agent by introducing a set of latent skills z ∈ [ −1 , 1 ] dim ( z ) and learning a skill conditional policy πθ ( ·|s , z ) . As in standard latent variable modeling , we assume a fixed , simple prior over the skills p ( z ) , e.g. , uniform . The learning objective of the skill policy is to maximize some intrinsic notion of reward . We denote the intrinsic reward as r̃ ( s , z , s′ ) to distinguish it from the task-specific reward defined previously . One such intrinsic reward , proposed in DADS ( Sharma et al. , 2019 ) , can be derived from a variational approximation to the mutual information between the skills and next states I ( s′ ; z|s ) as : r̃ ( s , z , s′ ) = log qν ( s ′|s , z ) 1 L ∑L i=1 qν ( s ′|s , zi ) where zi ∼ p ( z ) . ( 1 ) Here qν ( s′|s , z ) is a tractable variational approximation for the intractable posterior p ( s′|s , z ) . Intuitively , this r̃ learns predictable ( via the numerator ) and distinguishable ( via the denominator ) skills . Due to the mutual information objective , DADS also learns skills with high empowerment , which is useful for constraining the space of options for planning ; we discuss this in Appendix C. Model-Based Planning . Whereas RL methods act in the environment according to a parameterized policy , model-based planning methods learn a dynamics model fφ ( st+1|st , at ) to approximate P and use Model Predictive Control ( MPC ) to generate an action via search over the model ( Nagabandi et al. , 2018b ; Chua et al. , 2018 ) . At every timestep , MPC selects the policy π that maximizes the predicted H-horizon expected return from the current state s0 for a specified reward function r : πMPC = argmax π Eat∼π , st+1∼fφ [ H−1∑ t=0 γtr ( st , at ) ] . ( 2 ) We use Model Path Predictive Integral ( MPPI ) ( Williams et al. , 2015 ) as our optimizer . MPPI is a gradient-free optimization method that : ( 1 ) samples policies according to Gaussian noise on the optimization parameters , ( 2 ) estimates the policy returns , and ( 3 ) reweighs policies according to a Boltzmann distribution on the predicted returns . For more details , see Nagabandi et al . ( 2020 ) . For all dynamics models used in this work , we use a probabilistic ensemble ofN models { fφi } N−1i=0 , where each model predicts the mean and variance of the next state . For MPC planning , the returns are estimated via trajectory sampling ( Chua et al. , 2018 ) , where each policy is evaluated on each individual model for the entire H-length rollout and the returns are averaged . For policy optimization , each transition is generated by sampling from a member of the ensemble uniformly at random . 3 LIFELONG SKILL-SPACE PLANNING . In this section , we present Lifelong Skill Planning ( LiSP ) , our proposed approach for reset-free lifelong RL . We provide an outline of LiSP in Algorithm 1 . The agent initially learns a dynamics model fφ and a skill policy πθ from any available offline data . Thereafter , the agent continuously updates the model and policy based on online interactions in the reset-free lifelong environment . The agent uses skill-space planning to act in the environment and avoid sink states . In particular , LiSP learns the following distributions as neural networks : • A primitive dynamics model fφ used for both planning and policy optimization • A low-level skill policy πθ trained from generated model rollouts on intrinsic rewards • A discriminator qν for learning the intrinsic reward using Sharma et al . ( 2019 ) • A skill-practice distribution pψ to generate a curriculum for training skills 3.1 MODEL-BASED SKILL DISCOVERY . Our goal is to learn a skill-conditioned policy πθ ( a|s , z ) . In order to minimize interactions with the environment , we first learn a model fφ to generate synthetic rollouts for policy training . Since there is no start state distribution , the initialization of these rollouts is an important design choice . Algorithm 1 : Lifelong Skill Planning ( LiSP ) Initialize true replay buffer D , generated replay buffer D̂ , dynamics model ensemble { fφi } N−1i=0 , policy πθ , discriminator qν , and skill-practice distribution pψ if performing offline pretraining then Learn dynamics model fφ and train policy πθ with UpdatePolicy until convergence while agent is alive at current state s do Update dynamics model to maximize the log probability of transitions of D Update policy models with UpdatePolicy ( D , D̂ , fφ , πθ , qν , pψ ) Execute action from GetAction ( s , fφ , πθ ) and add environment transition to D We propose to learn a skill-practice distribution pψ ( z|s ) to define which skills to use at a particular state . pψ acts as a “ teacher ” for πθ , automatically generating a curriculum for skill learning . We include visualizations of the learned skills on 2D gridworld environments in Appendix E.2 . To actually learn the policy , we use the model to generate short rollouts , optimizing πθ with SAC , similar to model-based policy learning works that find long rollouts to destabilize learning due to compounding model errors ( Janner et al. , 2019 ) . To initialize the rollout , we sample a state from the replay buffer D and a skill to practice via pψ . The next state is predicted by the dynamics model fφ , where the model used to make the prediction is uniformly sampled from the ensemble . The transition is added to a generated replay buffer D̂ ; gradients do not propagate through fφ . Given minibatches sampled from D̂ , both πθ and pψ are independently trained using SAC to optimize the intrinsic reward r̃adjusted . Intuitively pψ only selects skills which are most useful from the current state , instead of arbitrary skills . This is summarized in Algorithm 2 . Algorithm 2 : Learning Latent Skills Hyperparameters : number of rollouts M , disagreement threshold αthres Function UpdatePolicy ( replay buffer D , generated replay buffer D̂ , dynamics model fφ , policy πθ , discriminator qν , skill-practice distribution pψ ) : for i = 1 to M do Sample si0 uniformly from D and latent zi from skill-practice pψ ( ·|si0 ) Generate si1 : = fφ ( ·|si0 , πθ ( ·|si0 , zi ) ) and add transition ( si0 , ai , zi , si1 ) to D̂ Update discriminator qν on { si0 , ai , zi , si1 } Mi=1 to maximize log qν ( s1|s0 , z ) Calculate intrinsic rewards r̃adjusted for D̂ with qν , αthres using Equations 1 and 3 Update πθ , qν , pψ using SAC with minibatches from D̂

Summary: The authors propose LiSP, a model-based planning method that performs model-predictive control using learned skills rather than actions. The skills are learned using DADS, with a modified reward function that additionally encourages all skills to stay within the support of training data to avoid sink states. The experiment results show stable learning progress on reset-free and ever-changing targets, compared to other baselines.