Reset-Free Lifelong Learning with Skill-Space Planning
Authors: Kevin Lu
Aditya Grover
Pieter Abbeel
Igor Mordatch
Venue: ICLR 2021 Poster (2021)
Keywords: reset-free
lifelong
reinforcement learning

Abstract: The objective of \textit{lifelong} reinforcement learning (RL) is to optimize agents which can continuously adapt and interact in changing environments. However, current RL approaches fail drastically when environments are non-stationary and interactions are non-episodic. We propose \textit{Lifelong Skill Planning} (LiSP), an algorithmic framework for lifelong RL based on planning in an abstract space of higher-order skills. We learn the skills in an unsupervised manner using intrinsic rewards and plan over the learned skills using a learned dynamics model. Moreover, our framework permits skill discovery even from offline data, thereby reducing the need for excessive real-world interactions. We demonstrate empirically that LiSP successfully enables long-horizon planning and learns agents that can avoid catastrophic failures even in challenging non-stationary and non-episodic environments derived from gridworld and MuJoCo benchmarks.

Published as a conference paper at ICLR 2021
RESET-FREE LIFELONG LEARNING WITH SKILL-SPACE PLANNING
Kevin Lu
Aditya Grover
UC Berkeley kz18berkeley.edu
UC Berkeley adityagecs sta.esedssstannoo..dd...
Pieter Abbeel
Igor Mordatch
UC Berkeley
Google Brain
pabbeelêcs, www.k/cccbbereey..e... edu
imordat com
ABSTRACT
The objective of lifelong reinforcement learning (RL) to optimize agents which can continuously adapt and interact in changing environments. However, current RL approaches fail drastically when environments are non-stationary and interac- tions are non-episodic. We propose Lifelong Skill Planning (LiSP), an algorithmic framework for non-episodic lifelong RL based on planning an abstract space higher-order skills. We learn the skills in unsupervised manner using intrinsic rewards and plan over the learned skills using a learned dynamics model. More- over, our framework permits skill discovery even from offline data, thereby reduc- ing the need for excessive real-world interactions. We demonstrate empirically that LiSP successfully enables long-horizon planning and learns agents that avoid catastrophic failures even in challenging non-stationary and non-episodic environments derived from gridworld and MuJoCo benchmark
INTRODUCTION
Intelligent agents, such as humans, continuously interact with the real world and make decisions to maximize their utility over the course of their lifetime. This is broadly the goal of lifelong reinforce- ment learning (RL), which seeks to automatically learn artificial agents that can mimic the continu- ous learning capabilities of real-world agents. This goal is challenging for current RL algorithms as real-world environments can be non-stationary, requiring the agents to continuously adapt to chang- ing goals and dynamics in robust fashions. contrast much of prior work in lifelong RI,, our focus is on developing RI. algorithms that can operate in non-episodic or "reset- freset-free" settings and learn from both online and offline interactions. This setup approximates real-world learning where might have plentiful logs of offline data but resets fixed start distribution are not viable and our goals and environment change. Performing well in this setting is key for developing autonomous agents that can learn without laborious human supervision in non-stationary, high high-stakes scenarios.
However, the performance of standard RL algorithms drops signif- icantly in non-episodic settings. To illustrate this issue, first pre-train agents to convergence in the episodic Hopper environ- ment (Brockman al. 2016) with state-of-the-art model-free and model-based RL algorithms: Soft Actor Critic (SAC) (Haarnoja al., 2018) and Model-Based Policy Optimization (MBPO) (Jan- ner et al.] 2019), respectively. These agents are then trained fur- ther a reset-free setting, representing a wewl-eawoore world scenario where agents seek to improve generalization via continuing to adapt at a test time where resets are more expensive. The learning curves are shown in Figure [ In spite of near- near-perfect initialization, all agents proceed to fail catastrophically, suggesting that current gradient- based RL methods are inherently unstable non-episodic settings.
Lifelong Hopper
SAC
MBPO
Reward
-15
3020 4000 5000 6020
Timesteps
Figure 1: RL without plan- ning fails without resets. Each line is one seed. The red line shows reward with no updates (i.e. frozen weights).
Project website and materials: http:///wc....oooooommeeeoeoooee

---

Published as a conference paper at ICLR 2021
This illustrative experiment complements prior work highlighting other failures of RL algorithms in non-stationary and non-episodic environments: Co-RRyes et al. (2020) find current RL algorithms fail to learn in simple gridworld environment without resets and Lu et al. (2019) find model- free RI. algorithms struggle to learn and adapt to nonstationarity even with access to the ground truth dynamics model. We can attribute these failures to RL. algorithms succumbing to sink states. Intuitively, these are states from which agents struggle to escape, have low rewards, and suggest catastrophic halting of learning progress (Lyapunov 1992). For example, an upright walking agent may fall over and fail to return to a standing position. possibly because of underactuated joints. A less obvious notion of sink state we use is that the agent simply fails to escape from it due to low learning signal, which is almost equally undesirable. A lifelong agent must seek to avoid such disabling sink states, especially in the absence of resets.
We introduce Lifelong Skill Planning (LiSP), an algorithmic framework for reset-free, lifelong RI. that uses long-horizon, decision-time planning in an abstract space of skills to overcome the above challenges. LiSP employs a synergistic combination of model-free policy networks and model-based planning, wherein we use a policy to execute certain skills, planning directly in the skill space. This combination offers two benefits: (I) skills constrain the search space to aid the planner in finding solutions to aan-horizon problems and (2) skills mitigate errors in the dynamics model by con- straining the distribution of behaviors. We demonstrate that agents learned via LiSP can effectively plan for longer horizons than prior work, enabling better long-term reasoning and adaptation.
Another key component of the LiSP framework is the flexibility learn skills from both online and offline interactions. For online learning, we extend Ammamccs-Awar Discovery of Skills (DADS), an algorithm for unsupervised skill discovery (Sharma et al., 2019), with skill-practice proposal distribution and a primitive dynamics model for generating rollouts for training. We demonstrate that the use of this proposal distribution significantly amplifies the signal for learning skills in reset- free settings. For offline learning from logged interactions, we employ a similar approach as above but with a modification of the reward function to correspond to the extent of disagreement amongst the models in probabilistic ensemble (Kidambi et al., 2020).
Our key contributions can be summarized as follows:
We identify skills as a key ingredient for overcoming the challenges to achieve effective lifelong RL in reset-free environments.
We propose Lifelong Skill Planning (LiSP), an algorithmic framework for reset-free life- long RL with two novel components: (a) a skill learning module that can learn from both online and offline interactions, and (b) a long-horizon, skill-space planning algorithm.
We propose new challenging benchmarks for reset-free, lifelong RL by extending grid- world and MuJoCo OpenAl Gym benchmarks (Brockman et al. 2016). We demonstrate the effectiveness of LiSP over prior approaches on these benchmarks in a variety of non- stationary, multi-task settings, involving both online and offline interactions.
2 BACKGROUND
Problem Setup. We represent the lifelong environment as sequence of Markov decision processes (MDPs). The lifelong MDP M is the concatenation of several MDPs (M,, where T; denotes the length of time for which the dynamics of M, are activated. Without loss of generality, we assume the sum of the T; (i.e., the total environment time) is greater than the agent's lifetime. The properties of the MDP M2 are defined by the tuple (S, P., Pt.T)) where S is the state space, A is the action space, P.: S x A x S - lR are the transition dynamics, r; S x A + IR is the reward function, and 7 E [0. is the discount factor. Consistent with prior work, we assume r; always known to the agent specifying the task; it is also easy to learn for settings where it is not known. We use P and r as shorthand to refer to the current M, with respect to the agent.
The agent is denoted by policy : S - A and seeks to maximize its expected return starting from the current state sy: arg max, Ee Paai- 1 y'r(s,, a.).. The policy π may be implemented procedure. We expect the agent optimize for
as parameterized function an oction-generatie the current M,, rather than trying to predict the future dynamics; e.g.. a robot may be moved to an arbitrary new MDP and expected to perform well, without anticipating this change in advance.
2

---

Published as a conference paper at ICLR 2021
Skill Discovery. Traditional single-task RL learns single parametrized policy Te(-Is). For an agent to succeed at multiple tasks, we can increase the flexibility of the agent by introducing a set of latent skills z E [_ 1]dim(()) and learning a skill conditional policy no no (-(1s, z). As in standard latent variable modeling, we assume a fixed, simple prior over the skills p(z), e.g., uniform. The learning objective of the skill policy is to maximize some intrinsic notion of reward. We denote the intrinsic reward as F(s, 2, s') distinguish from the task-specific reward defined previously. One such intrinsic reward, proposed in DADS (Sharma et al., (201)), can be derived from a variational approximation the mutual information between the skills and next states [(5; 228) as:
2(8,2,8') = log ( 2)
where 2; ~ p(z).
I E (6)))18,))))
Here (8'18, 2) is a tractable variational approximation for the intractable posterior p(51|,, Intu- itively, this F learns predictable (via the numerator) and distinguishable (via the denominator) skills. Due to the mutual information objective, DADS also learns skills with high empowerment, which is useful for constraining the space options for planning; we discuss this in Appendix C
Model-Based Planning. Whereas RL methods act in the environment according to a parameterized policy, model-based planning methods learn dynamics model fo [0(#141|8) at) approximate P and use Model Predictive Control (MPC) to generate an action via search over the model (Nagabandi al. 20186; Chua al.) 2018). At every timestep, MPC selects the policy ı that maximizes the predicted H-horizon expected return from the current state for a specified reward function r:
π MPC arg max E S8 fi
Σ rr(s),
We use Model Path Predictive Integral (MPPI) (Williams et al. 2015) as our optimizer. MPPI is a gradient-free optimization method that: (1) samples policies according to Gaussian noise on the optimization parameters, (2) estimates the policy returns, and (3) reweighs policies according to Boltzmann distribution on the predicted returns. For more details, see Nagabandi et al. (2020).
For all dynamics models used this work, we use a probabilistic ensemble of N models ) V where each model predicts the mean and variance of the next state. For MPC planning, the returns are estimated via trajectory sampling (Chua et al., 2018), where each policy is evaluated on each in- dividual model for the entire H H-length rollout and the returns averaged. For policy optimization, each transition is generated by sampling from a member of the ensemble uniformly at random.
3 LIFELONG SKILL-SPACE PLANNING
In this section, we present Lifelong Skill Planning (LiSP), our proposed approach for reset-free lifelong RL. We provide an outline of LiSP Algorithm [ The agent initially learns dynamics model fø and a skill policy тę from any available offline data. Thereafter, the agent continuously updates the model and policy based on online interactions in the reset-free lifelong environment. The agent uses skill-space planning to act in the environment and avoid sink states. In particular, LiSP learns the following distributions as neural networks:
A primitive dynamics model fo used for both planning and policy optimization
A low-level skill policy vo trained from generated model rollouts intrinsic rewards
A discriminator qu for learning the intrinsic reward using Sharma et al. (2019)
A skill-practice distribution pe to generate curriculum for training skills
3.1 MODEL-BASED SKILL DISCOVERY
Our goal is to leam skill-conditioned policy Te(a|s, z). In order to minimize interactions with the environment, we first learn a model fa to generate synthetic rollouts for policy training. Since there is start state distribution, the initialization of these rollouts is an important design choice.
3

---

Published as a conference paper at ICLR 2021
Algorithm 1: Lifelong Skill Planning (LiSP)
Initialize true replay buffer generated replay buffer D. dynamics model ensemble
{fe: N policy Te, discriminator gu, and skill skill-practice distribution p performing offline pretraining then
Learn dynamics model I and train policy with UpdatePoll cy until convergence
while agent is alive at current state s do
Update dynamics model to maximize the log probability of transitions of D
Update policy models with UpdatePolicy (D, DD fa. To, 9u. Pv)
Execute action from GetAction (s, fa. To) and add environment transition
We propose to learn a skill-practice distribution (z(s)) (ss to define which skills to use at particular state. Pe acts as a "teacher" for iTg. automatically generating a curriculum for skill learning. We include visualizations of the learned skills on 2D gridworld environments in Appendix H.2
To actually learn the policy, we use the model to generate short rollouts, optimizing Zy with SAC, similar to model-based policy learning works that find long rollouts to destabilize learning due to compounding model errors (Janner al. 2019). To initialize the rollout, we sample a state from the replay buffer DD and skill to practice via pv- The next state is predicted by the dynamics model fø. where the model used to make the prediction is uniformly sampled from the ensemble. The transition is added to a generated replay buffer D; gradients do not propagate through fø- Given minibatches sampled from D, both xo and Py: independently trained using SAC to optimize the intrinsic reward Fodjusted- Intuitively Pv only selects skills which are most useful from the current state, instead of arbitrary skills. This summarized in Algorithm 2
Algorithm 2: Learning Latent Skills
Hyperparameters: number of rollouts M, disagreement threshold Oxhires Function UpdatePolic i cy (replay buffer D, generated replay buffer D, dynamics model fo. policy Na. discriminator qu skill-practice distribution pa):
for i = M
Sample si uniformly from 'D and latent z' from skill-practice Più (-1ss)
Generate s := fo(-1s) To( [o(- s/. 2²)) and add transition (s6- a², 2", s{)
Update discriminator qu on {s). a², 2, si]
maximize log (s) (o.
Calculate intrinsic rewards Fadjusted for DD with qv, Athres using Equations
and 3
Update To, 42 Ps: using SAC with minibatches from D
3.1.1 OFFLINE SKILL LEARNING
A key problem in offline RL is avoiding value overestimation, typically tackled via constraining actions to the support of the dataset. We use the same algorithm to learn skills offline with simple adjustment to 7 based on the model disagreement (Kidambi et al. 2020). For hyperpa- rameters K. atthres E R+ we replace the intrinsic reward 7 with Fadjusted- penalizing F with -x the model disagreement exceeds than Othres- This penalty encourages the policy to stay within the support of the dataset by optimizing against an MDP which underestimates the value function. We approximate the expected la disagreement in the mean prediction of the next state, denoted Po, for model i, with a sample. This penalty captures the epistemic uncertainty and shown Equation 3
F
dis(s, a) E Esgill Me, (s.a)]1)) S atthres
Padjustedd
33)
-K
dis(s, a) > Otthees
3.2 PLANNING IN SKILL SPACE FOR RESET-FREE ACTING
As described in Section I we argue that the failures of RL in the lifelong setting arise chiefly from the naive application of model-free RI. In particular, it is imperative that the model not only be used
4

---

subset of Published as a conference paper at ICLR 2021
Skill Discriminator
Algorithm 2 Canerila synthabic rollouts for training Train policy on intrinsic rewards using roliouia
Dynamica Model Linsemblle
Algorithm: 3
Plan skill space to act In the environment
Skill- Practice Distribution p.
Skili Policy ",,
Incoming der training from data
Figure 2: Schematic for Lifelong Skill Planning (LiSP). LiSP learns set of skills using synthetic model rollouts and performs long-horizon planning in the skill-space for stable, safe lifelong acting.
for sample-efficient policy optimization (as in MBPO), but also that the model be used to safely act in the environment. The method in which acting is performed is important, serving both exploit what the agent has learned and crucially to maintain the agent's s safety in reset-free settings.
In this work, we propose to model-based planning via MPC. We differ from prior MPC works in that we plans with the set of skills from Section 3.1 which allow us to utilize the broad capabilities of model-free RL while still enabling the benefits of model-based planning, namely exploration and long-houe Long-horizon reasoning. The skills act prior for interesting options that the planner can utilize, seeking meaningful changes in the state. Also, the skills constrain the planner the action space which the policy is confident in. The model-free policy learns to act in a reliable manner and consequently more predictable and robust than naive actions. As result, we are able to perform accurate planning for longer horizons than before (Chua et al.] [018]] Nagabandi et al.) p020), Note that the complexity of our proposed approach is the same as prior MPC works, i.c. is linear in the length of the horizon. We summarize this subroutine in Algorithm 3
Algorithm 3: Skill-Space Planning
Hyperparameeees: population size S, planning horizon H, planning iterations P. discount y Function GetAct ion (current state so, dynamics model fo, policy mo):
for P planning iterations do
Sample skills {zt}² ~ 1] based on distribution of previous iteration
Estimate returns = {E'= >'r(s/, (-(s 21), ( using trajectory sampling, with states s² sampled from fo for skills z
Use MPPI update rule on A and z to generate new distribution of skills { #1-1 return a ~ To (-1s, 20)
Skills are first learned via Algorithm wherein the skill discriminator generates the intrinsic rewards and the skill- dkill-practcce distribution generates skill curriculum. We then plan using the skill policy and the dynamics model as per Algorithm
We summarize the LiSP subroutines in Figure 2
EXPERIMENTAL EVALUATIONS
We wish investigate the following questions with regards to the design and performance of LiSP:
Does LiSP learn effectively in lifelong benchmarks with sink states and nonstationarity?
What are the advantages of long-horizon skill-space planning over action-space planning?
Why is the skill-practic distribution important for learning in this setting?
Does LiSP learn a suitable set of skills fully offline for future use?
I Lifelong learning benchmarks. We extend several previously studied environments to the lifelong RL setting and distinguish these benchmarks with the prefix Liffelong. The key differences are that these environments have an online non-episodic phase where the agent must learn without resets and (optionally) an offline pretraining starting phase. The agent's 's replay buffer D is initialized with a starting dataset of transitions, allowing the agent have meaningful behavior from the outset. More details are in Appendix E and we open source the implementations for wider use. Unless stated otherwise, run 3 seeds for each experiment and plots show the mean and standard deviation.
Comparisons. We compare against select otate-of-the-ar algorithms to help illustrate the value of different design choices. The closest baseline ours is action-space MPC, also known as
5

---

Published as a conference paper at ICLR 2021
PETS (Chua et al. 2018). which directly ablates for the benefit of using skills for planning. We also compare to SAC (Haamojj et al. 2018), representing the performance of model-free RL on these tasks. Finally we consider MOReL (Kidambi et al. 2020). the offline model-based algorithm which proposed the disagreement penalty discussed in Section 3.1.1] Note that the latter two are principally single-task algorithms, while MPC is more suited for multiple reward functions. We believe these represent a diverse set of algorithms that could be considered for the reset-free setting,
4.1 EVALUATION ON LIFELONG BENCHMARKS
We begin by evaluating the overall LiSP framework (Algorithm I in non-episodic RL settings, par- ticularly how LiSP interacts with sink states and its performance in nonstationary reset-free settings.
Nonstationary Mujoco locomotion tasks. We evaluate LiSP on Hopper and Ant tasks from Gym [Brockman et al. 2016); we call these Lifelong Hopper and Lifelong Ant. The agents seek to achieve a target forward x- x-velocity, which changes over the agent's lifetime. Learning curves shown in Figure 3 Most of the LiSP seeds remain stable despite sink states and adapt instantly to the current tasks. As predicted by Figure [ the SAC and MOReL. agents are unstable and do poorly, fundamentally because their gradient updates are not stable. The long-horizon planning capabil- ity of LiSP also crucial, enabling LiSP outperform SAC and MOReL which lack the ability to plan. Furthermore, planning over the space of skills is necessary to achieve improvement over MPC, which is not capable of accurate planning on its own. LiSP outperforms all of the baselines tested.
Lifelong Hopper Performance
Lifelong Ant Performance
Perforaance
1.0
Performance
1.0
0.5
1000
2000
3000
4000
5000
500
1000
1500
2000
2500
3003
Timesteps
Timesteps
LiSP
SAC
MPC
MOReL
Figure 3: Learning without resets. Vertical lines denote task changes. Each blue line represents one seed of LiSP out of 5: the other algorithms have lower variance (since they fail), so we only show the mean of 3 seeds. Performance is normalized against (for more details, see Appendix E.I.I.
Minimizing resets with permanent sink states.
Resets in Volcano Env
We evaluate each method in lifelong 2D volcano gridworld environment where the agent navigates reach goals while avoiding pitfalls which perma- nently trap the agent if there is no intervention. Ev- ery 100 timesteps, the pitfalls and goals rearrange, which allow the agent to get untrapped; we consider this a "reset" if the agent was stuck as it required this intervention. We use limited pretraining for this en- vironment, i.e. we train the model but not the policies, reflecting the agent's ability to act safely while not fully trained. Figure[4showe the number of times the agent got stuck during training. We find all the model- methods, including LiSP, are easily capable of avoiding resets, suggesting model-based algorithms naturally suited for these settings, incorporating data to avoid failures.
Resets
20
30
0.2
0.4
0.6
0.8
1,0
Timesteps
x10*
MPC
MOReL
Figure 4: 2D Volcano environment.
4.2 ADVANTAGES OF LONG HORIZON SKILL-SPACE PLANNING
Next, we seek to clearly show the advantages of planning in the skill space, demonstrating the benefits of using skills in Algorithm Bnd arguing for the use of skills more broadly in lifelong RL.
Constraining model error. We can interpret skill-space planning as constraining the MPC opti- mization to more accurate subset of the action space which the policy operates in; this added accuracy is crucial for the improvement of LiSP vs MPC in the MuJoCo tasks. In Figure 5 we consider Lifelong Hopper and look the one-step dynamics model error when evaluated on actions generated by the skill policy vs uniformly from the action space. Note this requires privileged access to the simulator we cannot use these signals directly for planning. While most samples from both

---

Published as a conference paper at ICLR 2021
have similar model error, the variance in the residual error for random actions is high, exhibiting long tail. Even if the model is accurate for most samples, MPC can fatally overestimate on this tail.
Lifelong Hopper Prediction Error
skills
actions
AND 00 +400NN 040
0.00
0.02
0.04
0.06
0.08
Observation Error
Figure 5: & error in the next state prediction of the dynamics model f for actions sampled from the skill policy vs. uniformly at random. The error variance is greatly reduced by only using skills.
Constraining planning search space. This advantage in the one-step model error extends to long- horizon planning. In the Lifelong Hopper environment, accurate long-horizon planning is critical in order to correctly execute a hopping motion, which has traditionally made MPC difficult this environment. For good performance, ablate LiSP by planning with actions instead of skills (i.e. MPC), Accurate planning with actions is completely infeasible due the accumulated unconstrained model errors.
long horizon of 180 for planning. In Figure 6
Lifelong Hopper Stability
Lifelong Hopper Velocity
ccordinate
skill space
action space
0.00
0.50
0.75
1.25
3.75
0.25
0.75
1.00
2.25
1.50
1.75
Timesteps
K1P
Timesteps
x107
Figure 6: Long-horizon planning in Lifelong Hopper with skills instead of actions. Action-space planning is not stable with long horizons. The dashed red line denotes the target velocity.
Model updates are extremely stable. Furthermore, described in Figure found that SAC and MBPO fail in reset-free settings when denied access to resets they continue gradient updates, which attributed gradient-based instability. Here we try understand the stability of RL algorithms. In particular, since we found that policy/critic updates are unstable, we can instead consider how model updates affect planning. To do so, we consider SAC as before, as well as action-space MPC over a short 5-step horizon trying to optimize the same SAC critic (tied to the policy); we note this similar to [Sikchi al. (2020). The results are shown in Figure 7 If only the model is updated, planning is very stable, showing the resilience of planning to model updates; this best resembles LiSP, which does not rely on a policy critic for plamning. Alternatively, value function updates make planning unstable, supporting the idea that long-horizon planning improves stability vs relying on a value function. However, even the short 5-step planner avoids catastrophic failures, as the additional model stability quickly neutralizes value instability.
Lifelong Hopper with MPC Planning
Method
Reward
MPC (both model and policy updates)
MPC (only model updates)
SAC (policy updates)
0.4
Timesteps
x10
Figure 7: Pretrained action-space MPC agent with terminal value function placed in reset-free setting with continued gradient updates to the model and policy/value. Unlike the naive SAC agent that according greedy "I "I-step" planning, the MPC agent plans the same critic over short 5-step horizon and mostly avoids failure. Note this graph shows the same setting as Figure

---

Published as a conference paper at ICLR 2021
4.3 ONLINE SKILL LEARNING WITH A MODEL
In this section, we show the importance of the skill-practice distribution from Algorithm 2 for learn- ing setups that require hierarchical skills and involve short model-based rollouts. The skill-practice distribution is crucial for improving learning signal, generating useful skills for planning.
Hierarchical skill learning. We consider 2D Minecraft environment where the agent must learn hierarchical skills to build tools, which in turn are used to create higher level tools. We ablate against minimizing F Fadjusted and a baseline of using a practice distribution; the former acts as a sanity check whereas the latter represents the previous convention in skill discovery literature. Note that we do not compare against MOReL here since the asymptotic performance will be similar SAC, as they both rely Q-learning policy updates. Similar to the volcano environment, we only do limited pretraining on a small set of starting data. The learning curves are in Figure g Learning skills with directed practice sampling is necessary for learning useful skills. Without the skill-practice curriculum, the asymptotic performance of LiSP is significantly limited. Additionally, the fact that MPC performs significantly worse than LiSP suggests that planning in the skill space namely, planning in a space expressive skills greatly aids the search process and exploration.
2D Minecraft Rewards
Method
Return
sampling to maximize intrinsic reward sampling
reward sampling skills uniformll
20
SAC
MPC
Time steps
Figure 8: 2D Minecraft learning curves with varying skill-practics distributions. Practicing skills that maximize intrinsic reward outperforms naive methods for choosing which skills to practice.
Learning with short rollouts. Algorithm 2 learns skills using one-step rollouts from arbitrary states. These short rollouts significantly reduce learning signal relative to sampling skill from a fixed start state distribution and running the skill for an entire episode: rather than being able to "associate" all the states along episode with skill, the agent will be forced to associate all states with all skills. Intuitively, the skill- Kkill-practice distribution can alleviate this, associating states with certain skills which are important, making the skill learning process easier. We investigate this by analyzing DADS-Off (Sharma et al. 2020) an off-policy improved version of DADS and resampling skills every K timesteps. This simulates K K-step rollouts from arbitrary starting states, ie. an oracle version of LiSP with data generated from the real environment, without a skill-practice distribution. We show the learned skills with various K for Lifelong Ant in Figure 9 Despite DADS-Off using off-policy data, resampling skills hurts performance, leading to a loss of diversity in skills. When generating transitions with model as LiSP, it is unstable run long model rollouts due to compounding errors, making the skill kill--paactcee distribution critical.
>
1
5
20
200
Figure 9: Skill learning collapses for DADS-Off in Ant, even without the primitive dynamics model fa. Shown are the x-y positions, where each color denotes one skill starting from the origin. More diverse skills are better. From left right, skills are resampled during training every 1 (fully online, like LiSP), 5, 20, and 200 (fully episodic) timesteps.
4.4 LEARNING SKILLS ENTIRELY FROM OFFLINE DATA
Finally, we evaluate the entire LiSP framework (Algorithm)) I in fully offline setting without test- time updates, considering LiSP solely as an offline RL algorithm. Again, the most direct base- line/ablation LiSP is action-space MPC, also known PETS (Chua al. 2018), which can learn
8

---

Published as a conference paper at ICLR 2021
offline for multiple test-time tasks. We consider two versions: one that uses long horizon for plan- ning (H = 180, MPC-Long) and one with a short horizon (H = 25, MPC-Short). The former is a direct ablation, while the latter is closer to prior work and the PETS paper.
Learning skills offline for multiple tasks. We consider three Lifelong Hopper tasks, where tasks define target velocities. We use the same dataset Section 4.1 collected from a forward hop task. LiSP successfully learns a set of skills offline for planning for all tasks.
Our results are in Table [
Table 1:
Average performance when learning fully offline from datasets for Lifelong Hopper.
Though not explicitly designed for this setting, LiSP can learn skills offline for multiple tasks.
Task
LiSP (Ours)
MPC-LLong
MPC-Short
Forward Hop- Fast
0.84 = 0.02
0.27 + 0.10
0.27 + 0.04
Forward Hop- Slow
0.88 0.05
0.49 + 0.42
0.32 + 0.08
Backward Hop- Slow
0.81 + 0.01
0.38 + 0.16
0.27 + 0.06
5 DISCUSSION AND RELATED WORK
In this section, provide overview of other works related to LiSP, in similar fields to lifelong RL. An in-depth discussion of the related works is in Appendix A
Non-Episooe RL. The traditional approach to non-episodic RL is to explicitly learn a policy reset the environment (Even-Dar et al.] [2005] Han et al.) 2015] Eysenbach et al.] 2017). This requires somewhat hard-to-define notions of what reset should accomplish and still uses manual resets. We do not learn a reset policy, instead focusing on naturally safe acting via learning from offline data and effective planning. Zhu al. (2020) and Co-Reyes et al.] (2020) propose solutions to lack of learning signal in reset-free settings but only consider environments without sink states; the latter requires control over the environment
form a training curriculum. [u et al. (2019) and Lowrey et al. (2018) highlight the benefits of both planning and model-free RL for lifelong agents but require highly accurate world models. Offline RL learns policies exclusively from offline data,
which naturally lacks resets (Levine et al.) 2020 Agarwal et al., 2020; Wu et al.) 2019: Yu et al.) 2020
Kumar et al.] 2020), but most work is restricted to single-task, stationary settings.
Masel-Based Planning. Existing works in model-based planning are restricted to short horizons (Chua et al. 2018: Wang & Ba 2019 Nagabandi et al. 2020). Similarly to LiSP, some works (Kahn) ct al.) 2017 Henaff al. 2019) try reduce model error for planning by penalizing deviations outside the training set. We found embedding uncertainty into the cost function causes poor cost shaping; these works also still have relatively short horizons. Mishra et al. (2017) learns action priors used to generate actions for MPC that embed past actions in a latent space for constrained planning, similarly how we skill constraint, but only considers fairly simple manipulation task. Some works (Lowrey et al., 2018 Lu et al., 2019 Sikchi et al.) 2020). including in offline RL (Argenson) & Dulac-Amolldy 2020). use terminal value functions to aid planning, allowing for successful short- horizon planning; as discussed previously. this does not directly translate to good performance in the lifelong setting due to instability and nontriviality in learning this function.
Skill Discovery. The key difference from prior skill discovery work is the lack of episodes. The most relevant work to ours is DADS (Sharma al. 2019). Unlike DADS, we plan over the model with action predictions, which allows for accurate planning even if the discriminator is not accurate.
Most other works (Achiam et al.] 2018: Warde-Farley et al.] 2018 Hansen et al. 2019) Campos et al. 2020) are complementary methods that can help learn a set of skills. Some seek to use skills as pretraining for episodic learning, as opposed to our focus on safe reset-free acting.
CONCLUSION
We presented LiSP, an algorithm for lifelong learning based on skill discovery and long-horizon skill-space planning. To encourage future work in this space, we proposed new benchmarks that capture the key difficulties of lifelong RL in reset-free, nonstationary settings. Our experiments showed that LiSP effectively learns in non-episodic settings with sink states, vastly improving over prior RL methods, and analyzed ablations to show the benefits of each proposed design choice.
ACKNOW
We would like to thank Archit Sharma for advice on implementing DADS.

---

Published as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 1807. 10299, 2018.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114, PMLR, 2020.
Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. arXiv:2008. 05556, 2020.
arXiv preprint
Pierre- Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. Interna- tional Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. arXiv preprint arXiv: arXiv:/805. 1805:/8001111111 /21 2018.
John D Co-Reyes, Suvansh Sanjeev, Glen Berseth, Abhishek Gupta, and Sergey Levine, Ecological reinforcement learning. arXiv preprint arXiv:2006. 12478, 2020.
Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Reinforcement learning in pomdps without resets. 2005.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint
2017.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXiv: 1rX1v:1802.06070, 2018.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference Machine Learning, pp. 1920-1930. PMLR, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint 4rXiv:2004.07219, 2020.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv
preprint 1ale::1611.007. 2016.
William H Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al. The minerl competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv: 1ete:1999.,00,, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-25666 PMLR, 2019.
Weigiao Han, Sergey Levine, and Pieter Abbeel. Learning compound multi-step controllers un- der unknown dynamics. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6435-6442. IEFE, 2015.
10

---

Published as a conference paper at ICLR 2021
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farkee and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint 100x:::906005030.0 2019.
Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with uncer- tainty regularization for driving in dense traffic. arXiv preprint arXiv:1901.02700 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, 2019.
Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine. Uncertainty-awant reinforcement learning for collision avoidance. arXiv preprint arXiv: /70.001 182, 2017.
Maximilian Karl, Maximilian Soelch, Philip Beeker-Ehmen Djalel Benbouzid, Patrick van der Smagt, and Justin Bayer. Unsupervised real-time control through variational empowerment. arXiv preprint arXiv: /ttdv:77100000 1710.05101, 2017.
Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-Boisvert, Pierre-Luc Bacon, and Doina Precup. Options of interest: Temporal abstraction with interest functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4451, 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.05957, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ing catastrophic forgetting neural networks. Proceedings of the national academy of sciences, 114(13):3521-352 2017.
et al. Overcom-
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal agent- centric measure of control. In 2005 IEEE Congress on Evolutionary Compulation, volume 1, pp. 128-135. IEEE, 2005.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint wrXiv:2006.04779) 2020.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprins arXiv: 1802. 2018.
Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes, worst- case approach using model-based reinforcement learning, extended version. arXiv preprint arXiv: 10090, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv:1811.01844 2018
Kevin Lu, Igor Mordatch, and Pieter Abbeel. Adaptive online planning for continual lifelong learn- ing. arXiv preprint arXiv:1912.00 188, 2019.
Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion. 1992.
Nikhil Mishra, Pieter Abbeel, and Igor Mordatch. Prediction and control with temporal segment models. In International Conference on Machine Learning, pp. 2459-2468. PMLR, 2017.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. arXiv preprint arXiv: (eo..000500099 2018.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:1909 10618, 2019.

---

Published as a conference paper at ICLR 2021
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based arXiv preprint arXiv:1812.07671, 2018a. 8a.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model- deep reinforcement learning with model-free fine-tuning. In 2018 JEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018b.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112. PMLR, 2020.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint 1rXiv:1910.01708, 2019.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicraa, and Greg Wayne. Experience replay for continual learning. arXiv preprint 1rXiv: 1682, 2018.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1eXiv:1606.04000 2016.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment-an introduction. In Guided Self-Organization: Inception, pp. 67-114. Springer, 2014.
Jonathan Schwarz, Jelena Luketina, Wojeiech M. Czameekki, Agnieszka Trabska-Barwinska. Yee Whye Tch, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning, 2018.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. arXiv preprint 1rX.v:997.01665 2019.
Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emergent real-world robotic skills via unsupervised off- policy reinforcement learning. arXiv preprint arXiv:2004. /2974, 2020.
Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off- off-policy with online planning. arXiv preprint arXiv:2008. 10066, 2020.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction reinforcement learning, 1999.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. 2018.
Arun Venkatraman, Martial Hebert, and Bagnell. Improving multi-step prediction of learned time series models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint Ntt::/966.0800 2019.
David Warde-Farte Farleya Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv: / 1359, 2018.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint trXivv1509.001 149, 2015.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint 1911v:1911.11311.. 2019.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non- stationarity, arXiv preprint arXiv:2006. 10701, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
12

---

Published as a conference paper at ICLR 2021
Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, and Dinesh Jayaraman. Cautious adapta- tion for reinforcement learning in cafetty-ritical settings. In International Conference on Machine Learning. pp. 11055-11065 PMLR, 2020.
Ruihan Zhao, Kevin Lu, Pieter Abbeel, and Stas Tiomkin. Efficient empowerment estimation for unsupervised stabilization. In International Conference on Learning Representations, 2021.
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Ku- mar, and Sergey Levine. The ingredients of real-world robotic reinforcement learning. arXiv preprint arXiv:2004. 12500, 2020.
13

---

Published as a conference paper at ICLR 2021
A FURTHER DISCUSSION AND RELATED WORK
Model-Based Planning. Chua et al. 2018 propose to use ensemble of probabilistic dynamics models for planning, which we use, but by itself struggles to scale to higher-dimensional Mujoco tasks due to limited model accuracy; the use of model ensembling to mitigate enrors has also been previously explored by Nagabandi et al. (2018b) and Kurutach et al. (2018). Wang & Ba (2019) improves upon some of the weaknesses of Chua et al. (2018) by planning in the parameter space of policies, solving some MuJoCo tasks asymptotically. but is still not accurate enough to plan horizons long enough for environments that require long sequence of coordinated actions. Nagabandi et al. (2020) shows that an improved optimizer can improve MPC performance for short- horizon tasks. We note that all of these methods struggle plan for long horizons due compound- ing model errors, and none have been able to perform MPC in the Hopper environment without terminal value function (as in Sikchi al. (2200) and Argenson Dulac-Amold (2020), which is novelty of our work.
Wang & Ba (2119)
al. 2019), Argenson Dulac-Arnoid (2020). and [Sikchi et al. (2020)
initialize the actions for MPC with prior policy, but this is not the same as constraining the space. Consequently, they not have as strong long-horizon benefits, instead using other methods for at- taining strong performance. Sometimes early termination and low Gaussian noise for random shoot- ing is used to attempt to approximate a constraint (Sikchi et al., 2020), but this is not very effective in higher dimensional environments. Some works (Venkatraman et al. 2015 Mishra et al. 2017 Hafner et al.; 2019) try to improve planning accuracy with multi-step prediction losses, which is complementary our work and could improve performance. Furthermore, even though value func- tions for short horizon planning is not particularly promising, we note that terminal value functions still have other benefits that can improve performance when combined with long-horize planning,
such as improved exploration or long-term reasoning (Lowrey et al. 2018 Lu et al. 2019).
Hierarchical RL. Our work is somewhat similar to hierarchical option-critic architectures that "plan" without MPC (Sutton al. 1999. Bacon et al. 2017 Nachum et al.] [018 Khetarpal al, 2020), which is sometimes referred to as background-time planning" (Sutton & Barto (208)). How- ever, our skills are learned with an unsupervised reward and composed via MPC, which has the promise of more explicit long-term hierarchical benefits, whereas there is some evidence (Nachum) et al. 2019) that policy-based hierarchy only aids in exploration - not long-term reasoning and replaceable with monolithic exploration methods. In contrast planning adds more explicit and interpretable "decision-time planning". which may have better scaling properties in more complex environments, long as planning can be done accurately.
2D Minecraft Entropy
2.8
Entropy
2.3
0.0
0.5
1.0
1.5
Timesteps
x10
Figure 10: Entropy of skills proposed by skill-practice distribution. The red line shows the maximum possible entropy, which exists as the model class is a TanhGaussian (Haarnoja et al. 2018).
Our skill-practice distribution somewhat resembles the interest function of Khetarpal et al. 2020 both are designed to associate certain skills with certain states, forming a type of curriculum. In particular, they note that using a small number of skills improves learning early but can be asymp- totically limiting, which has some similarity to curriculums generated from the skill-practice distri- bution. We show the entropy of the skills generated for the 2D Minecraft environment in Figure I0 the skill-practics distribution automatically generates curriculum where it purposefully practices less skills (low entropy) early in training, and more skills (high entropy) later. Unlike Khettrpal et al.) 2020). our skill-practice distribution is not part of the policy update and is only used to generate
14

---

Published as a conference paper at ICLR 2021
curriculum. We also note that while this means that transitions generated from the model are not drawn from the prior distribution in the objective, since the critic Q(s, 2, a) learns a target for the next state using the same as Q 2, (a)s', 2)), it is correct without importance sampling.
Safety. Zhang et al. 2020)) perform risk-averse planning embed safety constraints, but require handcrafted safety function. This is similar to arguments from the Safety Gym work (Ray et al., 2019). which argues that the most meaningful way to formulate safe RL is with constrained op- timization problem. However, the safety in our work deals more with stability and control, rather than avoiding a cost function, so our notion of safety is generally different from these ideas. More concretely, the type of uncertainty and risk that are crucial to our setting are more epistemic, rather than aleatoric, so these types of works and distributional RL are not as promising. Consequently, we found that long-horizon planning and trying to perform accurate value estimation at greater scale to be more effective than constraining against a cost function or using explicit risk aversion. We note that some "empowerment" -style methods (Gregor et al. 2016; Karl et al.] 2017: Eysenbach et al.) 2018 Sharma et al., 2019) optimize an objective that aims to resemble a notion of stability; how- ever, this metric is hard estimate accurately in general, so we do not explicitly use it for any form of constrained planning. ZZhao et al. (202T) learn a more accurate estimator of the empowerment, which could be useful for future work in safety.
Nonstationarity. Much work nonstationarity focuses on catastrophic forgetting (Rusu et al, 2016 Kirkpatrick et al.) 2017; [Schwarz et al., 2018), which is not primary goal of our work as we are primarily concerned with acting competently in the current MDP without explicitly trying to remember all previous any arbitrary MDPs. Our approach to nonstationary lifelong learning follows Lu et al. (2019), but we do not require access to the ground truth dynamics P at world changes. Lecarpentier & Rachelson (2019) tackles nonstationarity in zero-shot for simple discrete MDPs by using averee planning; our experiments generally suggested that accurate and scalable planning was more important than explicit risk aversion for our settings. Other works (Nagabandi et al. (2018a] Finn et al. [019 Rolnick et al. (0018) are typically nonstationary at the abstraction of episodes, which generally does not require competencies such as online safe adaptation, and instead try to adapt quickly, minimize regret, or avoid catastrophic forgetting as previously discussed. Xic al. (2020) is closer our setting, as they consider nonstationarity within an episode, but they still require manual resets except in a 2D environment which does not contain sink states.
B FURTHER PLOTS FOR LEARNING DYNAMICS
In this section, we provide additional plots, seeking to give more insights on the learning of LiSP from the MuJoCo experiments from Section 4.1
In particular, we can consider the intrinsic reward, which is a proxy for the diversity of skills. Since the intrinsic reward is calculated under the model, which changes and has inaccuracies, high intrinsic reward under the model is not always the best indicator, whereas it is more reliable metric when learned from real world transitions in Sharma et al. al. (2020). Also, the intrinsic reward is sampled according to an expectation given by the skill-practice distribution, so these numbers cannot be directly compared with those given by the [Sharma et al. (2020) paper.
The intrinsic reward during the offline phases on the MuJoCo tasks are:
Offiine Lifelong Hopper Intrinsic Reward
Offline Lifelong Ant Intrinsic Reward
1.75
Intringc Reewrd
Intrincc Rwroard
1.25
0.25
04
Btet atiions
XLE
Steratin terstions
Figure 11: Intrinsic reward during the offline phase of training. Note that these are substantially different than those from episodic training, highlighting the role of exploration in skill discovery.
15

---

Published as a conference paper at ICLR 2021
C EPISODIC EVALUATION OF EMPOWERMENT
In this section, we consider LiSP as an episodic learning algorithm (i.e. in the standard skill discov- ery setting). In particular, to measure performance here consider how LiSP maximizes empow- ermens (Klyubin et al., 2005; Salge et al. 2014 Gregor et al. 2016). which roughly corresponds to how stable the agent is, how much "potential" it has. Empowerment is mathematically defined by the mutual information between the current state and future states given actions, and has motivated many skill discovery objectives, including the one studied in this paper. In fact, this can give benefits to reset-free learning as well: if all the skills maintain high empowerment, then this ensures that the agent only executes stable skills. Although we find we are able to solely use this type of constraint for safe acting instead having to rely on long horizons these types of skills help to reduce the search space for planning. For example, learning good skills reduce the search space for planning was very beneficial in the 2D Minecraft environment, so this is another experiment yielding insight to this property.
We compare LiSP to DADS (Sharma al., 2019); note that the main differences will be due to using model-generated rollouts for training the skill policy and the skill- sxill-practice distribution. We perform training in the standard episodic Hopper setting (without demos) for skill discovery algorithms, and approximate the empowerment by measuring the average z-coordinate of the Hopper, corresponding remove the
to the agent's height, proposed in Zhao et al.] (2021). To make this evaluation, planning component of LiSP, simply running the skill policy as in DADS, and maintain the standard LiSP algorithm otherwise, where both the skill policy and discriminator train exclusively from model rollouts. The changes to our hyperparameters are:
Every 20 epochs, sample 2000 environment steps
Generated replay buffer size of 20000
Generate 20000 model rollouts per epoch
Take 128 discriminator gradient steps per epoch
Take 256 skill policy gradient steps per epoch
Take 32 skill- pprccticc distribution gradient steps per epoch
We do not use disagreement penalty
Our results are shown in Figure 12] LiSP acts strong model-based skill learning algorithm, achieving 2 5x sample efficiency increase over DADS in this metric. The sample efficiency of LiSP is competitive model free algorithms on the Hopper environment, demonstrating that is possible to gain significant learning signal from few samples by utilizing model rollouts, competitive with state-of-the-arr algorithms that focus on only learning one task.
Average Height in Episodic Hopper
Height
1.0
0.5
0.0
0.2
0.4
0.6
0.8
1.0
Timesteps
x10
LiSP
DADS
Figure 12: Average height of the hopper for an evaluation episode of length 200 with randomly sampled latents, a proxy for measuring the average empowerment of the learned skills. LiSP shows considerable improvement in sample efficiency DADS this regard.
16

---

Published as a conference paper at ICLR 2021
D OFFLINE LEARNING WITH LIMITED DATA
In this section, we repeat the offline RL experiment from Section 4.4 where we perform offline training of LiSP on dataset and then use LiSP test time for multiple tasks without gradient updates, but on a smaller dataset. As described in Appendix D the original experiment used one million timesteps for training, representing a similar dataset size the D4RL suite of tasks (Fu et al. 2020). Here, we evaluate on the same dataset but using only the first 200000 timesteps, which represents a small dataset for this task (the replay buffer the beginning of training), and roughly corresponds to "medium-replay" in D4RI..
The results are shown in T We include the original results for comparison. Despite the reduced dataset size, LiSP still stays stable although has somewhat worse task performance. Since the size and composition of the dataset was not focus of our work, we don't include more extensive results, but believe this experiment indicates that LiSP would be promising for varying datasets.
Table 2: Average performance when learning fully offline from smaller dataset for Lifelong Hop- per. The LiSP agent still remains stable in this limited data regime.
Task
LiSP (200k)
LiSP (1m)
MPC-Long (1m)
MPC-Short (1m)
Forward Hop- Fast
0.80 + 0.01
0.84 + 0.02
0.27 + 0.10
0.27 : 0.04
Forward Hop- Slow
0.80 + 0.01
0.88 + 0.05
0.49 = 0,42
0.32 + 0.08
Backward Hop- Slow
0.79 + 0.01
0.81 + 0.01
0.38 = 0.16
0.27 + 0.06
E ENVIRONMENT DETAILS
In this section, we provide additional details about lifelong experiments and the environment setup, including the modifications to the reward functions, the datasets used for pretraining, and the metrics for performance used.
E.1 LIFELONG MuJoCo ENVIRONMENTS
Our environment settings were designed such that the optimal behaviors are similar to the optimal behaviors in the Gym version of these environments, and so that the difficulty of learning was similar (although the learning dynamics are different). This is generally accomplished by having some part of the reward for stability, in addition to the standard reward for accomplishing the task, which also emphasizes the health of an agent by considering the reward; standard benchmarks lack this because the signal comes from a termination function. While this extra shaping of the reward gives more signal than normal, these settings still hard for standard RL algorithms as highlighted above, and even episodic RL without early termination using the standard rewards tends to fail when episodes long relative time-to-failure.
In this work, we considered two MuloCo tasks: Lifelong Hopper and Lifelong Ant. The reward functions we use for these environments are as follows, where the height of the agent, Øvel is the x-velocity, and &pel is a target xeveoiciy:
Lifelong Hopper: -5(z 1.8)² - |ovel &oetl I + [rvet]
Lifelong Ant: -(2 - 0.7)2 - |rrvet - Svetl + [@vell
For the experiments in Section 4.1 we change the target velocity every 1000 timesteps according to the following ordering (chosen arbitrarily):
Lifelong Hopper: [0, 11,2,, -1]
Lifelong Ant: [1, 11
The dataset used for both tasks the replay buffer generated from SAC agent trained to convergence, which we set as one million timesteps per environment. This the standard dataset size used in offline RI. (Fu et al. 2020), although of higher quality. Despite this, our setting is still
17

---

Published as a conference paper at ICLR 2021
difficult and highly nontrivial due to sink states and gradient instability, We hope future works will continue to explore non-episodic settings with varying datasets.
E.1.1 PERFORMANCE METRICS
To summarize performance easily in our experiments, we use performance metric such that is possible to achieve an optimal performance of 1, and a representative "worst-case" performance is set O. This allows for easy comparison between tasks and represents difficulty harder tasks. The weighting of the performance is similar to the standard reward functions, prioritizing stability with the height term and rewarding competence with task-specific term, and easily represents the overall gap relatively perfect policy/desired behavior with single number.
For Lifelong Hopper, take the average height Eaeg the evaluation duration and the average velocity Tvel Note that it is impossible make similar normalization to I with returns, since impossible to attain the optimal height and velocity from every timestep, but the averages can be optimal. For some target velocity Peet, the performance use is given by:
1 0.8
1.32 (Zag 1.3) 0.2
Fvel Evell
The height is derived from approximately the height of an optimal SAC agent on the standard Hopper benchmark, representing the behavior we aim achieve.
For Lifelong Ant, we use similar metric:
1-0.5 0.5
0.52 (aag - 0.7) 0.5 / Foel Evel
E.2 2D GRIDWORLD ENVIRONMENTS AND SKILL VISUALIZATIONS
In this work, we considered two continuous 2D gridworld tasks: volcano and 2D Minecraft. The volcano environment allows to consider the safety of agents, and Minecraft is well-known video game setting that attractive for hierarchical RL due to the nature of tools (Guss et al.] (019); our version is simple minimal 2D version that faster to train. We visualize both these environments and skills learned on them below:
0.75
0.75
0.0
0.0
+1.23
-075
-0.75
+1.00
-1.00
Figure 13: Volcano environment (left) and 2D Minecraft environment (right). Colors denote differ- skills/latents learned by LiSP. Most skills seek perform behaviors relevant to the task (reaching the goal, interacting with the tiles), due to being trained on the data distribution of the agent.
E.2. VOLCANO ENVIRONMENT
In the volcano environment, there is lava (shown in dark red), which the agent is penalized for standing in but does not inhibit its movement. The green tile, located on the right side in the picture, denotes the target goal. The reward is given by negative L2 distance to this goal. The white squares denote free tiles. The primary square of interest is the brown tile, representing pitfall, where the agent will be trapped if it falls in. The state is the agent's 's x-y position, the position of the pitfall, as well as the position of the goal. We can see that the learned skills correspond to avoiding the tile, representing how the skills themselves can embed safety, whereas planning in action space would
18

---

Published as a conference paper at ICLR 2021
not achieve the same effect. This behavior follows from empowerment objective (see Appendix C in particular, LiSP is capable of offline evaluation of empowerment. We initialize the dataset with 100k demonstrations of trained SAC agent with noise applied to actions.
E.2.2 2D MINECRAFT ENVIRONMENT
In the 2D Minecraft environment, the agent maintains an inventory of up to one of each item, and receives a reward whenever obtaining a new item (either through mining or crafting) in increasing magnitudes depending on the hierarchy of the item (how many items were needed to obtain before- hand to obtain the item). The state the position the agent, the position of the blocks it can interact with, well its inventory, represented by vector of O's or I's to represent ownership of each item. There are four tiles interest:
A crafting table (bottom right), where the agent will craft all possible tools using the current materials in its inventory
A wood block (bottom left), where the agent will obtain wood for visiting
A stone block (top left), where the agent will obtain a stone for visiting if it has wooden pickaxe, consuming the pickaxe
An iron block (top right), where the agent will obtain iron for visiting if it has a stone pickaxe, consuming the pickaxe and yielding the maximum reward in the environment
The skill progression representing by the following ordering:
1. Walk to wood to mine
2. Bring wood to craft table to craft stick
3. Craft wooden pickaxe with both a wood and stick
4. Walk to stone to mine using wooden pickaxe
5. Craft stone pickaxe with both stone and a stick
6. Walk iron to mine using a stone pickaxe
There is skill reuse in that lower-level skills must be executed multiple times in order to accomplish higeer-level skills, forming hierarchy of skills in the environment. The dataset consists 2000 steps of manually provided demonstrations, which is very low amount of demonstrations.
F HYPERPARAMETERS
Our code can be found at: https://gihubb.oo//////ifttttt
For all algorithms (as applicable) and environments, we use the following hyperparameters, roughly based common values for the parameters in other works (note classify the skill-practice dis- tribution as policy/critic here):
Discount factor 7 equal 0.99
Replay buffer D size of 106
Dynamics model with three hidden layers of size 256 using tanh activations
Dynamics model ensemble size of 4 and learning rate 10-3 33 training every 250 timesteps
Policy and critics with two hidden layers of size 256 using ReLU activations
Discriminator with hidden layers of size 512 using ReLU activations
Policy, critic, discriminator learning rates of 3 x 10 training every timestep
Automatic entropy tuning for SAC
Batch size 256 for gradient updates
MPC population size S set to 400
MPC planning iterations P set 10
19

---

Published as a conference paper at ICLR 2021
MPC number of particles for expectation calculation set to 20
MPC temperature of 0.01
MPC noise standard deviation of I
For PETS, we use planning horizon of either 25 or 180, as mentioned
For DADS, we find it helpful to multiply the intrinsic reward by a factor of for learning (for both DADS and LiSP usage)
For LiSP specifically, our hyperparameters are:
Planning horizon of 180
Repeat skill for three consecutive timesteps for planning (not for environment interaction)
Replan at every timestep
Number of rollouts per iteration M set to 400
Generated replay buffer ú size
5000
Number of prior samples set to 16 in the denominator of the intrinsic reward
Number of discriminator updates iteration set to 4
Number of policy updates per iteration set to 8
Number of skill Akill-practice updates per iteration set to 4
Disagreement threshold Othres 0.05 for Hopper, 0.1 for Ant
Disagreement penalty x 30
20


Reviews:
[{"review_id": "VL4Ul-2uQ7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Edit: Having gone through the updates and the author's replies, I am increasing my score.\n\n\n## Summary\n\n* The paper proposes a lifelong reinforcement learning (LRL) approach that (i) learns a model of the env, (ii) uses that model to learn skills, (iii) plan in the abstract space of skills.\n\n## Strengths\n\n* Many design choices make it a good fit for real-life problems:\n  *  Skills can be discovered from both offline and online data.\n  *  Long horizon planning\n\n* The paper is based on an excellent intuition -- planning in the skill-space. I think the paper describes a reasonable instantiation of the idea. This instantiation is likely to work in practice.\n\n* The paper considers interesting experiments and ablations (though I can't entirely agree with all the results/conclusions -- more on this later). The experiments in the appendix are interesting too.\n\n## Areas to improve\n\n* The paper is somewhat dense to read. In some cases, I had to read the same text three times to make sure I do not misunderstand anything.\n\n* This point is more of a clarification question - Why are the skills sampled from [-1, 1]^dim (page 2 \"Skill Discovery\")\n\n* It would help if the paper had a more in-depth discussion about how $q_v$ and $p_{\\phi}$ are trained. For example, it is not obvious to me if the gradients flow to $p_{\\phi}$ (via $z_i$) when training $q_v$.\n\n* It would help if the authors discuss the complexity of the proposed approach, especially the `GetAction` method.\n\n* One significant weakness of the paper is the lack of comparisons with the baselines, which makes it hard to understand the usefulness of this work. For example, consider figure 1 - the reported metric \"performance\" is defined by the authors, making it difficult to understand how good is a \"performance\" of say 0.8. Having some other baselines would make the analysis more meaningful. In Figure 4, SAC should be seen as a lower bound for the baselines, and having some other, more reasonable non-stationary RL baselines would be useful. Similarly, in figure 5, it appears that using skills (vs. using actions) is much better because it had a lower variance. But just looking at figure 5, it is difficult to estimate how much lower is the variance (and the mean). Looking at the plot, I can not estimate the mean of the two distributions and how many samples are near the extreme model errors. This can be easily fixed by plotting the l2 error on the x-axis and pdf on the y-axis. One more thing I want to highlight here: The comparison does not seem to be fair because the \"skills\" are sampled from the trained skill-policy while the \"actions\" are sampled from a random policy. The model has been trained on the skills sampled from the trained policy, so the result is not surprising.\n\n* One broad concern I have is in understanding the novelty/setup of the paper. My understanding is that the paper claims they improve the performance for reset free lrl (problem setup) using skill space planning (solution setup)? If yes, should they compare against action planning based approaches? I think they try to clarify this in Figure 7, but I did not understand what LOOP agent is\n\nOverall, I think this is exciting and useful work. I have some questions (which I described above), and I might have mis-understood something.  I would be happy to update my score based on the author's answer. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Oa2R7EGwErD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY OF CONTRIBUTION:\n\nThis work presents the LiSP architecture for model-based lifelong learning and applies it to the problem of learning in continuing tasks with non-stationary rewards and no state resets.  LiSP builds on the DADS algorithm, which learns predictive dynamics model of an RL environment that depends on a continuous space of high-level skills, rather than primitive actions, in an unsupervised fashion that does not require an external reward function.  Unlike DADS, LiSP appears to learn an ensemble of primitive models, which is then used to generate simulated transitions from which the DADS algorithm learns its high-level model.  The goal of this work is to utilize the high-level predictive model learned via DADS to address two issues, reset free learning, and learning with non-stationary reward functions.  In this work, DADS is used to learn a dynamics model during a pretraining phase, and this model is then used as a component of a model-predictive control algorithm to select actions for the non-stationary, reset-free task.  The use of a model-based planning allows the agent to act optimally with respect to a non-stationary reward function (the agent has access to the full reward function at each time step).  The fact that the agent does not need to learn its policy from scratch potentially makes this approach far more robust to the permanent failure states.  The experimental results show that LiSP is effective in reset-free, non-stationary reward versions of the MuJoCo Hopper and Ant tasks, as well as a 2D Minecraft--like task.\n\nAREAS OF CONCERN:\n\nOne of the main issues with the paper as it is currently written is that the description of the LiSP algorithm is not detailed enough to understand exactly what the proposed method does, and how it differs from the existing DADS algorithm.  Specific questions that need to be answered are:\n  1) How is the primitive dynamics model f_phi trained?\n    a) How much data is used to train this model?\n    b) What exploration policy is used to update this model?\n    c) How accurate are the primitive models?\n  2) The description of LiSP alternates between describing a single model and an ensemble of models, how is this ensemble trained, and how are transitions sampled from it?\n  3) What loss is used to train the practice distribution p_psi?\n\nThe main concern with the current set of experiments is that they do not seem to isolate the two different sources of failure in the lifelong learning setting, that is, the lack of state resets and the non-stationarity of the reward function.  In Figure 1, it seems likely that MBPO fails due to the existence of a sink state, but it is possible that SAC is also failing due to a change in the reward function.  It seems unfair to compare SAC to MPC in the non-stationary reward setting, as a model-free algorithm will inevitably lose out in this setting to an MPC algorithm which can adapt instantly to a new reward.  It would be more informative to compare LiSP, MBPO and SAC in terms of the stability of their learned policies alone.  If the source of the instability of SAC is the combination of non-stationary rewards and sink-states, then ablation where only one or the other of these conditions is present would be useful to highlight this.\n\nIt is also unclear whether continuing to update the model and skill policy online is necessary.  Figure 7 suggests that not updating the skill policy actually improves stability, so it seems likely that freezing the predictive model would be beneficial as well, or would at the very least not lead to a meaningful loss of performance.\n\nThe results shown in Figure 8 are promising, but their significance is difficult to interpret.  It might be helpful to provide a more detailed discussion of the learned practice distribution, and why it leads to such a substantial improvement over the uniform distribution used in the original DADS algorithm.  It was originally suggested (Section 3.1) that the learned practice distribution is needed due to the use of a primitive model to generate synthetic training data.  It would be helpful to include an ablation which shows the differences in performance between a uniform and learned skill distribution when a primitive model is not used, and the DADS skill model is trained on raw transitions.\n\nThere are a few other minor questions that need to be addressed:\n  1) What does the LOOP agent mentioned in the caption for Figure 7 refer to?\n  2) In Figure 5, it appears that what we are evaluating is the error of the primitive (action-conditional) predictive model, rather than the high-level model, but this is not explicitly state.\n  3) It isn't entirely clear what the sink state for the Ant task looks like, as unlike Hopper, it would seem that even if the Ant falls down it should be able to push itself back up (The fact that we don't observe any runs failing for Ant in Figure 3 suggests that Ant has no sink state).\n  4) In what experiments is pretraining being used?  While it appears that the policies used in the Hopper environment are pretrained, it is not clear that pretraining is used in the Minecraft environment (Figure 8).\n\nCONCLUSION:\n\nThe core idea of this work is sound, and addresses major limitation of reinforcement learning in real-world settings, namely, the need for state resets.  While the algorithm presented is incremental, results suggesting that skill-space model-predictive control may be robust in reset-free learning where primitive MPC appears to fail.  While the experimental results presented here are somewhat limited, it is likely that other researchers could use them as the basis for more detailed investigation of the problem of reset-free learning.  At the very least, however, the presentation of the algorithm and the experimental results needs significant improvement, and my score reflects the current state of the paper, but could increase if the presentation issues above were addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5sduvzp2Z4K", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe authors propose LiSP, a model-based planning method that performs model-predictive control using learned skills rather than actions. The skills are learned using DADS, with a modified reward function that additionally encourages all skills to stay within the support of training data to avoid sink states. The experiment results show stable learning progress on reset-free and ever-changing targets, compared to other baselines.\n\nOverall, the specific topic of lifelong setting (i.e., reset-free learning) studied in the paper is pretty relevant, and the main results of the proposed method are promising. \n\n### Strength\n- The problem is significant and pretty relevant to the field.\n- LiSP seems to work well on the reset-free environments considered in the paper.\n\n### Weakness\n- The scope of non-stationarity investigated here is pretty narrow (only focus on non-stationarity of tasks, as mentioned in the appendix). It is very likely that the proposed framework will fail in other cases (e.g., change of terrain, etc).\n- Analysis of the newly introduced hyperparameters $\\kappa$ and $alpha$ (for the adjusted reward) is missing. How sensitive is LiSP to these parameters?\n- Figure 8: what do we want to test for this \"minimizing intrinsic reward\" case? What's the intuition about it?\n- While the authors have shown some pretty interesting observations/analysis, the writing is a little bit messy. So what exactly helps the agent in the reset-free setting / avoid the sink states? Is it because of the longer-step planning (as shown in Figure 7)? Or is it simply because the intrinsic rewards drive the skills to visit different states? Also, how can we tell from Figure 7 and Figure 1 that MBPO fails because of the instability of gradients? We only have numerical results on the environment returns here.\n\n### Additional Comments\n- I am willing to change my score based on the authors' response to the problems raised in mine and other people's reviews.\n\n=============== Edit ===============\n\nAfter reading the authors' response to me and other reviewers, I think my concerns are sufficiently addressed.\nTherefore, I update my rating from 5 to 6.\n\nThere is still room for improvement in terms of the writing: as raised by the other reviewers, the text is a little bit dense to read. \nIt would be great if the authors can further refine and brush-up the flow of the paper to make it more accessible.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1bWuL9Rgo_x", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a lifelong reinforcement learning framework in a non-stationary environment with non-episodic interactions. The proposed approach is to 1) learn \"skills\" - a world model - to maximize the intrinsic rewards using both online and offline data, and to 2) make best plans based on the learned world model. This approach is evaluated with Hopper and Ant tasks.\n\nOverall, I believe this paper is marginally below the acceptance borderline. I like the keywords - intrinsic rewards, catastrophic forgetting, non-stationary environment, model-based reinforcement learning. But I doubt the usefulness of the framework. The evaluation based on performing two tasks and insufficient comparison with other algorithms is another minus.\n\nPros:\n\n+ Lifelong learning is a machine learning area with a long history and the keywords such as intrinsic rewards, catastrophic ..., non-stationary environment, model-based reinforcement learning all shows that this paper is in the right direction.\n+ The formulation of \"skill\" - world model - as a latent variable is interesting\n+ Model-based reinforcement learning is an underexplored area by my assessment.\n\nCons:\n\n- The evaluation is too simple in comparison with the goal of lifelong learning. I would like to see how the agent performs for multiple tasks. \n- I would like to see how agents improve performance in new tasks, and at the same time, retain the performance of old tasks.\n- I have doubts that the skill formulation with some simple prior $p(z)$ is sufficient for complex tasks.\n- The formulation is less ambitious than what I would expect an accepted paper to be: this paper seems to learn a some-what fixed environment and just solve the tough model-based reinforcement planning problem with mpc. There are many potential issues with mpc in a complex world and I see no performance guarantee.\n- I hope to see how previous learned task knowledge could contribute to future learnings. But the proposed framework doesn't seem to accomondate that.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were excited by this work, which focuses on lifelong RL in non-stationary, non-episodic environments.  They found the approach compelling with exciting results on the tested domains.  However, even the more positive reviewers were concerned with the somewhat narrow scope of evaluation, which makes the paper somewhat less ambitious.  

In response to the reviews, the authors added extra experiments, clarifying text, and requested details that provide more depth and insight to the paper.  Still, the approach and paper is somewhat narrowly focused, but it does yield insights that should be useful for future works that solve this problem in a more general manner.
Decision: Accept (Poster)