Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a lifelong reinforcement learning framework in a non-stationary environment with non-episodic interactions. The proposed approach is to 1) learn "skills" - a world model - to maximize the intrinsic rewards using both online and offline data, and to 2) make best plans based on the learned world model. This approach is evaluated with Hopper and Ant tasks.

Overall, I believe this paper is marginally below the acceptance borderline. I like the keywords - intrinsic rewards, catastrophic forgetting, non-stationary environment, model-based reinforcement learning. But I doubt the usefulness of the framework. The evaluation based on performing two tasks and insufficient comparison with other algorithms is another minus.

Pros:

+ Lifelong learning is a machine learning area with a long history and the keywords such as intrinsic rewards, catastrophic ..., non-stationary environment, model-based reinforcement learning all shows that this paper is in the right direction.
+ The formulation of "skill" - world model - as a latent variable is interesting
+ Model-based reinforcement learning is an underexplored area by my assessment.

Cons:

- The evaluation is too simple in comparison with the goal of lifelong learning. I would like to see how the agent performs for multiple tasks. 
- I would like to see how agents improve performance in new tasks, and at the same time, retain the performance of old tasks.
- I have doubts that the skill formulation with some simple prior $p(z)$ is sufficient for complex tasks.
- The formulation is less ambitious than what I would expect an accepted paper to be: this paper seems to learn a some-what fixed environment and just solve the tough model-based reinforcement planning problem with mpc. There are many potential issues with mpc in a complex world and I see no performance guarantee.
- I hope to see how previous learned task knowledge could contribute to future learnings. But the proposed framework doesn't seem to accomondate that.
review_title: like the topic, have doubts about problem formulation

final_score: rating: 7: Good paper, accept
review_content: SUMMARY OF CONTRIBUTION:

This work presents the LiSP architecture for model-based lifelong learning and applies it to the problem of learning in continuing tasks with non-stationary rewards and no state resets.  LiSP builds on the DADS algorithm, which learns predictive dynamics model of an RL environment that depends on a continuous space of high-level skills, rather than primitive actions, in an unsupervised fashion that does not require an external reward function.  Unlike DADS, LiSP appears to learn an ensemble of primitive models, which is then used to generate simulated transitions from which the DADS algorithm learns its high-level model.  The goal of this work is to utilize the high-level predictive model learned via DADS to address two issues, reset free learning, and learning with non-stationary reward functions.  In this work, DADS is used to learn a dynamics model during a pretraining phase, and this model is then used as a component of a model-predictive control algorithm to select actions for the non-stationary, reset-free task.  The use of a model-based planning allows the agent to act optimally with respect to a non-stationary reward function (the agent has access to the full reward function at each time step).  The fact that the agent does not need to learn its policy from scratch potentially makes this approach far more robust to the permanent failure states.  The experimental results show that LiSP is effective in reset-free, non-stationary reward versions of the MuJoCo Hopper and Ant tasks, as well as a 2D Minecraft--like task.

AREAS OF CONCERN:

One of the main issues with the paper as it is currently written is that the description of the LiSP algorithm is not detailed enough to understand exactly what the proposed method does, and how it differs from the existing DADS algorithm.  Specific questions that need to be answered are:
  1) How is the primitive dynamics model f_phi trained?
    a) How much data is used to train this model?
    b) What exploration policy is used to update this model?
    c) How accurate are the primitive models?
  2) The description of LiSP alternates between describing a single model and an ensemble of models, how is this ensemble trained, and how are transitions sampled from it?
  3) What loss is used to train the practice distribution p_psi?

The main concern with the current set of experiments is that they do not seem to isolate the two different sources of failure in the lifelong learning setting, that is, the lack of state resets and the non-stationarity of the reward function.  In Figure 1, it seems likely that MBPO fails due to the existence of a sink state, but it is possible that SAC is also failing due to a change in the reward function.  It seems unfair to compare SAC to MPC in the non-stationary reward setting, as a model-free algorithm will inevitably lose out in this setting to an MPC algorithm which can adapt instantly to a new reward.  It would be more informative to compare LiSP, MBPO and SAC in terms of the stability of their learned policies alone.  If the source of the instability of SAC is the combination of non-stationary rewards and sink-states, then ablation where only one or the other of these conditions is present would be useful to highlight this.

It is also unclear whether continuing to update the model and skill policy online is necessary.  Figure 7 suggests that not updating the skill policy actually improves stability, so it seems likely that freezing the predictive model would be beneficial as well, or would at the very least not lead to a meaningful loss of performance.

The results shown in Figure 8 are promising, but their significance is difficult to interpret.  It might be helpful to provide a more detailed discussion of the learned practice distribution, and why it leads to such a substantial improvement over the uniform distribution used in the original DADS algorithm.  It was originally suggested (Section 3.1) that the learned practice distribution is needed due to the use of a primitive model to generate synthetic training data.  It would be helpful to include an ablation which shows the differences in performance between a uniform and learned skill distribution when a primitive model is not used, and the DADS skill model is trained on raw transitions.

There are a few other minor questions that need to be addressed:
  1) What does the LOOP agent mentioned in the caption for Figure 7 refer to?
  2) In Figure 5, it appears that what we are evaluating is the error of the primitive (action-conditional) predictive model, rather than the high-level model, but this is not explicitly state.
  3) It isn't entirely clear what the sink state for the Ant task looks like, as unlike Hopper, it would seem that even if the Ant falls down it should be able to push itself back up (The fact that we don't observe any runs failing for Ant in Figure 3 suggests that Ant has no sink state).
  4) In what experiments is pretraining being used?  While it appears that the policies used in the Hopper environment are pretrained, it is not clear that pretraining is used in the Minecraft environment (Figure 8).

CONCLUSION:

The core idea of this work is sound, and addresses major limitation of reinforcement learning in real-world settings, namely, the need for state resets.  While the algorithm presented is incremental, results suggesting that skill-space model-predictive control may be robust in reset-free learning where primitive MPC appears to fail.  While the experimental results presented here are somewhat limited, it is likely that other researchers could use them as the basis for more detailed investigation of the problem of reset-free learning.  At the very least, however, the presentation of the algorithm and the experimental results needs significant improvement, and my score reflects the current state of the paper, but could increase if the presentation issues above were addressed.
review_title: This work presents the LiSP architecture for model-based learning with non-stationary rewards, and applies it to the problem of lifelong-learning in environments without state resets.  Empirical results demonstrate that LiSP, which uses a skill-space predictive model and planning algorithm, is more robust to the lack of state resets than either action-space model-predictive control or model-free RL.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
The authors propose LiSP, a model-based planning method that performs model-predictive control using learned skills rather than actions. The skills are learned using DADS, with a modified reward function that additionally encourages all skills to stay within the support of training data to avoid sink states. The experiment results show stable learning progress on reset-free and ever-changing targets, compared to other baselines.

Overall, the specific topic of lifelong setting (i.e., reset-free learning) studied in the paper is pretty relevant, and the main results of the proposed method are promising. 

### Strength
- The problem is significant and pretty relevant to the field.
- LiSP seems to work well on the reset-free environments considered in the paper.

### Weakness
- The scope of non-stationarity investigated here is pretty narrow (only focus on non-stationarity of tasks, as mentioned in the appendix). It is very likely that the proposed framework will fail in other cases (e.g., change of terrain, etc).
- Analysis of the newly introduced hyperparameters $\kappa$ and $alpha$ (for the adjusted reward) is missing. How sensitive is LiSP to these parameters?
- Figure 8: what do we want to test for this "minimizing intrinsic reward" case? What's the intuition about it?
- While the authors have shown some pretty interesting observations/analysis, the writing is a little bit messy. So what exactly helps the agent in the reset-free setting / avoid the sink states? Is it because of the longer-step planning (as shown in Figure 7)? Or is it simply because the intrinsic rewards drive the skills to visit different states? Also, how can we tell from Figure 7 and Figure 1 that MBPO fails because of the instability of gradients? We only have numerical results on the environment returns here.

### Additional Comments
- I am willing to change my score based on the authors' response to the problems raised in mine and other people's reviews.

=============== Edit ===============

After reading the authors' response to me and other reviewers, I think my concerns are sufficiently addressed.
Therefore, I update my rating from 5 to 6.

There is still room for improvement in terms of the writing: as raised by the other reviewers, the text is a little bit dense to read. 
It would be great if the authors can further refine and brush-up the flow of the paper to make it more accessible.
review_title: Important topic, promising results. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Edit: Having gone through the updates and the author's replies, I am increasing my score.


## Summary

* The paper proposes a lifelong reinforcement learning (LRL) approach that (i) learns a model of the env, (ii) uses that model to learn skills, (iii) plan in the abstract space of skills.

## Strengths

* Many design choices make it a good fit for real-life problems:
  *  Skills can be discovered from both offline and online data.
  *  Long horizon planning

* The paper is based on an excellent intuition -- planning in the skill-space. I think the paper describes a reasonable instantiation of the idea. This instantiation is likely to work in practice.

* The paper considers interesting experiments and ablations (though I can't entirely agree with all the results/conclusions -- more on this later). The experiments in the appendix are interesting too.

## Areas to improve

* The paper is somewhat dense to read. In some cases, I had to read the same text three times to make sure I do not misunderstand anything.

* This point is more of a clarification question - Why are the skills sampled from [-1, 1]^dim (page 2 "Skill Discovery")

* It would help if the paper had a more in-depth discussion about how $q_v$ and $p_{\phi}$ are trained. For example, it is not obvious to me if the gradients flow to $p_{\phi}$ (via $z_i$) when training $q_v$.

* It would help if the authors discuss the complexity of the proposed approach, especially the `GetAction` method.

* One significant weakness of the paper is the lack of comparisons with the baselines, which makes it hard to understand the usefulness of this work. For example, consider figure 1 - the reported metric "performance" is defined by the authors, making it difficult to understand how good is a "performance" of say 0.8. Having some other baselines would make the analysis more meaningful. In Figure 4, SAC should be seen as a lower bound for the baselines, and having some other, more reasonable non-stationary RL baselines would be useful. Similarly, in figure 5, it appears that using skills (vs. using actions) is much better because it had a lower variance. But just looking at figure 5, it is difficult to estimate how much lower is the variance (and the mean). Looking at the plot, I can not estimate the mean of the two distributions and how many samples are near the extreme model errors. This can be easily fixed by plotting the l2 error on the x-axis and pdf on the y-axis. One more thing I want to highlight here: The comparison does not seem to be fair because the "skills" are sampled from the trained skill-policy while the "actions" are sampled from a random policy. The model has been trained on the skills sampled from the trained policy, so the result is not surprising.

* One broad concern I have is in understanding the novelty/setup of the paper. My understanding is that the paper claims they improve the performance for reset free lrl (problem setup) using skill space planning (solution setup)? If yes, should they compare against action planning based approaches? I think they try to clarify this in Figure 7, but I did not understand what LOOP agent is

Overall, I think this is exciting and useful work. I have some questions (which I described above), and I might have mis-understood something.  I would be happy to update my score based on the author's answer. 
review_title: Nice Idea

Meta-review: The reviewers were excited by this work, which focuses on lifelong RL in non-stationary, non-episodic environments.  They found the approach compelling with exciting results on the tested domains.  However, even the more positive reviewers were concerned with the somewhat narrow scope of evaluation, which makes the paper somewhat less ambitious.  

In response to the reviews, the authors added extra experiments, clarifying text, and requested details that provide more depth and insight to the paper.  Still, the approach and paper is somewhat narrowly focused, but it does yield insights that should be useful for future works that solve this problem in a more general manner.
Decision: Accept (Poster)