Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an approach for lifelong learning for a set of parametrically varying tasks, by maintaining a joint world model across all tasks encountered, as well as task-specific models. In the absence of task-specific data, the prior of the task-specific models is set to the world model posterior. The paper includes theoretical analysis that shows that this leads to increased sample efficiency on the task, and also includes empirical evaluation on a set of gym tasks with parametric variation (eg: gravity, mass of body parts), as well as meta-world reaching. 

strengths_and_weaknesses: **Update after Author Response**

I thank the authors for their detailed response. The authors have sufficiently addressed all of my concerns (described in detail below). In light of these clarifications, the  new experimental evidence included and given that the approach does seem original and intuitive, and the problem considered is of importance to the community (as noted in my original review), I am updating my score. With the new results, I think the paper will be of great relevance to the lifelong RL community.


-> How much does the world model prior help? The authors clarify that this is actually included in the paper, though not clearly described in the writing (which can be easily amended for the camera ready version). Further, the authors add new experiments (appendix L) to directly compare to a version of their method using just a fixed prior, to highlight the benefit of using the world model prior.

->  Can the method handle semantically varying tasks? Lifelong learning approaches will need to handle this setting when they're deployed for problems of interest in robotics. The initial experiments were run on parametrically varying tasks (different gravity, body mass etc) or transferring from reaching to reaching with a wall. I appreciate the effort of the authors in testing their approach on the much more challenging MT-10 and MT-50 benchmarks (Figure 5, Appendix H), which have actually different tasks (eg: opening a boxes/closing drawers/opening windows etc). These results make the claims of the paper \emph{significantly} strengthen the claims of the paper, and demonstrate the effectiveness of the approach in comparison to current methods.  

-> How does the method compare to robust policy ? The authors clarify that this considered in the paper that originally introduced the tasks considered in the initial experiments. Further, this concern is sufficiently addressed with the MT-10/MT-50 experiments. The authors also add discussion related to this in appendix Q. 

** Original Review ** 

Significance  - 
 
-> Lifelong learning is an important challenge for RL, as learning from scratch for each new task is infeasible, especially for robotics. Proposed solutions are more likely to be adopted by the community at large if they show signs of progress on tasks closer to the actual problem setting, i.e a stream of semantically different tasks that are encountered over an agent’s lifetime. The main results in the paper are presented on a set of gym tasks (cheetah, walker) with very simple parametric variations (changing gravity, changing body mass etc). For such domains, since access to all previous data is assumed, one could train a single robust model-free policy that can perform quite well, using algorithms like SAC. Furthermore, the metaworld suite includes semantically different tasks (reaching, pushing, pick-place etc) and a better evaluation of the algorithm would include evaluating the approach on these variations. Without this it is difficult to understand whether the approach is actually applicable to the problems faced by RL for deployment. 
 
-> More broadly, part of the issue stems from considering the set of tasks to only have parametric variation. While this restriction might provide added structure to help in theoretical analysis, it is worth considering if the same general idea (joint world model which is used to initialize task-specific models) can also be used for more general sets of tasks, that do not vary in terms of an interpretable parameter. This would open up the approach for evaluation on semantically varying tasks, as even in this case there are similarities to be shared across tasks, such as physics. 

Quality - 

->  The empirical results don’t include sample efficiency comparison of the approach without using the world model to initialize the task-specific models. This seems important as it’s one of the central claims of the paper. Furthermore, as described before the experimental domains are too far removed from settings of interest for the deployment of RL for lifelong learning. 

-> The probabilistic formulation of the individual task models and the joint world model and the corresponding update schemes are sound  and intuitive to follow. The theoretical result that initializing the task specific prior to the joint world model posterior leads to sample efficiency gains also seems intuitive, given that the joint model is trained across tasks. 
 
Originality -
 
->  The proposed approach using world models for continual learning does seem new, and can potentially be very impactful since transition dynamics should be preserved across tasks, and this is exactly what the models learn. Further, the authors provide a probabilistic framework for learning on a particular task while leveraging prior knowledge from a joint model, and provide theoretical analysis for the advantage in doing so. The related work section is adequate and includes most prior work. 
 
Clarity - 
 
-> The paper is clearly written and not too difficult to follow, and includes detailed algorithms that are sufficient for reproducing the method.
 
questions: -> How much sample efficiency benefit does using the joint world model yield, as opposed to re-training from scratch with only the task-specific model?

-> Does the approach scale to settings with more meaningful variation across tasks (eg- reach, push, pickplace) in metaworld? 
limitations: Limitation are not adequately discussed. As described before, the primary limitation is the quality of experimental domains considered.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes that to more effectively handle life-long reinforcement learning settings, where tasks are presented sequentially, one can use model-based techniques with model uncertainty measured globally across tasks and locally within tasks. By tracking model uncertainty and using planning over stochastic models, the method can better tackle both forward transfer (better performance on future tasks after experiencing previous tasks) and backward transfer (better performance on previous tasks after observing new information from future tasks). The paper demonstrates these results in two continuous, life-long reinforcement learning settings: a suite of Mujoco tasks, and Meta-World, comparing to other model-free techniques as well as approaches that do not model both global and local uncertainty.
strengths_and_weaknesses: Strengths
* (Significance) life-long RL is an increasingly critical setting, so paper is addressing an important topic. Furthermore, doing so in a model-based way as this paper demonstrates clearly has advantages over model-free methods, even when the planner used is exceptionally simple, which should encourage people to consider whether model-based methods may be more sensible for their setting.
* (Originality) The method is somewhat novel in estimating both global and local uncertainty within models of transitions using Bayesian Neural Networks. While the overall concept is similar to Nagabandi et al (2018) which models the global and local uncertainty using a MAML-like approach, this paper shows that such a technique is a better choice for the life-long learning setting than alternative life-long RL methods.
* (Clarity) The paper is clearly written. It was straightforward to follow the method and the figures were mostly well explained. As I mention below, I think some of the proofs could be moved to the appendix to make more space for results in the main body of the text.

Weaknesses
* (Quality / Clarity) The choice of baselines seemed somewhat odd, and made the overall message of the paper a bit muddled. In my reading, the most important aspect of the method is to model two layers of uncertainty in the MDPs: globally across tasks, and locally within tasks. The most relevant baselines for this question seem like those that also apply model-based methods to the problem of multi-task learning – for example Nagabandi et al, 2018. I understand their focus was different, but the technique seems very similar and it would help to have better contextualization for why it is different technically, and whether those technical differences matter. As it stands, it was not clear to me if the paper’s main message was an argument for the specific method introduced, or about using model-based methods with planning as a more stable approach to life-long reinforcement learning in general (and the multiple levels of uncertainty just being critical to get this to work). In either case, presenting baselines from both model-based and model-free techniques (and carefully signposting these) would help clarify the message and interpretations of the results.
* (Quality / Clarity) The number of experiments was reasonably small – only one page of results relative to 6 pages of method description. I think some of the proofs/lemmas could be safely moved to the supplement to make more space for results that better show advantages of multi-level uncertainty model-based vs. model-free for lifelong settings and leave more room for discussion around why multiple types of uncertainty are important empirically.
* (Originality) While I am not aware of anyone applying exactly this kind of approach to the continual RL setting, the method feels similar conceptually to several approaches from meta-learning that could also be applied to a life-long setting (but are not included as baselines in this paper). 
* (Significance) I am not sure if this will impact how people do lifelong reinforcement learning, as it’s unclear how computationally expensive BNNs are, and whether the approach would work for settings in which the tasks cannot be clearly mapped onto a single parameter which is varying

questions: * Can the authors elaborate on how the approach compares to other model-based techniques for RL from meta-learning? Are the technical differences important for getting the approach to work in the life-long setting?
* Can the authors clarify what they believe the main message of the paper is? Is it specifically about the presented method, more generally about the importance of multiple types of uncertainty, or about model-based vs. model-free approaches to life-long reinforcement learning?
* Can the authors discuss how the method would fare in settings that do not have obvious, simple ways in which the tasks differ for life-long RL? What would happen if the presented tasks actually share nothing in common? Would the method successfully avoid catastrophic forgetting?No ethical limitations to consider here. The authors could go into more detail on the technical limitations of the approach.

limitations: No ethical limitations to consider here. The authors could go into more detail on the technical limitations of the approach.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: * The authors tackle the problem of lifelong learning in RL and propose to use hierarchical Bayesian modeling to achieve transfer between tasks over the course of learning.  
* In particular a world model will be used alongside task specific modeling to extract common and specific structure in existing tasks to model new tasks. 
* This facilitates modeling new tasks by using the between-task posterior as a within-task prior for newly sampled tasks. 
* Tasks are related and sampled sequentially throughout learning and generally depend on a task latent to define specific task features.  Tasks are modeled with HiP-MDPs.
* The authors define forward transfer and backward transfer across tasks by using prior and posterior data to improve the performance across the set of sampled tasks.
* They next derive a performance bound that shows that fewer samples are required to train specific task models as the world model converges on the underlying distribution of environment dynamics.
* The authors further develop Variational Baysian exploration for Lifelong RL (VBLRL) for better scale by using variational inference to approximate a world model using Bayesian Neural Networks.  Both forward and backward (previously encountered tasks) task transfer is implemented to boost performance.
* The authors experiment with this approach on MuJoCo and Meta-world tasks against baselines in lifelong learning and demonstrate state of the art results.

**Claims**:

 1. Demonstrate better forward and backward transfer performance against state of the art lifelong RL algorithms.
 2. Bayesian modeling across tasks that improves sample efficiency.


strengths_and_weaknesses: **Strengths**

* Lifelong RL is an active area of research which presents a number of challenges that will require scalable approaches to overcome.  The authors target a task modelling approach that aims to address this through learning of the underlying dynamics with hierarchical Bayesian methods.
* The authors rigorously demonstrate that task models can be trained with fewer samples as the world model posterior approaches the true underlying dynamics of the tasks environment.  
* The authors cover prior work in using Bayesian methods in RL, lifelong RL, and Hip-MDP processes in their related work and provide some background to help contextualise their work.
* Ablation shows BNNs and backward transfer improved performance.
* Results look overall good and VBLRL improves over comparable baselines.  Overall the authors seemed to have satisfied the claims made, demonstrating that their VBLRL approach improves sample efficiency while attaining state of the art performance over a number of baselines.


**Weaknesses**

* There seem to be some missing references in sectionThere is no reference for VBLRL in section 1 and I could not find a reference for "Bayesian Exploration for Lifelong Reinforcement Learning" Fu et al. 2021.  There should be a reference here to prior work.  In the same section BNNs are referenced but no work is cited nor are they really explained.  In section 2, some explanation for both of these approaches would be helpful.
* While the authors demonstrate that the approach is effective the novelty of it seems somewhat limited.  However, the authors provide some theoretical justification.
* Section 4.2 seems like it could use more detail.  For example, figure 1 is fairly high level and doesn't seem to capture in a straightforward way the dynamics of the model learning across tasks.  The algorithms which explain the VBLRL in detail are in the appendix. I feel this section (and 4.3) are meant to really define the unified approach yet I didn't come away with a clear understanding of how back transfer and VBLRL are working here.

questions: 
* Does forward and backward transfer always result in monotonically increasing performance?  Are there cases where the transfer is counter-productive to performance improvement? Finally, if so, under what conditions does this typically occur?
* You show that sample efficiency improves in the task specific regime when the world model approaches the environment distribution, however is it always the case that the world model improves in this sense?  Is there reason to believe that this will be very limited for some environments?
* Was any study undertaken of the number of particles and what effect this has on posterior sampling and learning convergence?  What impact does this have on the computational complexity?

limitations: I couldn't see much discussion about limitations.
review_title: null

Meta-review: metareview: Unanimous accept from 3 experienced reviewers with good confidences

Important topic (lifelong RL), clearly explained, well-formulated with a variational and hierarchical Bayesian model, evaluated on a range of relevant experiments in discrete and continuous settings in MuJoCo, and also MetaWorld MT10 & MT50 domains in response to reliever osbM and thFd questions, adapted to the lifelong setting of rolling out tasks sequentially, to include like opening a boxes/closing drawers/opening windows.
confidence: Certain
Decision: Accept