Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposes a Hierarchical Bayesian approach for lifelong RL. The global world-model posterior models the world model shared across tasks and the task-specific model learns the dynamics within a specific task. The task-specific model achieves forward transfer by initializing from the global world model. The authors use mean-field variational approximation to scale the proposed model. Also, the authors introduce sample complexity analysis. The method is evaluated on two toy tasks (grid-world and box jumping) and one on MuJoCo simulator and showed superior performance to the previous works. 
main_review: 
- Clarity of the paper should be improved significantly. The current version makes understanding hard. 
     - The paper does not describe the notion of many notations (e.g., the indices j and k in Figure 1.) and denotes some distributions in its short-form, e.g., q_e or q_\theta^{m_i}, without introducing its full form. These are not just a few, but observed broadly across the entire paper, perhaps Section 3 requires the most significant improvement. 
     - The paper also explains the proposed method by directly explaining lines of the pseudo-code without discussing the big picture or the rationale behind the design.
     - It also skipped describing the full joint distribution of the model which is essential in describing such probabilistic models.
- The key idea of the proposed method is quite simple. It's like applying Variational Continual Learning to the world model learning in the RL setting. It can also be understood as Bayesian meta-learning but with sequential task exposure. 
- The experiment is also quite simple. The first two toy tasks are quite too toyish, very low-dimension and tow task complexity. The last MuJoCo experiments show the superiority of the proposed model. 
- The algorithms (e.g., variational continual learning, Bayesian meta learning, and mean-field variational inference) on which the proposed method is based are known to be not scalable in more complex settings like high-dimensional data (like image), many sequential tasks (e.g., 1000 or more tasks), and the mean-field approximation has significant limitation in its expressiveness. So, I'm somewhat doubtful if this method can be an important milestone toward more realistic and complex settings. 
summary_of_the_review: The writing clarity requires a significant improvement. It's currently a major drawback hindering the understanding of the proposed model (I understood at the later part of the paper but it was hard until reaching there). The evaluation is quite simple and uses toy tasks. I'm doubtful about the potential of the proposed model to extend to more realistic and complex settings such as image inputs.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This submission presents an approach for Bayesian model-based exploration in a lifelong RL setting, building upon existing approaches for Bayesian exploration (BOSS) and Bayesian multi-task modeling (HiP-MDPs). The approach keeps separate models for sampling transitions and rewards for each task, and each task model is drawn from a shared prior that models the distribution over tasks. The method continually updates the shared model with data from all observed tasks and the task-specific model with data from the current task. To achieve backward transfer, the approach replaces the task model with the shared model whenever the task model has not been sufficiently trained on a particular state-action pair. 

main_review: ############## Strengths ##############

1. The overall idea of model-based lifelong RL is very relevant, particularly since lifelong RL precisely seeks to reduce sample complexity
2. The high-level idea of replacing the task model with the world model whenever the task model is uncertain is intuitively appealing
3. The use of the task model permits deriving reduced sample complexity bounds thanks to the Bayesian formulation

############## Weaknesses ##############

1. The low performance of the agent in the more complex MuJoCo evaluations makes the results unconvincing
2. The technical approach is very closely tied to existing works, in particular HiP-MDPs and variants and BOSS
3. There are no comparisons to existing model-based lifelong RL methods

############## Arguments ##############

The primary contribution of this work is the introduction of one of the first approaches for model-based lifelong RL. Since one of the key desiderata of lifelong RL is to decrease the amount of experience needed to learn new tasks (forward transfer), using model-based techniques, which are inherently more data efficient than model-free techniques, seems to be a promising direction. 

However, the proposed method itself is fairly incremental. In particular, it heavily hinges on BOSS (as an exploration technique) and HiP-MDPs (as a multi-task model-based model). While the use of these models permits relatively simple adaptations of the proofs of BOSS to demonstrate decreased sample complexity (which is a nice plus), it does make the work contain very little technical insights. In particular, when adapting a multi-task method (HiP-MDPs) into a lifelong method, I would have expected to see considerable effort in designing a technique for updating the shared model with knowledge from new tasks. However, the authors simply train this shared model in a multi-task fashion with data from all tasks. This is not only costly in terms of memory footprint (requiring to store vast amounts of data for each task), but is also computationally inefficient, since the multi-task training step is executed iteratively _for each task_ (O(n^2) cost). The one piece of technical insight offered in the submission that I found interesting was the idea of replacing the task model with the world model whenever the task model is not yet well trained. Unfortunately, the approach itself is very heuristic (relying on thresholds on the uncertainty across sampled models) and the heuristic choices are not discussed in detail or validated empirically (e.g., via ablative tests or sensitivity analysis on the thresholds). I intuit that the method is quite sensitive to these thresholds: if they are too high, the task model will almost always be used, even if it's uncertain; if they are too low, the world model will be used too frequently and task performance will likely degrade by reverting to the "average" world model. Would it not be possible to do some sort of soft combination (of the task and world model) weighted by the uncertainty, instead of a hard selection of one or the other? 

However, my main concerns with the submission lie on the empirical evaluation. The first major concern is that all the agents evaluated on MuJoCo domains achieve very low performance, as compared to the results in the papers cited in the submission for the experimental design of these tasks (Mendez et al. and Wang et al.). While it is clear that the reduced performance stems from the choice of using fewer environment interactions (which itself is a nice choice, given the use of model-based techniques), this does raise questions about the conclusions drawn from these results. In particular, how useful or informative are forward and backward transfer if the agent hasn't really learned any meaningful behaviors? It would be useful to include videos of the learned behaviors to assess whether the transfer results are in any way significant from a behavioral perspective, or if they're simply minor reward increases across poor behaviors. 

My second major concern is the choice of baselines for these evaluations. On one hand, the authors chose to only compare against model-free lifelong RL techniques. In this setting, it should certainly be expected that the model-based approach outperforms those baselines simply by the nature of the underlying techniques. This is not novel insight. Yet the authors claim these as general improvements over existing lifelong RL methods, which seems like a stretch. Instead, the authors should have considered existing model-based lifelong RL methods (e.g., [1]) for a more apples-to-apples comparison. Note that [1] is a task-agnostic method, so this would require some adaptation to handle the setting where the agent is given access to task indicators. On the other hand, the authors make claims about forward transfer, but in the MuJoCo domains there is no comparison to a single-task or no-transfer baseline, like used in the box-jumping task. Such a baseline is critical for assessing whether the approach is actually achieving transfer across tasks, since improvements w.r.t. single-task training are precisely what demonstrate transfer. As one additional comment regarding the evaluations, there is no information about the implementation details of any of the baselines, including their model architectures and hyper-parameters. How were these chosen to guarantee a fair comparison?


############## Additional feedback ##############

The following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.

Intro
- I wonder if the example of different houses and toothbrushes matches the HiP-MDP formulation introduced immediately after
- The intro is fairly clear and describes the solution approach well. 
- I'd suggest including an example that more closely matches the HiP-MDP formulation, since this is the formulation adopted throughout the paper.

Sec 4
- The ideas seem to be very closely related to the original HiP-MDP papers, especially the BNN extension of Killian et al. (2017)
- The notation for the BNN needs a fair bit of work. The authors never explain what the "particles" are. Are these the (s,a,r,s') tuples sampled from the sequences given by the combination of CEM and the BNN? This becomes increasingly relevant in 4.1 where the authors define their approach to backward transfer. My understanding is that "aleatory variability" is modeled as the BNN's internal variance, whereas the epistemic uncertainty is measured as the variance in the (mu, sigma) output by the BNN across sampled particles. Is this understanding correct?

Sec 5
- The gird-world evaluation shows nothing about forgetting/backward transfer. 
- Box-jumping: Why no comparison to a single-task variant of the solution? This is required to assess forward transfer. Also no backward transfer measure.
- MuJoCo: again, no single-task learner, so it's unclear if there's forward transfer. Plus, the rewards are very low, so it seems that even VBLRL is not solving the tasks. How useful are these results then? Even though VBLRL is the best, it's not really achieving meaningful behaviors.
- I disagree with the claim that the model "cannot" suffer from forgetting, since certainly the wrong choice of threshold for backward transfer could lead to forgetting.
- How was this hyper-parameter chosen?

Typos
- Sec 2, second paragraph: the task facing a single agent -> the agent facing a single task?

[1] Nagabandi et al. Deep online learning via meta-learning: Continual adaptation for model-based RL. ICLR 2019.

summary_of_the_review: 
Unfortunately, I recommend the rejection of this work. While I agree with the premise of the submission that model-based lifelong RL is a relevant area of research, with potential implications on real-world applications of lifelong RL, the submission as it stands appears to not be ready for publication. On the technical side, the approach seems to add just a few incremental changes to multi-task HiP-MDPs to adapt them to the lifelong setting. This on its own is perhaps relatively minor, since the novelty comes from adapting it to a new problem setting. However, such technically incremental contributions should generally be accompanied by strong empirical evaluations, which is not the case in this work. In particular, the low overall performance of all agents on MuJoCo domains suggests that none of the agents are learning to achieve meaningful behaviors, which raises questions about the conclusions reached by the authors. Moreover, the authors should have compared (at least qualitatively, but ideally also empirically) to existing work in lifelong model-based RL. On the flip side, the submission does include an interesting insight of replacing the task-specific model with the shared model whenever the task model is uncertain.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper deals with the problem of lifelong RL, also referred to as meat-RL, where an agent attempts to solve a sequence of tasks in order to facilitate the solution of a novel task. The framework follows that of Baxter 2000 (albeit that paper deals with supervised learning), and has been widely studied in recent years.  The basic assumption is that the tasks are drawn from an underlying task-distribution, and each task (an MDP) is stochastically selected from  a task-specific distribution. The authors work with a Bayesian framework, assuming a hierarchical distribution of the two levels, and learn the two levels separately. This framework has the advantage of providing both estimates and uncertainty estimates. For the discrete case they present a sample complexity analysis, and suggest a variational approach to practical learning. Finally, experiments are provided supporting the utility of the approach. 

The formal framework is that of hidden-parameter MDPs (HiP-MDPs) from Doshi-Velez 2016, and each MDP is modeled based on a transition and a reward model based on a hidden parameter. As more tasks are encountered the posterior over world models sharpens, and, being used to learn new tasks, is expected to facilitate learning. The learning of each new task is as in BOSS, and takes place by sampling from the learned MDP distribution, creating a mixed MDP, and using standard model-based approaches to solve these. 
main_review: The main theoretical contribution suggested in the paper is a PAC-MDP of Theorem 1 for a single task. This theorem is based on Lemma 2, for which a full proof is not provided in the main text or appendix, so its veracity cannot be verified. Moreover, it is based on the assumption that the posterior is consistent, which I believe is what needs to be shown in a meta-learning setup, and cannot be assumed. The form of the bound is also strange as it depends on \delta rather than on \ln(1/\delta) as in previous bounds (e.g., Strehl 2006 and Asmuth 2009), and its dependence on \gamma and \epsilon is also worse than previous bounds.  As far as I understand this is a bound for the single instance setting rather than for meta-learning. 

Following the theoretical part the authors develop a variational approach based on probabilistic networks that model the posterior distribution. As far as I am aware their variational approach is rather standard, although the authors do not refer to previous work. Also, their use of model-predictive control is common in current ML applications, but, again, this is not mentioned or discussed (e.g., the length of the future horizon and how is it selected). 

The authors conclude by presenting numerical simulations for a grid-world and some MuJoco problems for continuous control. While the method compares well to simple baselines it is hard to assess performance relative to more recent work such as Liu et al., "Taming MAML: Efficient Unbiased Meta-Reinforcement Learning" and the other baselines measured in it. 

summary_of_the_review: The paper is phrased within a sequential approach to meta-learning that has been widely studied within the supervised learning community (e.g., Baxter 2000, Pentina and Lambert, "A PAC-Bayesian Bound for Lifelong Learning" 2014 and much later work), with explicit performance bounds. It would be nice to acknowledge these roots. The present approach is plausible, and combines previous work, such as HiP-MDPs, BOSS, variational Bayes, in a sensible manner. However, I do not find that the level on innovation in this combination of approaches suffices for publication at ICLR, nor did I find the theoretical or experimental results of sufficient  interest (see comments above). 

Following rebuttal: Following the authors' response and my response to their rebuttal I have lowered my assigned grade due to my dissatisfaction with their replies, which served to enhance my existing concerns about the paper.
review_title: null

Meta-review: The topic of this paper is timely and important.  However, ultimately the reviewers remained unconvinced that this paper provides a sufficiently clear and sufficiently significant advance to lifelong RL.

As an additional note, the setting under investigation here is not the full lifelong learning setting.  E.g., several of the challenges outlined by Schaul et al. [1] are not treated, and this work is, instead, situated in a somewhat typical multi-task setting with substantlal structure.  That is not bad, but it would be good if this is reflected clearly in all the statements, and, e.g., in the title of the work.

The authors are encouraged to carefully take the provided feedback and see how they can use it to improve their work.  This is an important research direction.  It was just felt the current submission was not quite ready for publication yet.

[1] https://arxiv.org/abs/1811.07004
Decision: Reject