Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper measures the generalization ability of model-based agents, i.e. MuZero, in comparison to model-free agents. The authors identify three key factors for procedural generalization: planning, self-supervised representation learning, and procedural data diversity. However, they find that these factors do not necessarily provide the same benefit for task generalization. They argue for a move towards self-supervised model-based agents trained in rich, procedural, multi-task environments. 
main_review: The paper systematically investigates how MuZero, a SOTA model-based agent, performs on procedural generalization (in Procgen) and task generalization (in Meta-World). They identify three factors that improve procedural generalization through careful experiments. Next, they test their effect for task generalization and find only a weak positive transfer to unseen tasks. However, the experiments and results testing the effect of planning and data diversity for task generalization were hard to follow.
summary_of_the_review: The authors provide a thorough evaluation of MuZero, a model-based agent, on procedural generalization in Procgen and task generalization in Meta-World. They identify three factors that improve procedural generation and demonstrate challenges in task generalization.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explores the application of the MuZero agent for tasks which require generalization across environments, namely ProcGen and MetaWorld. 
On ProcGen, they find that MuZero in its standard form performs on par or better than strong model-free methods. Furthermore, when combined with auxiliary self-supervised learning (SSL) losses, there is a significant jump in performance which achieves a new state of the art. The paper includes interesting control experiments disentangling the effects of different components. For example, it shows that both MuZero’s modified targets for the value functions, as well as the tree search for action selection, each separately contribute to performance. Another interesting finding is that adding auxiliary SSL objectives can help generalization performance on unseen environments, even when they do not improve performance on the training environments, which I found surprising but useful. 

Results are also reported on task generalization benchmarks from MetaWorld. Here the results are less strong, and self-supervision does not appear to help. There appears to be some transfer between tasks, but it is limited. 
main_review: Pros:
- very clear and well-written paper
- very strong results on ProcGen
- informative ablations and insights
- novel application of model-based methods to procedurally generated environments

Cons:
- limited novelty of methods 
- the authors do not mention code release in their reproducibility statement

In section 4.2, is there a reason why MuZero is only trained for 50M while the baselines are trained for 400M steps? It would be helpful to see how MuZero performs when it is run for 400M steps. Or is this too slow? If so, this should be noted somewhere. If it is too slow, then it would be helpful to compare to the baseline performance at 50M steps.
summary_of_the_review: Overall this is a strong paper and I recommend acceptance. It investigates the application of a model-based RL (MBRL) to procedurally generated environments, which contrasts with most existing MBRL works which on run on singleton environments. In addition to strong results on ProcGen, the ablations are quite informative in understanding the effect of the different algorithmic components. The MetaWorld results are a bit disappointing, but these are nevertheless helpful to include.

My main issue with the paper is that the authors do not mention code release in their reproducibility statement. While the detailed appendix is appreciated, this is not a substitute for releasing code (I am also not aware of thoroughly tested open-source implementations of MuZero). I strongly urge the authors to make their code open source, otherwise it will be difficult for the research community to build on this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper evaluates how well model-based RL, specifically MuZero, generalizes in comparison to model-free RL. It empirically compares how planning, representation learning, and data diversity affect the generalization of agents. To evaluate the effect of planning, a Q-learning agent is constructed to be as similar as possible to MuZero without the MCTS. In experiments, it is found that planning, self-supervised representation learning (reconstruction, contrastive, self-predictive) and data diversity all improve generalization performance. However, results are not similar in the Meta World benchmarks, where self-supervision did not seem to improve results much. The paper concludes that self-supervision is a promising approach to improving the generalization of MBRL agents in procedural environments, but perhaps it does not improve task generalization.
main_review: Strengths

- While prior works have shown some sort of self-supervised representation learning to be important in learning world models [1], this work seems to be the first to evaluate its effect on generalization. Since multi-task RL and generalization seems to be an active research area, I think this is a valuable contribution.
- The paper shows that some conclusions as to generalization performance cannot be drawn when using a small training set, which suggests researchers should evaluate their methods on diverse sets of environments to see whether the method affects generalization.

Weaknesses

- The paper talks about procedural generalization and task generalization, where the former corresponds to generalizing to unseen configurations of an environment with the same reward function and the latter corresponds to unseen reward functions in the same environment. Examples of each include ProcGen and ML-1 for procedural generalization and ML-10 and ML-45 for task generalization. To me, it is unclear that ML-10 and ML-45 exactly fit into the category of "task generalization", since there are different types of objects (different environments) in the different task instances.
- ML-10 and ML-45 have significantly fewer training environments than the proc-gen tasks, where the paper claims that the improvement of self-supervised representation learning is less apparent with lower data diversity. Perhaps the results on task-generalization are simply due to this? I am curious as to whether there is truly a difference in generalization performance when measuring generalization to new reward functions rather than new environment configurations. Perhaps there is a better environment to test this in where the data diversity can be controlled for?

[1] Babaeizadeh, Mohammad, et al. Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning. [https://arxiv.org/abs/2012.04603](https://arxiv.org/abs/2012.04603)
summary_of_the_review: This paper shows that self-supervised representation learning not only improves the training performance of MBRL but also the generalization performance and also points out that researchers may want to evaluate their algorithms on tasks with high data diversity. However, I feel as though the argument that task generalization differs from procedural generalization is not well supported by the experiments. I do not recommend accepting this paper in its current form.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a systematic empirical study of the effect of planning and model learning on generalization performance, using the MuZero agent. They use 2 environments, Procgen and Meta-World, to respectively explore procedural generalization to new variants of the environment with the same reward structure and task generalisation to new structures of the reward function in the same environment. Their main contributions are their empirical results, specifically that additional reconstruction or self-supervised losses enable the MueZero agent to achieve state-of-the-art performance in procedural generalization, but are not enough to promote a similar increase in performance in task generalization tasks in Meta-World. Finally, they also explore the data diversity dimension, showing that having more diverse data during training can help procedural generalization even more.
main_review: Strengths: 

- The paper is very well written and the motivation and findings are clearly presented and easy to follow.
- They provide a good ablation study of the different losses and explored a few interesting cases.

Weaknesses:

- The main weakness of the paper lies in that the scope is somewhat limited to an empirical study with only a small number of take-away learnings for the community, which are not necessarily very surprising. 
- Minor: “Although significant strides have been made in model-based systems in recent years [27], the most popular model-based benchmarks consist of identical training and testing environments [e.g. 26, 68] and do not measure or optimize for for generalization at all.” remove double “for”.

summary_of_the_review: The work is very clearly presented, easy to understand and presents a number of ablation cases on some important considerations for RL agents, e.g. model-free, planning and model learning. The learnings from the work are clear, though not necessarily very surprising.
review_title: null

Meta-review: The paper evaluates the generalization capabilities of model-based agents, in particular, MuZero, compared with model-free agents. Reviewers agree that the paper is well-written and the topic is interesting. The ablation study is especially interesting, as it disentangles the effect of different algorithmic components. Some concerns are raised about the significance of this work, as the scope is limited to an empirical study and the results are not necessarily very surprising. 

Since the paper presents clear results on an important and relevant topic, I recommend acceptance.
Decision: Accept (Poster)