Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper investigates how and why planning might be beneficial in model-based reinforcement learning settings. To that end, the authors ask three questions on planning in MBRL: (1) How does planning benefit MBRL agents? (2) Within planning, what choices drive performance? (3) To what extent does planning improve generalization? In order to answer these questions, the authors investigate the performance of MuZero in a variety of learning challenges while systematically ablating the algorithm to find how each part of the algorithm effects the overall performance.

In its current form the paper is marginally below the acceptance threshold.

The reasoning for my judgement is as follows:
The paper makes strong claims about how planning effects model-based RL algorithms based on their experimental results. However, the results are only based on five different seeds. Given the fundamental conclusions the authors formulate, the results are not statistically relevant enough. (For further intuition about the impact of a small number of seeds in DRL, please refer to ‘Deep Reinforcement Learning that Matters’, Figure 5 by Henderson 2017). The strong language in combination with the limited statistical relevance result in the experimental design not being sufficient.

Having that said, the paper itself is well written and provides important insights to the community. I would strongly encourage the authors to either adjust the language or (better) run additional experiments to strengthen the paper. With more runs for each of the experiments, I would recommend a clear accept.

Minor comment: At the beginning of the second paragraph, the authors state ‘Many have suggested that models will play a key role in generally intelligent artificial agents’. The supporting papers are essentially just two different author-sets, making this another instance of too strong of a statement with too little foundation.

-----------------------------------------------------------------------------------------
After reading the authors response to all reviewers, I believe all questions to be sufficiently addressed. I will therefore (happily) raise my score.

review_title: Official Blind Review #4

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary:
This paper analyzes the role of planning in the model-based reinforcement learning agent, based on evaluating MuZero on eight tasks (i.e. Ms.Pacman, Hero, Minipacman, Sokoban, 9x9Go, Acrobot, Cheetah, and Humanoid), which have discrete action spaces. The conducted experiments show three major implications: (1) Of the three parts in which search is used (i.e. search at evaluation time, search at training time for exploration, and using search result as a policy target), the role of serving as a policy improvement target was most substantial. (2) Deep tree search did not make a significant contribution to performance, and a simple Monte-Carlo rollout could be performant enough for MBRL. Also, a too small or too large search budget can be harmful to the performance of the MBRL agent. (3) Search at evaluation time was helpful for zero-shot generalization especially when the model is accurate.


pros:
- The paper is well-written and well-organized. Hypotheses and the experiments are well-designed and seem thorough.

cons:
- The analyses are limited to MuZero that deals with discrete action space and to the deterministic environments. Since MuZero is a particular instance of MBRL, where only the reward and value prediction are performed in the latent-state space, it is unclear that the conclusions of the work can be applied to other instances of MBRL (e.g. MBRL methods dealing with dynamics model that operates on the original state space). I am not fully convinced that the results here can be generalized to other classes of MBRL.


some questions and comments:
- It is interesting that the main benefit of the search is serving as a policy target. What are the advantages of policy improvement through search in MuZero compared to directly optimizing policy using analytic gradients in latent space like Dreamer (Hafner et al, 2020)?

- In Figure 4c, why does the performance get worse as the number of simulations increases in Acrobot and Cheetah? Or in Figure 6 (Simulator, Green line), why the performance of (# Simulations = 3125) is worse than (# Simulations = 125)? This may not be explained as compounding model errors since the agent is using the exact simulator.

- It would be have been great to see the learning curve even for the results of Section 4.3, similarly to Appendix D2 that presents the learning curve for the results of Section 4.2.


review_title: Good analysis for the role of search in MBRL

final_score: rating: 5: Marginally below acceptance threshold
review_content: Overview of the paper: 
This paper studies three empirically questions about the planning part in Muzero, an algorithm integrating direct learning, model-learning, and planning. These three questions, as written in the paper, are that 1) for what purposes is planning most useful? 2) what design choices in the search procedure contribute most to the learning process 3) does planning assist in generalization across variations of the environment. The paper answers all these three questions using experiments: 1) the major reason for the performance improvement using planning is maintaining a policy that approximates the policy found by the MCTS algorithm, 2) simpler and shallower planning is often as performant as more complex planning, 3)  search at evaluation time only slightly improves zero-shot generalization.

Comments:
Overall I think the paper is not ready to publish. 

While the 3 questions asked in the paper are quite general and apply to many model-based algorithms, the paper provided general answers to them using empirical results only for the Muzero algorithm under deterministic environments, which doesn't look appropriate to me. The Muzero algorithm is different from many other planning algorithms, such as Dyna-style planning, MPC, value iteration, etc. And deterministic environments are easy cases compared with stochastic or even partial observable environments. I would suggest the authors rephrase the questions and answers to make them more specific so that the results in the paper can support conclusions well.

While there are multiple things making me confused, I would like to highlight the following one as an example because that almost makes me think their answer to their first question is incorrect. The answer is drawn from the results shown in figure 3, which illustrates how important three design choices (following MCTS policy in both training and testing, following MCTS policy in training and prior policy in testing, and following prior policy in both training and testing) are in planning. These results could tell us how much more performance the algorithm achieves by following MCTS policy in training or testing, but they can not tell us how much more performance the algorithm achieves by updating the prior policy towards the MCTS policy compared with other approaches. That is, in all cases, the algorithm updates its prior policy towards the MCTS policy. 

The other thing I feel not appropriate is, while the paper claimed that "we systematically study the role of planning and its algorithmic design choices in a recent state-of-the-art MBRL algorithm, MuZero", the planning part is not the only part that varies across different design choices. In particular, because the model parameters are learnable and all the variations of algorithms they tested have different updates to model parameters. The resulting learned models are different in these variations. Thus it is not appropriate to conclude that the performance difference between different variations is solely the result of planning. It might also come from differences in learned models.

Thanks for the author's detailed response.

In terms of the first question, I do appreciate the value of the paper as a nice empirical study of Muzero and other similar MBRL algorithms. Meanwhile, many MBRL algorithms are not like Muzero. For example, value-based planning algorithms don't maintain an explicitly parameterized policy. Therefore the conclusions here may not apply to all cases. Making its conclusions more precisely will not undermine the value of the paper, instead, it provides readers clearer results. I do see in the revised version, the authors changed their language in the discussion about the result. But maybe clearer results themselves are better.

My second concern is addressed in the updated version of the paper, with additional experiments. Cool!

In terms of the third question, after reading your response, I think there is a very interesting question. When we test planning algorithms, should we give the agent a fixed model and a fixed representation or fixed algorithms learning the model and the representation? After thinking for a while, I can see the advantages and disadvantages of both cases. So I would change my mind and agree with the authors that their choice of testing is valid. But I do hope this choice being mentioned in the paper because people like me would typically consider the other one.

I would like to raise my score to 5.


review_title: Needs to rephrase claims and design experiments more carefully.

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper tries to disentangle the role of planning in model-based reinforcement learning with a number of different ablations and modifications to MuZero. Specifically, the authors analyze the overall contribution of planning by omitting planning from which it is originally used in MuZero, and investigate different planner settings that can drive performance. In addition, they check the generalization advantage of MBRL. Overall, the paper is well-written, and experiments are conducted appropriately. The results provide some insights that other researchers in the MBRL community can leverage for their future work. My major concern is the lack of direct ablation study that can clearly show the advantage of planning in providing a good learning signal. See the detailed comments below.

Comments:

* To bolster the argument that planning contributes the most in providing a good learning signal, “(Data)” or “(Data+Eval)” ablation can be done:  training update is done with targets generated with MCTS of $D_{\text{UCT}} = D_{\text{tree}} = 1$ while the agent uses full MCTS during training rollouts. Since the current experiments do not ablate planning in training, the effect of planning is only implicitly examined.
* In the extreme, model-free version of “(Data)” could be implemented in which $q$ is trained instead of $v$ while $v^{\text{MCTS}}$ and $\pi^{\text{MCTS}}$ is replaced with $\max q$ and $\arg\max q$. If this change is viable, it would also provide a chance to confirm the effect of planning for exploration only.
* BFS is a too weak planner to compare with. How about random shooting methods?

Styles & Typo:

* Use “\citet” when appropriate; when a citation is used as a subject in the sentence.
* Line 3 of Algorithm 2 in Appendix: $k=1 \dots S$ → $k=1 \dots B$ ?

review_title: Good ablation study that provides some insights on the role of planning in MBRL.

Meta-review: The reviewers appreciated the author replies, the additional experiments (more runs but also more ablations/baselines), and the updated paper. Also R2 is now largely satisfied (but seems to have been too late to post a public reply or to raise the score of the review).

The paper provides important insights in model-based RL and its connections to planning, by studying MuZero with systematic ablations. Hence a valuable contribution to the community. All (major) cons have been addressed in the revision.
Decision: Accept (Poster)