Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents an algorithm called EfficientZero, a sample efficient model-based RL algorithm built upon MuZero. The authors demonstrate the efficiency of the proposed method via atari games, and call for the research of MCTS based RL algorithms in a wider community. The core contribution lies in the adaption of MuZero to settings where data is limited. The paper is overall interesting, easy to follow, and technically sound. 
main_review: Originality & significance:  I have concerns regarding the novelty and applicability. From the perspective of methods, we have seen the application of model based, model free, and planning to atari games. This paper is a direct enhancement of MuZero, I would like to see how sample efficient it is on the testbeds used in Muzero, e.g., board games, with limited data settings. Since the vision of this paper claims to contribute a step towards real world RL applications such as robot manipulation, I would like to see some discussions on how the proposed method would be a step forward since atari is still a toy game mostly used in academia. At least the authors are encouraged to conduct experiments on simulation environments (e.g., continuous action space), which is similar to what "Mastering Atari with Discrete World Models" did.  
Also, MCTS based methods usually suffer from handling real time senarios, where  a significant real world application requires quick response,e.g., online gaming, auto driving, robot manipulation, etc. A discussion on this point is necessary, since the authors hope to accelerate applicability of mcts based reinforcement learning. 

Quality: the paper is technically sound; the experiments design for atari is ok. But I would like to see more experiments on other cases with limited data. As claimed by the authors, EfficientZero is ought to be a sample efficient model based RL algorithm for handling cases with limited data. 

Clarity: the paper is overall easy to follow, but with some grammatical issues. The authors need to improve the wording. 
limitations_and_societal_impact: I would suggest the authors to provide a thorough discussion on the limitations of this work, compared to existing works.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors consider the problem of sample-efficient reinforcement learning focused on image-based data, particularly in the Atari domain. Building on the MuZero model, they introduce three key changes to improve the learned model and thereby the sample efficiency: (a) a loss to enforce consistency between rolled-out and direct state abstractions learned by the model; (b) an end-to-end approach for predicting the sum of discounted rewards ("value prefix") used to generate a MCTS Q-value estimate, and (c) the use of an additional MCTS rollout for bootstrapping the state-value estimate with replay buffer data, to limit the off-policy bias of multi-step estimates from old data. They show that the proposed changes result in a significant performance boost in the low-data (100k) Atari domain, over both the original muZero and a variety of sample-efficient baselines. The also provide an ablation study that demonstrates improvement from the combination of the three interventions over each individually. The authors also release the source code.
main_review: **Quality:**

The interventions generally seem quite reasonable, well-motivated and applicable to other tasks, and the empirical results are compelling. The paper has a good related work section (including quite recent results like [9] that work on similar themes). The authors also release their source code (I assume the url will be included in the supplementary in the final submission).

One key point that was not clear to me is the computational cost of the MCTS bootstrapping in the off-policy correction. What is the difference in train time with and without this?

I'm not fully convinced by the self-consistency reconstructions on page 2 in the appendix. The decoder is trained independently to reconstruct the observations from the state abstractions s_t? The decoder (particularly in the middle row) appears to be reconstructing impossible states, whereas I would assume that in the failure state it would simply reconstruct a static image of a probable game state (independent of s_t).

Can you list MuZero as well in Table 2? 

**Originality:**

The paper proposes several improvements on top of the existing MuZero architecture. The self-supervised consistency loss appears to have been concurrently proposed (though not yet peer-reviewed) by de Vries et al., and the imaginary rollouts have been used by others albeit without the fine-grained tradeoff with real-world experience. The "value prefix" learning seems to be the most original idea and is compelling; although it seems to be the weakest intervention according to the ablations, the authors do show an additional train vs. validation comparison to illustrate the effectiveness. Altogether, I believe that illustrating the individual and combined benefits of the interventions and releasing the code illustrates a strong contribution to the community.

**Significance:**

As mentioned, the gains in overall performance are impressive. It would be nice to see results in one other problem domain (e.g. 3D exploration); the focus on Atari in the RL community risks overfitting to a single domain, although the proposed interventions don't appear to be domain-specific.

**Clarity:**

The aleatory uncertainty point (Line 177) is used to motivate the value prefix loss. I found both this and the explanation in 4.2 to be somewhat confusing, although I see the point after some reflection. I think it would help to point out that the network might equally assign responsibility for a reward to multiple states in the future, leading to an additive error in the estimate.

Can you explain / expand on the role of the projector in the main text in Section 4.1?

Figure 4: why do you report two different losses in train and validation?

Minor text suggestions:

Line 109: ...and use[s] model-based value estimate[s] to bootstrap.

Line 135: Open bracket in \tilde{V}(s_{t+k}) is superscript

Equation 2: Can you add brackets around the argmax argument for clarity?

Line 175: [Ne]ither the reward ...

Line 182: Off-policy [nature] of multi-step value

Line 221: ...adjacent populations provide [natural] two views... (natural is confusing here)

Line 229: The state aliasing problem has [a] negative...

Figure 2 caption: ...the right player didn't move and miss[ed] the ball.

Line 296: ...since it requires [fewer] samples.

Line 303: ...run[s] MCTS search...

Line 304: ...into [a] CPU part and [a] GPU part...

Line 307: ...to avoid performance issue[s] with Python...

Line 331: ...which also first propose[d] the Atari...

Line 333: ...a method that use[s] contrastive...

Line 346: ...our limit[ed] data scenario...

Line 360: ...can outperform [a] human player....

Line 361: ...our method outperform[s] human[s] in...

Line 365: ...three issues that prevent[s] MuZero (remove s)

Line 373: Due to limited computation resources... (no on)

Line 361: cause the [difficulty] in predicting...
limitations_and_societal_impact: No issues.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents EfficientZero - a variant of MuZero that targets data efficiency in the low-data regime. The authors identify 3 factors that hinder the data efficiency of MuZero in the low-data regime, namely lack of supervision on learning the model, compounding errors in value estimation, and lack of off-policy correction with stale data. To provide richer supervision for learning the model, EfficientZero employs an auxiliary self-supervised objective similar to SimSiam. To combat compounding errors, EfficientZero directly learns the "value prefix" rather than per-step rewards. Finally, EfficientZero utilizes the model to address off-policy correction. EfficientZero achieves a new state of the art on the Atari 100k benchmark, outperforming existing methods by a large margin. Ablation studies demonstrate the importance of all three proposed components - removing any of them results in significant performance drop.
main_review: Things to like:
* 1. Overall the paper is well written. The authors successfully stated the problem of interest, their hypothesis on the potential issues with current methods, their proposed solutions, and the empirical verification. The writing is easy to follow. Although there are some minor errors and typos, they do not hinder the readability.
* 2. EfficientZero addresses important problems in value-equivalent models and model-based RL. MuZero, and other value-equivalent models, all suffer the problem of lacking learning signals. The compounding error is also a well-known issue in the model-based RL context. Admittedly, the solutions used by EfficientZero are not completely novel and are mostly variants of existing methods. However, the empirical success of applying them to state-of-the-art agents like MuZero is still significant. I believe the community can benefit from the success of EfficientZero.
* 3. The empirical results are very impressive. EfficientZero achieves a new state of the art on the Atari 100k benchmark and is the first to achieve super-human performance in terms of the mean human-normalized score. This is a big step forward.
* 4. An open-source implementation of MuZero/EfficientZero can be a valuable asset to the community.

Things that can be improved:
* 1. There is some inconsistency between the identified issues and the proposed solutions. In L40-42, it says "... three components are essential to the image input model-based RL agent: ..., ..., and an efficient use of the model to explore the world". I understand the third component as using the model for better exploration. However, the corresponding solution is to use the model for off-policy correction. Unless I misunderstand something, this seems a bit inconsistent to me. Similar confusion was found in L256 as well. "[predicting the value prefix] helps the MCTS to explore better". I do not fully understand the connection between less compounding error and better exploration. 
* 2. Asadi *et al* [1] proposed a multi-step model to combat the compounding error, which also directly learns the "value prefix". The authors should as least mention it in the related work.
* 3. For model-based off-policy correction, MuZero already uses the value output of MCTS as the bootstrap target (see the text associated with Eq. (1) in [3]). Section 4.3 is not clear on this. What is new here is a heuristic for choosing the unroll length $l$.
* 4. The authors could include a bit more discussion on the relative importance of the three components in the ablation study section. Based on the results in Table 1 in the appendix, removing the consistency loss seems to cause the biggest performance drop. Does it imply that richer learning signals are what MuZero lacks the most in the low-data regime?

Questions for the authors:
* 1. Since off-policy correction is crucial in this low-data regime, did the authors try existing methods like Retrace($\lambda$) [2]? The current solution may be too aggressive at truncating the trajectories, which may cause slower learning.
* 2. Could the authors comment on the impact of the three components in the high-data regime, like 200M frames? The off-policy correction may be less effective in a more online setting as the problem itself is less severe. But the other two components may still help.

Minor Errors and Typos:
* 1. L22, a reference to AlphaZero is missing.
* 2. L33, "Muzero" --> "MuZero".
* 3. L88, duplicated reference [37].
* 4. L151, "... prioritize explore promising part of the tree" --> "prioritize exploring...".
* 5. L350-351, the term "self-play network" pops up from nowhere.

References:
* [1] Asadi *et al*, 2019, *Combating the Compounding-Error Problem with a Multi-step Model*.
* [2] Munos *et al*, 2016, *Safe and efficient off-policy reinforcement learning*.
* [3] Schrittwieser *et al*, 2020, *Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model*.
limitations_and_societal_impact: There is no foreseeable negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose EfficientZero, a sample efficient model-based visual RL algorithm built on MuZero. By making three changes, namely using self-supervised learning to learn the environment model, predicting the value f, and using the model to correct off-policy value targets, EfficientZero achieves state-of-the-art performance on the Atari 100k benchmark. The authors also provide an efficient implementation for future research.
main_review: The proposed approach is the first one to achieve human-level (or comparable) performance with limited data. The three modifications have been studied by ablations, and every of them is shown effective respectively. Besides, the paper is well structured and easy to follow. However, I still have the following comments that the authors need to address. 

1. MuZero Reanalyze in [30] uses a target network to provide the n-step bootstrapped target value, while EfficientZero uses MCTS root values in off-policy correction. In addition, EfficientZero in this paper also uses dynamic n (denoted by l). The ablation tests in this paper should separate these two issues (using MCTS root values and dynamic n) for analysis. It is more interesting to see which is more significant. BTW, it is critical to show the required computation costs like Table 1 in [30]. 
2. In the experiments (Section 6.5 and Appendix), the ablation study of removing three components (the self-supervised consistency loss, the end-to-end value prefix learning, and the model-based off-policy correction) are applied to 3 games only in Section 6.5 and 5 other games in Appendix. It would be better if these can be applied to all the other games. 
3. This paper does not do ablation tests for data augmentation, which has been shown to have non-trivial improvement of the performance in SPR and DrQ. However, it is still interesting to see ablation tests for EfficientZero (without data augmentation) to see how significant it is when compared with other components. 
4. This paper states “It is worth to note that MuZero does not explicitly learn the environment model. Instead, it solely relies on the reward and value prediction to learn the model.” However, the policy signal should also be involved for learning the environment model.

Following are related to the presentation. 
Line 33: Muzero -> MuZero
Line 74: reference of SimSiam should be [7]
Line 88: such as Go, Chess and Shogi [37, 37] -> [36, 37]
Line 148: To be precise, Formula (2) is called a variant of the PUCT algorithm.
Line 175: “Either” the reward “or” the value “provide” enough training signal -> “Neither” the reward “nor” the value “provides” enough training signal
Line 178: the predicted “reward” still have large prediction errors -> “rewards”
Figure 1: “next-state” component should be “dynamic” or “transition”
Line 229: an important “roll” in MCTS -> “role”
Line 267: “use l rewards” should be “use l-step rewards”
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The reviewers were in universal agreement the paper proposed a novel, well justified method with good empirical results. Please include results on mujoco; it would be particularly interesting to include the ablation where MCTS is only used to generate learning targets, but not to act, in order to improve response time (you suggest this ablation in the rebuttal, and is also investigated in [1]). Also, if possible, please discuss or include some experiments in the 'data-rich' regime (high number of training frames). A method that strongly improves performance in the low-data regime, but catastrophically degrades performance as the amount of experience scales up, is of more limited interest than one which is data efficient while retaining good performance in data-rich regime. 

[1] On the role of planning in model-based deep RL, Hamrick et. al

Decision: Accept (Poster)