Mastering Atari Games with Limited Data
Authors: Weirui Ye
Shaohuai Liu
Thanard Kurutach
Pieter Abbeel
Yang Gao
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
MCTS
sample-efficient

Abstract: Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train.  Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 194.3% mean human performance and 109.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at https://github.com/YeWR/EfficientZero. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community. 

Mastering Atari Games with Limited Data
Weirui Ye" Shaohuai Liu" Thanard Kurutach" Pieter Abbeel Yang Gao"
"Tsinghua University, UCC Berkeley, t SSannhaii Qi Zhi Institute
Abstract
Reinforcement learning has achieved great success in many applications. However, sample efficiency remains key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an clusive goal. We propose a sample efficient model-based visual RL algorithm built MuZero, which we name EfficientZero. Our method achieves 190.4% mean human performance and 116.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time algorithm achieves super-human performance little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while consume 500 times less data. EfficientZero'ss low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in easy-to-understand manner and is available at https: / //gi thub. com/ YeWR/ /ew//fffi Efficii ieficcenn cieteeerr.. We hope it will accelerate the research of MCTS-bass RL algorithms in the wider community.
Atari games with such
Mean Human kormalined Srove Meari 18341
Median Human hormaized Scove Wear 100k)
2.00
135
1.543
3.00
0.5
0.0
0.33
Ordia OTTaROon CUR
DVD
20%
McDor
Cate
5.00
SenPis OTTRAIbu CURL
D10
57%
Data
Figure 1: Our proposed method EfficientZero 170% and 180% better than the previous SoTA performance in mean and median human normalized score and is the first outperform the average human performance on the Atari 100k benchmark. The high sample efficiency and performance of EfficientZero bring RL closer to the real-world applications.
Introduction
Reinforcement learning has achieved great success on many challenging problems. Notable work includes DQN [24], AlphaGo [33] and OpenAl Five [5]. However, most of these works come the cost of a large number of environmental interactions. For example, AlphaZero [34] needs to play 21 million games at training time. On the contrary, professional human player only play around 5 games per day, meaning would take human player 11,000 years to achieve the same amount of experience. The sample complexity might be less of an issue when applying RI. algorithms in simulation and games. However, when comes to real-world problems, such as robotic
"(ywr20, 1iush20) tsinghua. edu. Bayangiissheiingaaaa... [thanard, kurutach, Bberkeeeyee
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

manipulation. healthcare, and advertisement recommendation systems, achieving high performance while maintaining low sample complexity is the key viability.
People have made a lot of progress in sample efficient RL in the past years [8, 10, 35, 22, 21, 32, 18]. Among them, model-based methods have attracted lot of attention, since both the data from real environments and the "imagined data" from the model can be used to train the policy, making these methods particularly sample-efficient [8, 10]. However, most of the successes are in state-based environments. In image-based environments, some model-based methods such as MuZero [27] and Dreamer V2 [14] achieve super-hume performance, but they are not sample efficient; other methods such as SimPLe [18] is quite efficient but achieve inferior performance (0.144 human normalized median scores). Recently, auggenee and sef--sueer methods applied to model- free methods have achieved more success in the data-efficient regime [32]. However, they still fail to achieve the levels which can be expected of a human.
Therefore, for improving the sample efficiency as well as keeping superior performance, we find the following three components are essential to the model-based visual RL agent: self-supervised environment model, mechanism to alleviate the model compounding error, and method to correct the off-policy issue. In this work, we propose EfficientZero, model-based RL algorithm that achieves high performance with limited data. Our proposed method is built on MuZero. We make three critical changes: (1) use self- supervisd learning to learn a temporally consistent environment model, (2) learn the value prefix in an end-to-end manner, thus helping to alleviate the compounding error in the model, (3) use the learned model to correct off-policy value targets.
As illustrated as Figure 1, our model achieves state-of-the-art performance on the widely used Atari [4] 100k benchmark and it achieves super-human performance with only hours of real-time gameplay. More specifically, our model achieves 190.4% mean human normalized performance and 116.0 median human normalized performance. As a reference, DQN [24] achieves 220% mean human normalized performance, and 96% median human normalized performance, at the cost of 500 times more data (200 million frames). To further verify the effectiveness of EfficientZero, we conduct experiments on some simulated robotics environments of the DeepMind Control (DMControl) suite. It achieves sfate-of-the-at performance and outperforms the state SAC which directly learns from the ground truth states. Our sample efficient and high-performance algorithm opens the possibility of having more impact on many real-world problems.
2
Related Work
2.1 Sample Efficient Reinforcement Learning
Sample efficiency has attracted significant work in the past. In RL with image inputs, model-based approaches [13, 12] which model the world with both a stochastic and a deterministic component, have achieved promising results for simulated robotic control. Kaiser et al. [18] propose to an action- codionncondiiooned video prediction model, along with a policy learning algorithm. It achieves the first strong performance
Atari games with as little as 400k frames. However, Kiclak [19] and van Hasselt et al. [39] argue that this is not necessary to achieve strong results with model-based methods, and they show that when tuned appropriately, Rainbow [16] can achieve comparable results.
Recent advances in self-super learning, such as SimCLR [6], MoCo [15], SimSiam [7] and B BYOL [11] have inspired representation learning in image-based RL. Srinivas et al. [35] propose to use contrastive learning in RL algorithms and their work achieves strong performance on image-based continuous and discrete control tasks. Later, Laskin et al. [22] and Kostrikov et al. [21] find that contrastive learning is not necessary, but with data augmentations alone, they can achieve better performance. Schwarzer et al. [32] propose temporal consistency loss, which is combined with data augmentations and achieves state-of trrt performance. Notably, our self-supervised consistency loss is quite similar to Schwarzer et al. [32], except we use SimSiam [6] while they use BYOL [1] I] as the base suff-super learning framework. However, Schwarzer et al. [32] only apply the learned representations in model-free manner, while we combine the learned model with mrodel-based exploration and policy improvement, thus leading to more efficient use of the environment model.
Despite the recent progress in the sample-efficient RI., today's s RI. algorithms are still well behind human performance when the amount of data limited. Although traditional model-based RI. is considered more sample efficient than model-free ones, current model-free -free-frre methods dominate in

---

terms of performance for image- image-input settings. In this paper, we propose a model-based RL algorithm that for the first time, achieves super-human performance on Atari games with limited data.
2.2 Reinforcement Learning with MCTS
Temporal difference learning [24, 38, 40, 16] and policy gradient based methods [25, 23, 29, 31] are two types of popular reinforcement learning algorithms. Recently, Silver et al. [33] propose to use MCTS as a policy improvement operator and has achieved great success in many board games, such as Go, Chess, and Shogi [34]. Later, the algorithm is adapted to learn the world model at the same time [27]. It has also been extended deal with continuous action spaces [17] and offline data [28]. These MCTS RL algorithms are hybrid of model-based learning and model-free learning.
However, most of them are trained with a lot of environmental samples. Our method is built on top of MuZero [27], and we demonstrate that our method can achieve higher sample efficiency while still achieving competitive performance on the Atari 100k benchmark. de Vries al. [9] have studied the potential of using auxiliary loss similar to our supervuue consistency loss. However, they only test on two low dimensional state-based environments and find the auxiliary loss has mixed effects on the performance. On the contrary, we find that the consistency loss is critical in most environments with high dimensional observations and limited data.
2.3 Multi-Step Value Estimation
In Q-learning [41], the target Q value is computed by one step backup. In practice, people find that
incorporating multiple steps of rewards at once, i.e. 24 = Ex=o ylut+ + U where ut+s is the reward from the replay buffer, ue+k is the value estimation from the target network, to compute the value target 2, leads to faster convergence [24, 16]. However, the use of multi-step value has off-policy issues, since Ue+ii not generated by the current policy. In practice, this issue is usually ignored when there is a large amount of data since the data can be thought as approximately on-policy. TD(A) [36] and GAE [30] improve the value estimation by better trading off the bias and the variance, but they do not deal with the off-policy issue. Recently, image input model-based algorithms such as Kaiser et al. [18] and Hafner et al. [12] use model imaginary rollouts to avoid the off-policy issue. However, this approach has the risk of model exploitation. Asadi et al. [2] proposed a multi-step model to combat the compounding error. Our proposed model-based off-policy correction method starts from the rewards in the real-world experience and uses model- based value estimate to bootstrap. Our approach balances between the off- off-policy issue and model exploitation.
3 Background
3.1 MuZero
Our method is built on top of the MuZero Reanalyze [27] algorithm. For brevity, we refer to it as MuZero throughout the paper. MuZero is policy learning method based on the Monte-Carlo Tree Search (MCTS) algorithm. The MCTS algorithm operates with an environment model, a prior policy function, and a value function. The environment model is represented as the reward function R. and the dynamic function G: r R(st, at), $ G(s,, at), which are needed when MCTS expands new node. In MuZero, the environment model is learned. Thus the reward and the next state are approximated. Besides, the predicted policy p: = acts as a search prior over actions helps the MCTS focus more promising actions when expanding the node. MCTS also needs value function (s:)) that measures the expected return of the node 8t, which provides a long-tery evaluation of the tree's leaf node without further search. MCTS will output an action visit distribution T. over the root node, which is potentially a better policy, compared to the current neural network. Thus, the MCTS algorithm can be thought of as policy improvement operator.
node.
In practice, the environment model, policy function, and value function operate on hidden abstract state st, both for computational efficiency and ease of environment modeling. The abstract state extracted by a representation function on observations o,: 8; = H((r)). All of the mentioned models above are usually represented as neural networks. During training, the algorithm collects roll-out data in the environment using MCTS, resulting in potentially higher quality data than the current neural network policy. The data is stored in replay buffer. The optimizer minimizes the
3

---

following loss on the data sampled from the replay buffer:
C(u,, re) + (Te, + X2
Here, is the reward from the environment, = R(st, 0z) is the predicted reward, TT is the output visit count distribution of the MCTS, pt P({+) is the predicted policy, 2, 7 is the bootstrapped value target and nt = ((*+) (t)) is the predicted value. Specifically, the reward function R. policy function P, value function V. the representation function H and the dynamics function g are trainable neural networks. It is worth noting that MuZero does not explicitly learn the environment model. Instead, it solely relies on the reward, value, and policy prediction to learn the model.
3.2 Monte-Carlo Tree Search
Monte-Carke Tree Search [l, 33, 34, 14], or MCTS, is a heuristic search algorithm. In our setup, MCTS is used to find action policy that is better than the current neural network policy.
More specifically, MCTS needs an environment model, including the reward function and the next- state function. It also needs value function and a policy function, which act as heuristics for the tree search. MCTS operates by expanding a search tree from the current node. It saves computation by selectively expanding few nodes. In order to find high-quality decision, the tree expansion process has to balance between exploration versus exploitation, i.e. balance between expanding a node that is promising with many visits versus expanding a node with lower performance but fewer visits. MCTS employs the UCT [26, 20] rule, i.. UCB [3] on trees. At every node expansion step, UCT will select node follows [14]:
k = =rg max { Q(5 (Q(s.a)+ + ((ss
P(s, a) 22, (s, (
Es bs, +1
1+ (+, C1 + log
(2)
C2
where, Q(s, a) is the current estimate of the Q-value, P(s, a) is the current neural network policy for selecting this action, helping the MCTS prioritize exploring promising part of the tree. During training time, P(s, a) is usually perturbed by noises to allow explorations. N(s,a) a) denotes how many times this state-action pair is visited in the tree search, and ((s, b) denote that of a's siblings. Thus this term will encourage the search to visit the nodes whose siblings are visited often, but itself less visited. Finally, the last term gives a weights to the previous terms.
After expanding the nodes for a pre-defined number of times, the MCTS will return how many times each action under the root node is visited, as the improved policy to the root node. Thus, MCTS can be considered policy improvement operator in the RL setting.
4 EfficientZero
Model-based algorithms have achieved great success in samplc-efficient learning from low- dimensional states. However, current visual model-based algorithms either require large amounts of training data or exhibit inferior performance to model-free algorithms in data-limited settings [32]. Many previous works even suspect whether model-based algorithms can really offer data efficiency when using image observations [39]. We provide a positive answer here. We propose the Effi cientZero, a model- based algorithm built on the MCTS, that achieves super-human performance on the 100k Atari benchmark, outperforming the previous SoTA to a large degree.
When directly running MCTS-based RL. algorithms such as MuZero, we find that they do perform well on the limited-data benchmark. Through our ablations, we confirm the following three issues which pose challenges to algorithms like MuZero in data-limited settings.
Lack of supervision on environment model. First, the learned model in the environment dynamics is only trained through the reward, value and policy functions. However, the reward is only scalar signal and in many scenarios, the reward will be sparse. Value functions are trained with bootstrapping, and thus noisy. Policy functions trained with the search process. None of the reward, value and policy losses can provide enough training signals to learn the environment model.
Hardness to deal with aleatoric uncertainty. Second, we find that even with enough data, the predicted rewards still have large prediction errors. This is caused by the aleatoric uncertainty of the underlying environment. For example, the environment is hard to model. The reward prediction

---

errors will accumulate when expanding the MCTS tree to large depth, resulting in sub-optimal performance in exploration and evaluation.
Off-policy issues of multi-step value. Lastly, when computing the value target, MuZero uses the multi-step reward observed in the environment. Although this allows the reward to be propagated to the value function faster, we find that it suffers from severe off-policy issues and hinders convergence in the limited data scenario.
To address the above issues, we propose the following three critical modifications, which can greatly improve performance when samples are limited.
4.1 Self-Supervised Consistency Loss
In previous MCTS RL algorithms, the environment model is either given or only trained with rewards, values, and policies, which cannot provide sufficient training signals due to their scalar nature. The problem is more severe when the reward is sparse the bootstrapped value is not accurate. The MCTS policy improvement operator heavily relies on the environment model. Thus, it is vital to have an accurate one.
similarity
predictor
stop-grad
projector
projector
$c+1
We notice that the output 8141 from the dynamic function G should be the same as 814-1 i.e. the output of the rep- resentation function H. with input of the next observation Oc+1 (Fig. 2). This can help to supervise the predicted next state 4141 using the actual S(+1, which is a tensor with at least a few hundred dimensions. This provides $1+1 with much more training signals than the default scalar reward and value.
next-state
ar
%/1
S
representation
representation
O2 Figure 2: The self-supervised consis- tency loss.
Oze1
More specifically, we adopt the recently proposed Sim- Siam [7] self-supervised framework. SimSiam [7] is a self-supervised method that takes two augmentation views of the same image and pulls the output of the second branch close to that of the first branch, where the first branch is encoder network without gradient, and the second branch is the same encoder network with the gradient and a predictor head. The predictor head can simply be a two-layer MLP.
Note that SimSiam only learns the representation of individual images, and is not aware of how different images are connected. The learned image representations of SimSiam might not be good candidate for learning the environment transition function, since adjacent observations might be encoded to very different representation encodings. We propose self-supervised method that learns the transition function, along with the image representation function in an end-to-end manner. Figure shows our method. Since we aim to learn the transition between adjacent observations, we pull o, and 0r+1 close to each other. The transition function is applied after the representation of or, such that 8 is transformed to $141, which now represents the same entity as the other branch. Then both of $141 and 6c+1 go through common projector network. Since 8t+1 is potentially more accurate description of 0,+1 compared to $1+1. we make the 0++1 branch as the target branch. is common in self-supers self-supervised learning that the second or the third layer from the last is chosen the features for some reason. Here, we choose the outputs from the representation network or the dynamics network as the hidden states rather than those from the projector or the predictor.
The two adjacent observations provide two views of the same entity. In practice we find that applying augmentations to observations such as a random small shift of 0-4 pixels on the image helps to further improve the learned representation quality [35, 32]. We also unroll the dynamic function recurrently for 5 further steps and also pull $2+4 close to s (k 1, ..... 5). Please see the Appendix for more implementation details.
4.2 End-To-Ena Prediction of the Value Prefix
In model-based learning, the agent needs to predict the future states conditioned on the current state and series of hypothetical actions. The longer the prediction, the harder to predict it accurately, due
5

---

to the compounding error in the recurrent rollouts. This is called the state aliasing problem. The environment model plays an important role MCTS. The state aliasing problem harms the MCTS expansion, which will result in sub-optimal exploration as well as sub-optimal action search.
Predicting the reward from aliased state is hard problem. For example, as shown in Figure the right agent loses the ball. If we only see the first observation, along with future actions, it is very hard both for an agent and human to predict at which exact future timestep the easy
player would lose point. However, to predict the agent will miss the ball after sufficient number of timesteps he does not move. In practice, a human will never try to predict the exact step that he loses the point but will imagine over a longer horizon and thus get a more confident prediction.
Figure 3: A sample trajectory from the Atari Pong game. In this case, the right player didn't move and missed the ball.
Inspired by this intuition, we propose an end-to-end method to predict the value prefix. We notice that the predicted reward is always used in the estimation of the Q-value Q(s, a) in UCT of Equation 2
(80, a) Σ Pret + + *044
where ro+i is the reward predicted from unrolled state $145. We name the sum of rewards k-1 yfre+i as the value prefix, since it is used prefix in the later Q-value computation.
- ,......... in an end-to-end We propose predict value prefix from the unrolled states (St. $t+1.
manner, i.c. malue-prefix = fs; $1+1. -1) Here f is some neural network architecture that takes in a variable number of inputs and outputs scalar. We choose the LSTM in our experiment. During the training time, the LSTM is supervised at every time step, since the value prefix can be computed whenever new state comes in. This per-step rich supervision allows the L.STM can be trained well even with limited data. Compared with the naive per step reward prediction and summation approach, the end-to-end value prefix prediction more accurate, because it can automatically handle the intermediate state aliasing problem. See Experiment Section 5.3 for empirical evaluations. As a result, it helps the MCTS to explore better, and thus increases the performance. See the Appendix for architectural details.
4.3 Model-Based Off-Policy Correction
In MCTS RL algorithms, the value function fits the value of the current neural network policy. However, in practice as MuZero Reanalyze does, the value target is computed by sampling trajectory from the replay buffer and computing: 24 Eb= y'uc+i + y Ut+k. This value target suffers from off-policy issues, since the trajectory is rolled out using an older policy, and thus the value target is no longer accurate. When data is limited, we have to reuse the data sampled from much older policy, thus exaggerating the inaccurate value target issue.
In previous model-free settings, there is no straightforward approach to fix this issue. On the contrary, since we have a model of the environment, we can use the model to imagine an "online experience". More specifically, we propose to use rewards of dynamic horizon from the old trajectory, where I 1< and should be smaller the trajectory is older. This reduces the policy divergence by fewer rollout steps. Further, we redo an MCTS search with the current policy on the last state 8t+1 and compute the empirical mean value at the root node. This effectively corrects the off policy issue using imagined rollouts with current policy and reduces the increased bias caused by setting / less than k. Formally, we propose to use the following value target:
21
o Jutts + 7 MCTS
(4)
where <= and the older the sampled trajectory, the smaller the MCTS (s++1) is the root value of the MCTS tree expanded from 8141 with the current policy, as MuZero non-Reanalyze does. See the Appendix for how to choose I. practice, the computation cost of the correction two times on the reanalyzed side. However, the training will not be affected due to the parallel implementation.

---

5 Experiments
In this section, we aim to evaluate the sample efficiency of the proposed algorithm. Here, the sample efficiency is measured by the performance of each algorithm at common, small amount of environment transitions, i.e. the better the performance, the higher the sample efficiency. More specifically, we use the Atari 100k benchmark. Intuitively, this benchmark asks the agent to learn to play Atari games within two hours of real-world game time. Additionally, we conduct some ablation studies to investigate and analyze each component on Atari 100k. To further show the sample efficiency, we apply EfficientZero to some simulated robotics environments on the DMControl 100k benchmark, which contains the same 100k environment steps.
5.1 Environments
Atari 100k Atari 100k was first proposed by the SimPLe [18] method, and is now used by many sample-efficient RL works, such as Srinivas et al. [35]. Laskin et al. [22], Kostrikov et al. [21], Schwarzer et al. [32]. The benchmark contains 26 Atari games, and the diverse set of games can effectively measure the performance of different algorithms. The benchmark allows the agent to interact with 100 thousand environment steps, i.e. 400 thousand frames due to a frameskip of 4, with each environment. 100k steps roughly correspond to 2 hours of real-time gameplay, which far less than the usual RL settings. For example, DQN [24] uses 200 million frames, which is around 925 hours of real-time gameplay. Note that the human player's performance is tested after allowing the human to get familiar with the game after 2 hours as well. We report the raw performance on each game, as well as the mean and median the human normalized score. The human normalized score is defined as: (scorCagent scorCrandom) / (scorehuman ScOTCrandom).
We compare our method to the following baselines. (1) SimPLe [18]. a model-based RL algorithm that learns an action conditional video prediction model and trains PPO within the learned environment. (2) OTRainbow [19], which tunes the byper-parameters of the Rainbow [16] method to achieve higher sample efficiency. (3) CURL [35], which uses contrastive learning as a side task to improve the image representation quality. (4) DrQ [21], which adds data augmentations to the input images while learning the original RL objective. (5) SPR [32], the previous SoTA in Atari 100k which proposes to augment the Rainbow [16] agent with data augmentations as well as a multi-step consistency loss using B YOL-style self-supervision.
(6) MuZero [27] with our implementations and the same typer-parameters as EfficientZero. (7) Random Agent (8) Human performance.
DeepMind Control 100k Tassa et al. [37] propose the DMControl suite, which includes some challenging visual robotics tasks with continuous action space. And some works [12, 35] have benchmarked for the sample efficiency on the DMControl 100k which contains 100k environment steps data. Since the ACTS-based methods cannot deal with tasks with continuous action space, we discretize each dimension into 5 discrete slots in MuZero [27] and EfficientZero. To avoid the dimension explosion, we evaluate EfficientZero in three low-dimensional tasks.
We compare our method to the following baselines. (1) Pixel SAC, which applies SAC directly to pixels. (2) SAC-AE [42], which combines the SAC and an auto-encoder to handle image-based inputs. (3) State SAC, which applies SAC directly to ground truth low dimensional states rather than the pixels. (4) Dreamer [12]. which learns world model and is trained in dreamed scenarios. (5)
CURL [35], the previous SoTA in DMControl 100k. (6) MuZero [27] with action discretizations.
5.2 Results
Table I shows the results of EfficientZero on the Atari 100k benchmark. Normalizing our score with the score of human players, EfficientZero achieves mean score of 1.904 and median score of 1.160. As a reference, DQN [24] achieves a mean and median performance of 2.20 and 0.959 on these 26 games. However, it is trained with 500 times more data (200 million frames). For the first time, an agent trained with only hours of game data can outperform the human player in terms of the mean and median performance. Among all games, method outperforms the human in 14 out of 26 games. Compared with the previous state-of-the-ar method (SPR [32]), we are 170% and 180% better in terms of mean and median score respectively.
Apart from the Atari games, EffciennZeeo achieves remarkable results in the simulated tasks with continuous action space. As shown in Table 2, EffcientZero outperforms CURL, the previous SoTA,
7

---

to considerable degree and keeps a smaller variance but MuZero cannot work well here. Notably. EfficientZero achieves comparable results the state SAC, which consumes the ground truth states as input and is considered as the oracles.
Table 1:
Scores achieved on the Atari 100k benchmark (32 seeds). EfficientZero achieves super- human performance with only hours of real-time game play. Our method is 170% and 180% better than the previous SoTA performance, in mean and median human normalized score respectively.
Game
Random
Human
SimPLe
OTRainbow
CURL
DrQ
SPR
MuZero
Ours
Alien
227.8
7127.7
616.9
824.7
558.2
771.2
801.5
530.0
1140.3
Amidar
5.8
82.8
142.1
176.3
Assault
222.4
742.0
527.2
351.9
600.6
452.4
571.0
500.1
1407.3
Asterix
210.0
8503.3
1128.3
628.5
734.5
603.5
977.8
1734.0
16843.8
Bank Heist
14.2
753.1
34.2
182.1
131.6
168.9
380.9
192.5
361.9
BattleZone
2360.0
37187.5
5184.4
4060.6
14870.0
12954.0
16651.0
7687.5
17938.0
Boxing
0.1
12.1
2.5
1.2
6.0
35.8
15.1
44.1
Breakout
1.7
30.5
16.4
9.8
4,9
16.1
17.1
48.0
406.5
ChopperCmd
811.0
7387.8
1246.9
1033.3
1058.5
780.3
974.8
1350.0
1794.0
Crazy Climber
107805
21327.8
42923.6
80125_3
Demon Attack
152.1
1971.
208.1
711.8
817.6
1113.4
545.2
3527.0
13298.0
Freeway
0.0
29.6
20.3
25.0
26.7
9.8
24.4
21.8
21.8
Frostbite
65.2
4334.7
254.7
231.6
1181.3
331.1
1821.5
255.0
313.8
Gopher
257.6
2412.5
771.0
778.0
669.3
636.3
715.2
1256.0
3518.5
Hero
1027.0
30826.4
2656.6
6458.8
6279.3
3736.3
7019.2
3095.0
8530.1
Jamesbond
29.0
302.8
125.3
112.3
471.0
236.0
365.4
87.5
459.4
Kanguroo
52.0
3035.0
323.1
605.4
872.5
940.6
3276.4
62.5
962.0
Krull
1598.0
2665.5
4539.9
3277.9
4229.6
4018.1
3688.9
4890.8
6047.0
Kung Fu Master
258.5
22736.3
17257.2
5722.2
14307.8
9111.0
13192.7
18813.0
31112.5
Ms Pacman
307.3
6951.6
1480.0
941.9
1465.5
960.5
1313.2
1265.6
1387.0
Pong
-20.7
1.3
-5.9
Private Eye
24.9
69571.3
58.3
100.0
218.4
-13.6
124.0
56.3
100.0
Qbert
163.9
13455.0
1288.8
509.3
1042.4
854.4
669.1
3952.0
15458.1
Road Runner
11.5
7845.0
5640.6
2696.7
5661.0
8895.1
14220.5
2500.0
18512.5
Seaquest
68.4
42054.7
286.9
384.5
301.2
583.1
208.0
1020.5
Up N Down
533.4
11693.2
3350.3
2847.6
2955.2
3180.8
28138.5
2896.9
16095.7
Normed Mcan
0.000
1,000
0.443
0.264
0.381
0.357
0.704
0.562
1.904
Normed Median
0.000
1.000
0.144
0.204
0.175
0.268
0,415
0.227
1.160
Table 2: Scores achieved by EfficientZero (mean & standard deviation for 10 seeds) and some baselines on some -i-diemnsional environments on the DMControl 100k benchmark. EfficientZero achieves statc-of-art performance and comparable results to the state-based SAC.
Task
CURL
Dreamer
MuZero
SAC-AE
Pixel SAC
State SAC
EfficientZero
Curtpole. Swingup
5824: 146
326::27
218.5 : 122
311:111
419:140
835::22
813::19
Reacher, Easy
538:1: 233
314:155
493 +: 145
274::14
145::30
7461:25
9521134
Ball cup. Catch
769-
246 174
542 4: 270
82
312:
746199
942-117
5.3 Ablations
In Section 4, we discuss three issues that prevent MuZero from achieving high performance when data is limited: (I) the lack of environment model supervision, (2) the state aliasing issue, and (3) the off-policy target value issue. We propose three corresponding approaches to fix those issues and demonstrate the usefulness of the combination of those approaches on a wide range of 26 Atari games. In this section, we will analyze each component individually.
Each Component Firstly, we do an ablation study by removing the three components from our full model one at time. As shown in Table 3, we find that removing any one of the three components will lead a performance drop compared to our full model. Furthermore, the richer learning signals are the aspect Muzero lacks most in the low-data regime as the largest performance drop is from the version without consistency supervision. As for the performance the high-data regime, We find that the temporal consistency can significantly accelerate the training. The value prefix seems to be helpful during the early learning process, but not as much in the later stage. The off-policy correction not necessary as it specifically designed under limited data.

---

Table 3: Ablations of the self-supervised consistency, end-to-end value prefix and model-based off- off-policy correction. We remove one component time and evaluate the corresponding version on the 26 Atari games. Each component matters and the consistency one is the most significant. The detailed results are attached in the Appendix.
Game
Full
consistency
w.o. value prefix off officpoliy correction
Normed Mean
1.904
0.881
1.482
1.455
Normed Median
1.160
0.552
0.836
Cument state
Nex: 1 stote
Next state
Next state
Next state
Next saate
DUC1
No car sotancy
With consitency
Figure 4: Evaluations of image reconstructions based latent states extracted from the model with or without self-supervised consistency. The predicted next states with consistency can basically be reconstructed into observations while the ones without consistency cannot.
Temporal Consistency As the version without celf-supervise consistency cannot work well in most of the games, we attempt to dig into the reason for such phenomenon. We design decoder D to reconstruct the original observations, taking the latent states as inputs. Specifically, the architecture of 'D and the H. symmetrical, which means that all the convolutional layers replaced by deconvolutional layers in D and the order of the layers are reversed in D. Therefore, H is an encoder obtain state st from observation o, and 2 tries decode the o, from st. In this ablation, we freeze all parameters of the trained EfficientZero network with or without consistency respectively and the tool visualize the current states and unrolled states, shown in different rows of Figure 4. Here we note that Mosa the trained EfficientZero model with consistency and M is the one without consistency,
reconstructed results are shown in different columns of Figure 4. We regard the decoder
As shown in Figure 4, in terms of the current state 8t, the observation is reconstructed well enough in the two versions. However, it is remarkable that the the decoder given Muun not reconstruct images from the unrolled predicted states & while the one given Meun can reconstruct the basic observations.
To sum up, there are some distributional shifts between the latent states from the representation network and the states from the dynamics function without consistency. The consistency component can reduce the shift and provide more supervision for training the dynamics network.
Value Prefix We further validate our assumptions in the end-to-end learning of value prefix, i.c. the state aliasing problem will cause difficulty in predicting the reward, and end-to-end learning of value prefix can alleviate this phenomenon.
To fairly compare directly predicting the reward versus end-to-end learning of the value prefix, need to control for the dataset that both methods are trained on. Since during the RL training, the the method, opt to load half-trained Pong model and rollout total 100k steps as the common static dataset. We split this dataset into a training set and a validation Then we run both the direct reward prediction and the value prefix method on the training split.
dataset distribution is determined

---

As shown in Figure 5, we find that the direct reward prediction method has lower losses on the
Training
Validation
1.5-
Reoard Value Prof
10-
Heaard talus Prafie
Valkeete (t3)
0.-
training set. However, the value prefix's ' validation error is much smaller when unrolled for 5 steps. This shows that the value pre- fix method avoids overfitting the hard reward prediction problem, and thus it can reduce the state aliasing problem, reaching a bet- ter generalization performance.
Toirkng
9.6-
0.6
0.4
0.
0.2
0.2
0.0
2.5
5.0
7.5
10,0
12.5
15.0
0.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
Traniin steps 00
Taining stops 10
Figure 5: Training and validation losses of direct reward prediction method and the value prefix method.
Off-Policy Correction To prove the effectiveness of the off-policy correction component, we compare the error between the target values and the ground truth values with or without off-policy correction. Specifically, the ground truth values are estimated by Monte Carlo sampling.
We train a model for the game UpNDown with total 100k training steps, and collect the trajectories different training stages respectively (20k, 40k, .... 100k steps). Then we calculate the ground truth values with the final model. We choose the trajectories at the same stage (20k) and use the final model to evaluate the target values with or without off- off-policy correction, following the Equation We evaluate the 1.1 error of the target values and the ground truth, as shown in Table 4. The error of unrolled next 5 states means the average error of the unrolled 1-5 states with dynamics network from current states. The error is smaller in both current states and the unrolled states with off-policy correction. Thus, the correction component does reduce the bias caused by the off-policy issue.
Table Ablations of the off off-policy correction: LI error of the target values versus the ground truth values. Take UpNDown an example.
States
Current state
Unrolled next states (Avg.)
All states (Avg.)
Value error without correction Value error with correction 0,533
0.765
0.636
0.657 0.569
0.576
Furthermore, we also ablate the value error of the trajectories at distinct stages in Table 5. We can find that the value error becomes smaller the trajectories fresher. This indicates that the off-policy issue is severe due to the staleness of the data. More significantly, the off-policy correction can provide more accurate target value estimation for the trajectories distinct time-steps as all the errors with correction shown in the table are smaller than those without correction at the same stage.
Table 5: Ablations of the off-policy correction: Average LI error of the values of the trajectories at distinct stages. Take UpNDown an example.
Stages of trajectories
20k
40k
60k
80k
100k
Value error without correction
0.657
0.697
0.628
0,574
0.441
Value error with correction
0.569
0.552
0.537
0.488
0.397
6
Discussion
In this paper, we propose a sample-efficient model-based method EfficientZero. It achieves super- human performance on the Atari games with as little as 2 hours of the gameplay experience and state-of-the-as performance ablation studies to examine the effectiveness of the proposed components. This work one step towards running RI. in the physical world with complex sensory inputs. In the future, we plan to extend it to more directions, such as better design for the continuous action space. And we also plan to study the accelcration of MCTS and how to combine this framework with life-long learning.
some DMControl tasks. Apart from the full results, we do detailed
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program New Generation Artificial Intelligence" (Grant No. 2021AAA0150000).
References
[1] Bruce Abramson. The expected-outcome model of two-player games. Morgan Kaufmann, 2014.
[2] Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michel L Littman. Combating the compounding-error problem with a multi-step model. arXiv preprint arXiv: 1905. / 13320, 2019.
[3] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002.
[4] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[5] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 191iv:1912.06688/ 2019.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1,597-1677. PMLR, 2020.
[7] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011. 10566, 2020.
[8] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforce- ment learning in handful of trials using probabilistic dynamics models. arXiv preprint 180i 12114, 2018.
[9] Joery A de Vries, Ken S Voskuil, Thomas M Moerland, and Aske Plaat. Visualizing muzero models. arXiv preprint arXiv:2102.12924, 2021.
[10] Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICMI--1),, pages 465 462-472. Citeseer, 2011.
[11] Bann-Batteen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
[12] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 191:::191..01603, 2019.
[13] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2566 PMLR, 2019.
[14] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02199 2020.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9733 2020.

---

[16] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[17] Thomas Hubert, Julian Schritwieser, loannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning complex action spaces. arXiv preprint arXiv:2104.06303 2021.
[18] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model- based reinforcement learning for atari. arXiv preprint arXiv:1903.00377 2019.
[19] Kacper Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency? arXiv preprint arXiv:2003.1018: 2020.
[20] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pages 282-293. Springer, 2006.
[21] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649, 2020.
[22] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Picter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. /4990, 2020.
[23] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (SKVvv550900997/ 2015.
[24] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-555 -533, 2015.
[25] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[26] Christopher D Rosin. Multi-armed bandits with episode context, Annals of Mathematics and Artificial Intelligence, (3):0033-23, 2011.
[27] Julian Schritwweserr, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-600 609, 2020.
[28] Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. arXiv preprint arXiv:2104.06200 2021.
[29] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[30] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 1506.024833 2015.
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X7.v1707.00 2017.
[32] Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self- prefitredie representations.
12

---

[33] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre. George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mas- tering the game of with deep neural networks and tree search. nature, 529(7587):444 489, 2016.
[34] David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354-3333 2017.
[35] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. arXiv preprint arXiv:2004.04136, 2020.
[36] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[37] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint
2018.
[38] Hado Van Hasselt, Arthur Guez. and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[39] Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? arXiv preprint arXiv: 00000000524,,. 2019.
[40] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanetot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning. pages 1995-2003. PMLR, 2016.
[41] Christopher John Comish Hellaby Watkins. Learning from delayed rewards. 1989.
[42] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pincau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint 1900071910.01741, 2019.
13


Reviews:
[{"review_id": "zQ7tonpazxc", "reviewer": "Reviewer_gNMH", "summary": "This paper presents an algorithm called EfficientZero, a sample efficient model-based RL algorithm built upon MuZero. The authors demonstrate the efficiency of the proposed method via atari games, and call for the research of MCTS based RL algorithms in a wider community. The core contribution lies in the adaption of MuZero to settings where data is limited. The paper is overall interesting, easy to follow, and technically sound. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality & significance:  I have concerns regarding the novelty and applicability. From the perspective of methods, we have seen the application of model based, model free, and planning to atari games. This paper is a direct enhancement of MuZero, I would like to see how sample efficient it is on the testbeds used in Muzero, e.g., board games, with limited data settings. Since the vision of this paper claims to contribute a step towards real world RL applications such as robot manipulation, I would like to see some discussions on how the proposed method would be a step forward since atari is still a toy game mostly used in academia. At least the authors are encouraged to conduct experiments on simulation environments (e.g., continuous action space), which is similar to what \"Mastering Atari with Discrete World Models\" did.  \nAlso, MCTS based methods usually suffer from handling real time senarios, where  a significant real world application requires quick response,e.g., online gaming, auto driving, robot manipulation, etc. A discussion on this point is necessary, since the authors hope to accelerate applicability of mcts based reinforcement learning. \n\nQuality: the paper is technically sound; the experiments design for atari is ok. But I would like to see more experiments on other cases with limited data. As claimed by the authors, EfficientZero is ought to be a sample efficient model based RL algorithm for handling cases with limited data. \n\nClarity: the paper is overall easy to follow, but with some grammatical issues. The authors need to improve the wording. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would suggest the authors to provide a thorough discussion on the limitations of this work, compared to existing works. ", "review_text": "", "extra_scores": {"ethical_concerns": "no ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "eku7HLW8kf_", "reviewer": "Reviewer_4xQE", "summary": "The authors consider the problem of sample-efficient reinforcement learning focused on image-based data, particularly in the Atari domain. Building on the MuZero model, they introduce three key changes to improve the learned model and thereby the sample efficiency: (a) a loss to enforce consistency between rolled-out and direct state abstractions learned by the model; (b) an end-to-end approach for predicting the sum of discounted rewards (\"value prefix\") used to generate a MCTS Q-value estimate, and (c) the use of an additional MCTS rollout for bootstrapping the state-value estimate with replay buffer data, to limit the off-policy bias of multi-step estimates from old data. They show that the proposed changes result in a significant performance boost in the low-data (100k) Atari domain, over both the original muZero and a variety of sample-efficient baselines. The also provide an ablation study that demonstrates improvement from the combination of the three interventions over each individually. The authors also release the source code.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Quality:**\n\nThe interventions generally seem quite reasonable, well-motivated and applicable to other tasks, and the empirical results are compelling. The paper has a good related work section (including quite recent results like [9] that work on similar themes). The authors also release their source code (I assume the url will be included in the supplementary in the final submission).\n\nOne key point that was not clear to me is the computational cost of the MCTS bootstrapping in the off-policy correction. What is the difference in train time with and without this?\n\nI'm not fully convinced by the self-consistency reconstructions on page 2 in the appendix. The decoder is trained independently to reconstruct the observations from the state abstractions s_t? The decoder (particularly in the middle row) appears to be reconstructing impossible states, whereas I would assume that in the failure state it would simply reconstruct a static image of a probable game state (independent of s_t).\n\nCan you list MuZero as well in Table 2? \n\n**Originality:**\n\nThe paper proposes several improvements on top of the existing MuZero architecture. The self-supervised consistency loss appears to have been concurrently proposed (though not yet peer-reviewed) by de Vries et al., and the imaginary rollouts have been used by others albeit without the fine-grained tradeoff with real-world experience. The \"value prefix\" learning seems to be the most original idea and is compelling; although it seems to be the weakest intervention according to the ablations, the authors do show an additional train vs. validation comparison to illustrate the effectiveness. Altogether, I believe that illustrating the individual and combined benefits of the interventions and releasing the code illustrates a strong contribution to the community.\n\n**Significance:**\n\nAs mentioned, the gains in overall performance are impressive. It would be nice to see results in one other problem domain (e.g. 3D exploration); the focus on Atari in the RL community risks overfitting to a single domain, although the proposed interventions don't appear to be domain-specific.\n\n**Clarity:**\n\nThe aleatory uncertainty point (Line 177) is used to motivate the value prefix loss. I found both this and the explanation in 4.2 to be somewhat confusing, although I see the point after some reflection. I think it would help to point out that the network might equally assign responsibility for a reward to multiple states in the future, leading to an additive error in the estimate.\n\nCan you explain / expand on the role of the projector in the main text in Section 4.1?\n\nFigure 4: why do you report two different losses in train and validation?\n\nMinor text suggestions:\n\nLine 109: ...and use[s] model-based value estimate[s] to bootstrap.\n\nLine 135: Open bracket in \\tilde{V}(s_{t+k}) is superscript\n\nEquation 2: Can you add brackets around the argmax argument for clarity?\n\nLine 175: [Ne]ither the reward ...\n\nLine 182: Off-policy [nature] of multi-step value\n\nLine 221: ...adjacent populations provide [natural] two views... (natural is confusing here)\n\nLine 229: The state aliasing problem has [a] negative...\n\nFigure 2 caption: ...the right player didn't move and miss[ed] the ball.\n\nLine 296: ...since it requires [fewer] samples.\n\nLine 303: ...run[s] MCTS search...\n\nLine 304: ...into [a] CPU part and [a] GPU part...\n\nLine 307: ...to avoid performance issue[s] with Python...\n\nLine 331: ...which also first propose[d] the Atari...\n\nLine 333: ...a method that use[s] contrastive...\n\nLine 346: ...our limit[ed] data scenario...\n\nLine 360: ...can outperform [a] human player....\n\nLine 361: ...our method outperform[s] human[s] in...\n\nLine 365: ...three issues that prevent[s] MuZero (remove s)\n\nLine 373: Due to limited computation resources... (no on)\n\nLine 361: cause the [difficulty] in predicting...", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No issues.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "R6syLRJ6d3", "reviewer": "Reviewer_CuAV", "summary": "The authors propose EfficientZero, a sample efficient model-based visual RL algorithm built on MuZero. By making three changes, namely using self-supervised learning to learn the environment model, predicting the value f, and using the model to correct off-policy value targets, EfficientZero achieves state-of-the-art performance on the Atari 100k benchmark. The authors also provide an efficient implementation for future research.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach is the first one to achieve human-level (or comparable) performance with limited data. The three modifications have been studied by ablations, and every of them is shown effective respectively. Besides, the paper is well structured and easy to follow. However, I still have the following comments that the authors need to address. \n\n1. MuZero Reanalyze in [30] uses a target network to provide the n-step bootstrapped target value, while EfficientZero uses MCTS root values in off-policy correction. In addition, EfficientZero in this paper also uses dynamic n (denoted by l). The ablation tests in this paper should separate these two issues (using MCTS root values and dynamic n) for analysis. It is more interesting to see which is more significant. BTW, it is critical to show the required computation costs like Table 1 in [30]. \n2. In the experiments (Section 6.5 and Appendix), the ablation study of removing three components (the self-supervised consistency loss, the end-to-end value prefix learning, and the model-based off-policy correction) are applied to 3 games only in Section 6.5 and 5 other games in Appendix. It would be better if these can be applied to all the other games. \n3. This paper does not do ablation tests for data augmentation, which has been shown to have non-trivial improvement of the performance in SPR and DrQ. However, it is still interesting to see ablation tests for EfficientZero (without data augmentation) to see how significant it is when compared with other components. \n4. This paper states “It is worth to note that MuZero does not explicitly learn the environment model. Instead, it solely relies on the reward and value prediction to learn the model.” However, the policy signal should also be involved for learning the environment model.\n\nFollowing are related to the presentation. \nLine 33: Muzero -> MuZero\nLine 74: reference of SimSiam should be [7]\nLine 88: such as Go, Chess and Shogi [37, 37] -> [36, 37]\nLine 148: To be precise, Formula (2) is called a variant of the PUCT algorithm.\nLine 175: “Either” the reward “or” the value “provide” enough training signal -> “Neither” the reward “nor” the value “provides” enough training signal\nLine 178: the predicted “reward” still have large prediction errors -> “rewards”\nFigure 1: “next-state” component should be “dynamic” or “transition”\nLine 229: an important “roll” in MCTS -> “role”\nLine 267: “use l rewards” should be “use l-step rewards”", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"ethical_concerns": "N.A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "MUtLuRIVTx7", "reviewer": "Reviewer_deNp", "summary": "This paper presents EfficientZero - a variant of MuZero that targets data efficiency in the low-data regime. The authors identify 3 factors that hinder the data efficiency of MuZero in the low-data regime, namely lack of supervision on learning the model, compounding errors in value estimation, and lack of off-policy correction with stale data. To provide richer supervision for learning the model, EfficientZero employs an auxiliary self-supervised objective similar to SimSiam. To combat compounding errors, EfficientZero directly learns the \"value prefix\" rather than per-step rewards. Finally, EfficientZero utilizes the model to address off-policy correction. EfficientZero achieves a new state of the art on the Atari 100k benchmark, outperforming existing methods by a large margin. Ablation studies demonstrate the importance of all three proposed components - removing any of them results in significant performance drop.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Things to like:\n* 1. Overall the paper is well written. The authors successfully stated the problem of interest, their hypothesis on the potential issues with current methods, their proposed solutions, and the empirical verification. The writing is easy to follow. Although there are some minor errors and typos, they do not hinder the readability.\n* 2. EfficientZero addresses important problems in value-equivalent models and model-based RL. MuZero, and other value-equivalent models, all suffer the problem of lacking learning signals. The compounding error is also a well-known issue in the model-based RL context. Admittedly, the solutions used by EfficientZero are not completely novel and are mostly variants of existing methods. However, the empirical success of applying them to state-of-the-art agents like MuZero is still significant. I believe the community can benefit from the success of EfficientZero.\n* 3. The empirical results are very impressive. EfficientZero achieves a new state of the art on the Atari 100k benchmark and is the first to achieve super-human performance in terms of the mean human-normalized score. This is a big step forward.\n* 4. An open-source implementation of MuZero/EfficientZero can be a valuable asset to the community.\n\nThings that can be improved:\n* 1. There is some inconsistency between the identified issues and the proposed solutions. In L40-42, it says \"... three components are essential to the image input model-based RL agent: ..., ..., and an efficient use of the model to explore the world\". I understand the third component as using the model for better exploration. However, the corresponding solution is to use the model for off-policy correction. Unless I misunderstand something, this seems a bit inconsistent to me. Similar confusion was found in L256 as well. \"[predicting the value prefix] helps the MCTS to explore better\". I do not fully understand the connection between less compounding error and better exploration. \n* 2. Asadi *et al* [1] proposed a multi-step model to combat the compounding error, which also directly learns the \"value prefix\". The authors should as least mention it in the related work.\n* 3. For model-based off-policy correction, MuZero already uses the value output of MCTS as the bootstrap target (see the text associated with Eq. (1) in [3]). Section 4.3 is not clear on this. What is new here is a heuristic for choosing the unroll length $l$.\n* 4. The authors could include a bit more discussion on the relative importance of the three components in the ablation study section. Based on the results in Table 1 in the appendix, removing the consistency loss seems to cause the biggest performance drop. Does it imply that richer learning signals are what MuZero lacks the most in the low-data regime?\n\nQuestions for the authors:\n* 1. Since off-policy correction is crucial in this low-data regime, did the authors try existing methods like Retrace($\\lambda$) [2]? The current solution may be too aggressive at truncating the trajectories, which may cause slower learning.\n* 2. Could the authors comment on the impact of the three components in the high-data regime, like 200M frames? The off-policy correction may be less effective in a more online setting as the problem itself is less severe. But the other two components may still help.\n\nMinor Errors and Typos:\n* 1. L22, a reference to AlphaZero is missing.\n* 2. L33, \"Muzero\" --> \"MuZero\".\n* 3. L88, duplicated reference [37].\n* 4. L151, \"... prioritize explore promising part of the tree\" --> \"prioritize exploring...\".\n* 5. L350-351, the term \"self-play network\" pops up from nowhere.\n\nReferences:\n* [1] Asadi *et al*, 2019, *Combating the Compounding-Error Problem with a Multi-step Model*.\n* [2] Munos *et al*, 2016, *Safe and efficient off-policy reinforcement learning*.\n* [3] Schrittwieser *et al*, 2020, *Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model*.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no foreseeable negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were in universal agreement the paper proposed a novel, well justified method with good empirical results. Please include results on mujoco; it would be particularly interesting to include the ablation where MCTS is only used to generate learning targets, but not to act, in order to improve response time (you suggest this ablation in the rebuttal, and is also investigated in [1]). Also, if possible, please discuss or include some experiments in the 'data-rich' regime (high number of training frames). A method that strongly improves performance in the low-data regime, but catastrophically degrades performance as the amount of experience scales up, is of more limited interest than one which is data efficient while retaining good performance in data-rich regime. 

[1] On the role of planning in model-based deep RL, Hamrick et. al

Decision: Accept (Poster)