On the role of planning in model-based deep reinforcement learning
Authors: Jessica B Hamrick
Abram L. Friesen
Feryal Behbahani
Arthur Guez
Fabio Viola
Sims Witherspoon
Thomas Anthony
Lars Holger Buesing
Petar Veličković
Theophane Weber
Venue: ICLR 2021 Poster (2021)
Keywords: model-based RL
planning
MuZero

Abstract: Model-based planning is often thought to be necessary for deep, careful reasoning and generalization in artificial agents. While recent successes of model-based reinforcement learning (MBRL) with deep function approximation have strengthened this hypothesis, the resulting diversity of model-based methods has also made it difficult to track which components drive success and why. In this paper, we seek to disentangle the contributions of recent methods by focusing on three questions: (1) How does planning benefit MBRL agents? (2) Within planning, what choices drive performance? (3) To what extent does planning improve generalization? To answer these questions, we study the performance of MuZero (Schrittwieser et al., 2019), a state-of-the-art MBRL algorithm with strong connections and overlapping components with many other MBRL algorithms. We perform a number of interventions and ablations of MuZero across a wide range of environments, including control tasks, Atari, and 9x9 Go. Our results suggest the following: (1) Planning is most useful in the learning process, both for policy updates and for providing a more useful data distribution. (2) Using shallow trees with simple Monte-Carlo rollouts is as performant as more complex methods, except in the most difficult reasoning tasks. (3) Planning alone is insufficient to drive strong generalization. These results indicate where and how to utilize planning in reinforcement learning settings, and highlight a number of open questions for future MBRL research.

Published as a conference paper at ICLR 2021
ON THE ROLE OF PLANNING IN MODEL-BASED DEEP REINFORCEMENT LEARNING
Jessica B. Hamrick, Abram L. Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Witherspoon, Thomas Anthony, Lars Buesing, Petar Veličković, Théophane Weber" DeepMind, London, UK
ABSTRACT
Model-based planning is often thought to be necessary for deep, careful reason- ing and generalization in artificial agents. While recent successes of model- reinforcement learning (MBRL) with deep function approximation have strength- ened this hypothesis, the resulting diversity of model-based methods has also made it difficult to track which components drive success and why. In this paper, we seek to disentangle the contributions of recent methods by focusing on three questions: (1) How does planning benefit MBRL agents?
(2) Within planning, what choices drive performance? (3) To what extent does planning improve gen- eralization? To answer these questions, we study the performance of MuZero (58), a ofte--of-the-ar MBRL algorithm with strong connections and overlapping com- ponents with many other MBRL algorithms. We perform a number of interven- tions and ablations of MuZero across a wide range of environments, including control tasks, Atari, and 9x9 Go. Our results suggest the following: (1) Planning is most useful the learning process, both for policy updates and for providing more useful data distribution. (2) Using shallow trees with simple Monte-Carlo rollouts is as performant as more complex methods, except in the most difficult reasoning tasks. (3) Planning alone is insufficient to drive strong generalization. These results indicate where and how to utilize planning in reinforcement learning settings, and highlight a number of open questions for future MBRL research.
Model-based reinforcement learning (MBRL) has seen much interest in recent years, with advances
yielding impressive gains over model-free methods in data efficiency [12] 15 25 76). zero- and
few-shot learning [16 37 60), and strategic thinking [
58). These methods combine planning and learning in a variety of ways, with planning specifically referring to the process of using a learned or given model of the world to construct imagined future trajectories or plans.
62
63
64
Some have suggested that models will play a key role in generally intelligent artificial agents [4 50 55 (6, (5 (67. with such arguments often appealing to model-based aspects of human cognition as proof of their importance [24] 26 28 4I), While the recent successes of MBRL methods lend evidence to this hypothesis, there huge variance in the algorithmic choices made to support such advances. For example, planning can be used to select actions at evaluation time [e.g.. 12] and/or for policy learning [e.g.. 34]; models can be used within discrete search [e.g.. 58 or gradient-based planning [e.g., 25 29]: and models can be given [e.g.. 45 or learned [e.g.. [2]. Worryingly, some works even come to contradictory conclusions, such as that long rollouts can hurt performance due to compounding model errors in some settings [e.g.. 34], while performance continues to increase with search depth in others (58). Given the inconsistencies and non-overlapping choices across the literature, it can be hard to get clear picture of the full MBRL space. This in turn makes it difficult for practitioners to decide which form of MBRL is best for a given problem (if any).
The aim of this paper is to assess the strengths and weaknesses of recent advances in MBRL to help clarify the state of the field. We systematically study the role of planning and its algorithmic design choices in a recent state-of-the-art MBRL algorithm, MuZero (58) Beyond its strong performance, MuZero's use of multiple canonical MBRL components (e.g., search-based planning, a learned model, value estimation, and policy optimization) make it good candidate for building intuition about the roles of these components and other methods that use them. Moreover, as discussed in the
"Correspondence addressed to: ( {jhamrick ck, theophane) en

---

Published as a conference paper at ICLR 2021
next section, MuZero has direct connections with many other MBRL methods, including Dyna [67],
MPC [T], and policy iteration [33
To study the role of planning, we evaluate overall reward obtained by MuZero across wide range standard MBRL environments: the DeepMind Control Suite (70), Atari 8 Sokoban [3T], Minipac- man [22] and 9x9 Go 42 Across these environments, we consider three questions. (1) For what purposes is planning most useful? Our results show that planning- which can be used separately for policy improvement, generating the distribution of experience learn from, and acting at test- time-is most useful in the learning process for computing learning targets and generating data. (2) What design choices in the search procedure contribute most to the learning process? We show that deep. precise planning is often unnecessary to achieve high reward in many domains, with two-step planning exhibiting surprisingly strong performance even in Go. (3) Does planning assist in gener- alization across variations of the environment- common motivation for model-based reasoning? We find that while planning can help make up for small amounts of distribution shift given a good enough model, it is not capable of inducing strong zero-shot generalization on its own.
BACKGROUND AND RELATED WORK
Model-based reinforcement learning (MBRL) 9 26
49 74 involves both learning and plan- ning. For our purposes, learning refers to deep learning of a model, policy, and/or value function. Planning refers to using a learned or given model to construct trajectories or plans. In most MBRL. agents, learning and planning interact in complex ways, with better learning usually resulting in better planning, and better planning resulting in better learning. Here, we are interested in under- standing how differences in planning affect both the learning process and test-time behavior.
47
MBRL methods can be broadly classified into decision- time planning, which use the model to seleet actions, and background planning, which use the model to up- date a policy [68]. For example, modee-predietive control (MPC) [11 is classic decision-time planning method that uses the model to optimize sequence of actions starting from the current environment state. Decision- time planning methods often feature robustness to un- certainty and fast adaptation to new scenarios [c.g.. (76), though may be insufficient in settings which require long- term reasoning such as in sparse reward tasks or strategic games like Go. Conversely, Dyna [67 is classic back- ground planning method which uses the the model to sim- ulate data which to train policy via standard model- free methods like Q-learning or policy gradient. Back- ground planning methods often feature improved data ef- ficiency over model-free methods [e.g.. [34,, but exhibit the same drawbacks as model-free approaches such brittleness to out-of nut-of-distributionn experience at test time.
act
plan
update
1Ag
observe
Model-based approximate policy iteration. The agent updates its policy using targets computed via plan- ning and optionally acts via planning during training, at test time, or both.
Figure I:
number of works have adopted hybrid approaches combining both decision-time and background planning. For example, Guo et al. [23]. Mordatch et al. (48) distill the results of a decision-time planner into policy. Silver al. (11. Tesauro & Galperin 7T do the opposite, allowing a policy to guide the behavior of decision-time planner. Other works do both, incorporating the distillation or imitation step into the learning loop by allowing the distilled policy from the previous iteration to guide planning on the next iteration, illustrated by the "update" arrow in Figure 1 This results in form of approximate policy iteration which can be implemented both using single-step 40
or multi-step [17 [8] updates, the latter of which is also referred to expert iteration [3] dual
policy iteration 666) Such algorithms have succeeded in board games [ 63 644, discrete-action
MDPs 27 51 58 and continuous control 43
45
65
In this paper, we focus our investigation on MuZero [58] a state-of-thh-art member of the approx- imate policy iteration family. MuZero is useful testbed for analysis not just because of its strong performance, but also because it exhibits important connections to many other works in the MBRL literature. For example, MuZero implements a form of approximate policy iteration [18 and
2

---

Published as a conference paper at ICLR 2021
has close ties to MPC in that it uses MPC as subroutine for acting. There is also interesting connection to recent Dyna-style MBRL algorithms via regularized policy optimization: specifically, these methods simulate data from the model and then update the policy this data using TRPO 34 39 46 [5). Recent work by Grill et al. (20) showed that the MuZero policy update approximates TRPO, making the learning algorithm implemented by these Dyna-style algorithms quite similar to that implemented by MuZero. Finally, the use of value gradients [I0 19 (25] [29] leverages model gradients compute policy gradient estimate; this estimate is similar (without regularization) to the estimate that MCTS converges to when used as a policy iteration operator (20 75). Thus, our
results with MuZero have implications not just for its immediate family but to MBRL methods more broadly.
In contrast to other work in MBRL, which focuses primarily on data efficiency [e.g.. 3d or model learning (e.g. our primary concern in this paper is in characterizing the role of planning with respect to reward after a large but fixed number of learning steps, as well as to zero-shot generalization performance (however, we have also found our results to hold when performing the same experiments and measuring approximate regret). We note that MuZero can also be used in a more data-efficiis manner ("MuZero Reanalyze") by using the search re-compute targets on the same data multiple times [58] but leave exploration of its behavior in this regime to future work.
Our analysis joins number of other recent works that seek to better understand the landscape of MBRL methods and the implications of their design choices. For example, Chua et al. [12 perform careful analysis of methods for uncertainty quantification in learned models. Other research has investigated the effect of deep versus shallow planning [32 35,, the utility of parametric models in Dyna over replay 773, and benchmark performance of a large number of popular MBRL algorithms in continuous control tasks (74). Our work is complementary to these prior works and focuses instead on the different ways that planning may be used both during training and at evaluation.
2 PRELIMINARIES: OVERVIEW OF MUZERO
MuZero uses a learned policy and learned value, transition and reward models within Monte- Carlo tree search (MCTS) [3 38 both to select actions and to generate targets for policy learning (Fig- ure T.. We provide brief overview of MuZero here and refer readers to [opppeddix A and Schrit- twieser et al. [58] for further details. Algorithm 1 [and present pseudocode for MuZero and MCTS, respectively.
Model MuZero plans in a hidden state space using a learned model μi parameterized by 0 and comprised of three functions. At timestep i, the encoder embeds past observations into a hidden state, ot = he Given a hidden state and an action in the original action space, the (de- terministic) recurrent dynamics function predicts rewards and next states, rě. st = go h-1 at df-1), where & is the number of imagined steps into the future starting from a real observation at time t. In addition, the prior (not used in the classic Bayesian sense of the term) predicts policy and value for a given hidden state, The v² = fo())), and is used to guide the tree search.
Search Beginning at the root node st, each simulation traverses the search tree according to search policy until previously unexplored action is reached. The search policy a variation on the pUCT rule [53 38) that balances exploitation and exploration, and incorporates the prior to guide this (see Equation Section A.3 After selecting an unexplored action af at state of the tree is expanded
.+1 predicted by the =f+1 and value 0g,t f++ state s sf+1, policy Tg. by adding a new node with reward rh.
model. The value and reward are used to form a bootstrapped estimate of the cumulative discounted reward, which is backed up to the root, updating the estimated return Q and visit count N of each node on the path. After B simulations, MCTS returns a value u MCTS (the average cumulative discounted reward at the root) and policy n MCTS (a function of the normalized count of the actions taken at the root during search).
Acting After search, action is sampled from the MCTS policy, a, ~ x MCTS and is executed in the environment to obtain reward renr, Data from the search and environment are then added to a replay buffer for use in learning: {o,, a,, reny n. MCTS . MMCS)) }.
Learning The model is jointly trained to predict the reward, policy, and value for each future timestep k = 0... K. The reward target is the observed environment reward, yenv The policy target
3

---

Published as a conference paper at ICLR 2021
MCTS The value target is the n-step bootstrapped discounted is the MCTS- policy Wt+k return 2t riti + TT2-2 + ..... +
"MMMS For reward, value, and policy losses f",e", and E", respectively, the overall loss is then (()) = Ex= E' ("),,,, + +2" (2 Re+*) + (= "t*k M. Note that MuZero is not trained to predict future observations or hidden states: the learning signal for the dynamics comes solely from predicting future rewards, values, and policies. We do, however, perform additional experiments in Section D.2 in which we train MuZero to also predict observations via additional reconstruction loss; our results indicate that the observation- based model is slightly more accurate but does not qualitatively change MuZero's 's behavior.
T+n + -n
3 HYPOTHESES AND EXPERIMENTAL METHODS
Our investigation focuses on three key questions: (1) How does planning drive performance in MuZero? (2) How do different design choices in the planner affect performance? (3) To what extent does planning support generalization? To answer these questions, we manipulated number of different variables across our experiments (Figure 2| see also alsofSection B.T]: the maximum depth we search within the tree (Dtree), the maximum depth optimize the exploration-explootation tradeoff via pUCT (DuCt), the search budget (B), the model (learned or environment simulator), and the planning algorithm itself (MCTS or breadth-firts search).
(1) Overall contributions of planning Performance in many model-free RL algorithms is driven by computing useful policy improvement targets. We hypothesized that, similarly, using the search for policy improvement (as opposed to exploration or acting) is a primary driver of MuZero's 's per- formance, with the ability to compare the outcome of different actions via search enabling even finer-grained and therefore more powerful credit assignment. To test this hypothesis, we imple- mented several variants of MuZero which use search either for learning, for acting, or both. When learning, we compute π MCTS either using full tree search (Duree oo) or using only one step of lookahead (Doree = 1). When acting, we sample actions either from the policy prior Tg, or from a MCTS computed using Diree = xc. These choices allow us to define the following variants (see ta- ble in in(Fiuure 3):
which trains and uses the model in one-step way to generate learning targets, similar to having a Q-function; "Learn", which uses the search only for learning: "Data", which uses the search only to select actions during training while using the model in one-step way to generate targets; "Ltar, which uses the search both for learning and for action selection during training: and "LeaanDDtta+Eval" which corresponds to full MuZero. See [Section B.2 for more details about these variants.
(2) Planning for learning One feature of MCTS is its ability to perform "precise and sophisticated lookahead" [58]. To what extent does this lookahead support learning stronger policies? We hypothesized that more complex planning like tree search- as opposed to simpler plan- ning, like random shooting- and deeper search is most helpful for learning in games like Go and Sokoban, but less helpful for the other environments. To test this, we manipulated the tree depth (Duree) UCT depth (Ducr), and search (B) Note that our aim with varying DUCT is to evaluate the effect of simpler versus more complex planning, rather than the effect of exploration.
Search budget B = 6
Tree depth Dtree 3
UCT depth DVer = 2
UB. L
Figure 2: For nodes at depth d DUCT, we select actions according to PUCT (Section 2/. while for nodes at depth DUCT < d < Direes we select actions by sampling from Te,.. Nodes at depth d = Derce (and deeper) are not e
(3) Generalization in planning Model-based reasoning is often invoked as way to support generalization and flexible reasoning [e.g.. 26 [1]. We similarly hypothe- sized that given a good model, planning can help improve zero-shot generalization. First. we evaluated the ability of the individual model components to generalize new usage patterns. Specifically, we evaluated pre-trained agents using: larger search budgets than seen during
expanded; instead, we stop the search and backup using Ug,s. The search bud- get B is equal to the number of nodes in the tree aside from the root sQ.
account for this, we used MPO-style update ① in the search budget experiments, similar Grill al. 20 See Section B.3
'uuZer''s policy targets suffer from degeneracies at low visit counts 20
27
4

---

Published as a conference paper at ICLR 2021
training, either the learned model the environment simulator, and either MCTS breadth-fire search (see [Section B.4]. Second, we tested generalization to unseen scenarios by evaluating pre-trained Minipacman agents of varying quality (assessed by the number of unique mazes during training) on novel mazes drawn from the same or different distributions as in training.
RESULTS
We evaluated MuZero on eight tasks across five domains, selected to include popular MBRL envi- ronments with wide range of characteristics including episode length, reward sparsity, and varia- tion of initial conditions. First, we included two Atari games [8] which are commonly thought to require long-term coordinated behavior: Ms. Pacman and Hero. We additionally included Mini- pacman [ toy version of Ms. Pacman which supports procedural generation of mazes. We
also included two strategic games that are thought to heavily rely on planning: Sokoban [22 51)
and 9x9 Go 42 Finally, because much work in MBRL focuses on continuous control [e.g.. 74), we also included three tasks from the DeepMind Control Suite [70]: Acrobot (Sparse Swingup), Cheetah (Run), and Humanoid (Stand). We discretized the action space of the control tasks in Tang & Agrawal 669. Grill et al. [20]. Three of these environments also exhibit some amount of stochasticity and partial observability: the movement of ghosts in Minipacman is stochastic; Go is two-player game and thus stochastic from the point of view of each player independently; and using a limited number of observation frames in Atari makes partially observable (c.g., is not possible to predict when the ghosts in Ms. Pacman will change from edible to dangerous). Further details of all environments are available in Appendixx C.
4.1 BASELINES
1.0
Before beginning our analysis, we tuned
Noraliizd Reward
MuZero for each domain and ran baseline ex- periments with a search budget of B = 10 sim-
0.8
ulations in Minipacman, B 25 in Sokoban,
0.6
B = 150 in 9x9 Go, and B = 50 in all other environments. Additional hyperparame- ters for each environment are available in App pendix C and learning curves in [Section D.3j unless otherwise specified, all further experi- ments used the same hyperparameters as the baselines. We obtained the following median final scores, computed using the last 10% of steps during training (median across seeds): 620.07 on Acrobot, 885.57 on Cheetah, 787.98
0.4
0.2
0.0
1S
L
D
L+D L+D+E
Acrobot
Hero
Sokoban
Cheetah
Ms. PM
9x9 Go
on Humanoid, 29916.97 on Hero, 45346.71 on
Humanoid
Mini PM
Ms. Pacman, 310.1 on Minipacman, 0.97 on Sokoban (proportion solved), and 0.72 on 9x9 Go (proportion games won against Pachi 10k 77 bot with strong amateur play). These baselines are all very strong, with the Atari and Control Suite results being competitive with state-of-the-art 222 31 58).
Train Update
Train Act
Test Act
Duve 1
Te,t
Disee 00
me,1
Diree
Drres
STA,1
L+D L+D++
Deve
D
IT.,1
Divee D Derec
Figure Contributions of planning perfor- mance, where 0.0 is the performance attained by a randomly initialized policy (Table 8). and 1.0 that obtained by full MuZero Table 7). Grey bars
4.2 OVERALL CONTRIBUTIONS OF PLANNING
We first compared vanilla MuZero to differ- ent variants which use search in varying ways, as described in |Section 3 To facilitate com- parisons across environments, we normalized scores to lie between the performance attained by a randomly initialized policy (Table 8) and the full version of MuZero Table 7. Figure 3 shows the results. Across environments, the "One-Step" variant has median strength of 7% (N 80).
show medians across environments, and error bars show 95% confidence intervals of the median. IS=One-Step. L=Learn, D=Data, E=Eval. See Figure for environment error bars.
5

---

Published as a conference paper at ICLR 2021
Control
Games
Strategy
Reward
Dive
Dyee
Dues
Acroctt
Chawbsh
Humanoid
(b)
Haro
Norm.
Mr. PM
Mini PM
Soloban
DuCT
DUcT
DUCt
SkB Do
Reward
(c)
10
20
6
10
20
50
# Simmtaaions
Simulations
Simulations
Figure 4: Effect of design choices on the strength of the policy prior. All colored lines show median normalized reward across ten seeds (except Go, which uses five seeds), with error bars indicating min and max seeds. Rewards are normalized by the median scores in Table 9 All agents use search for learning and acting during training only. (a) Reward as function of Dreee Here, DUCT = Drre (b) Reward as function of DUCT. Here, Dtree = oo and the number of simulations is the same as the baseline.
and the number of simulations is the same the baseline.
(c) Reward as function of search budget during learning. Here, Ducr and Diree oo (except in Go, where Duct oc).
Although this variant is not entirely model-free, does remove much of the dependence the model, thus establishing a useful minimal-planning baseline to compare against. Using a deeper search solely compute policy updates ("Learn")) improves performance 68.5% (N 80). The "Data" variant-vv where search is only used to select actions actions-siimiarlly improves over "One-Step 66.7% (N 80). These results indicate both the utility in training multi-step model, and that search may also drive performance by enabling the agent to learn from different state distribu- tion resulting from better actions (echoing other recent work leveraging planning for exploration 45 66)). Allowing the agent to both learn and act via search during training ("Learn+Data")) fur- ther improves performance to median strength of 90.3% (N = 80). Finally, search at evaluation ("Learn+Data+Eval") brings the total up to 100%.
Using the search for learning, for acting during training, and for acting at test time appears to pro- vide complementary benefits (see also Table 12). This confirms hypothesis that important benefit of search is for policy learning, but also highlights that search can have positive impacts in other ways, too. Indeed, while some environments benefit more from model-based learning signal (Cheetah, Sokoban, Go) others benefit more from model-based data (Hero, Minipacman, Hu- manoid, Ms. Pacman, Acrobot). Combining these two uses enables MuZero perform well across most environments (though Hero and Minipacman exhibit their best performance with "Data"; this effect is examined further in Section 4.3). Search evaluation time only provides small boost in most environments, though it occasionally proves to be crucial, as in Acrobot and Go. Overall, using search during training ("Lean+Data") is the primary driver of MuZero's performance over the "One-Step" baseline, leveraging both better policy targets and an improved data distribution.
discussed Section MuZero has important connections to number of other approaches in MBRL; consequently, our results here suggest implications for some of these other methods. For example, the "Learn" variant of MuZero similar to Dyna-style methods that perform policy up- dates with TRPO (34 39 46 52). Given the improved performance of "Learn+Data" over "Learn", these methods may similarly benefit from also using planning to select actions, and not just for learning. The "Data" variant of MuZero is similar in spirit to Dyna-2 [61]; results suggest that works which implement this approach [5]6] may similarly benefit from incorporating planning into policy and value updates. "Learn+Data++ shows the contribution of MPC "Learn+Date

---

Published as a conference paper at ICLR 2021
MCTS w/ Model
Control
Games
Strategy
Cheetah
Norm. Reward
Sokoban
MCTS w/ Simulator
Reward
(b)
BFS w/ Simulator
Norm. Reward
25
125
625
25
N Simulations
Simulations
Simulations
Figure 5: Effect of search at evaluation as function of the number of simulations, normalized by the median scores in Table 1О All colored lines show medians across seeds, with error bars indicating min and max seeds. (a) MCTS with the learned model. (b) MCTS with the environment simulator. (c) Breadth-firs search (BFS) with the environment simulator. Results with the learned model are similar and can seen in Figure
and combined with our results in Section 4.4 highlights the importance of guiding MPC with robust policy priors and/or value functions. Of course, major difference between MuZero and all these other methods is that MuZero uses a value equivalent model which is not grounded in the original observation space [21]. We therefore reran some of our experiments using model that trained to reconstruct observations in order to measure any potential implications of this difference, but did not find that such models change the overall pattern of results (see [Section D.2 and Figure 13
4.3 PLANNING FOR LEARNING
Tree depth Figure 4a shows the result of varying tree depth D E 21, 5, oc) while keeping DUCT Duee and the search budget constant. Scores are normalized by the "Learn+Data" agent from Section 4.2] Strikingly, Dtree does not make much a difference in most environments. Even in Sokoban and Go, we can recover reasonable performance using Dueee 2, suggesting that deep tree search may not necessary for learning a strong policy prior, even in the most difficult reasoning domains. Looking at individual effects within each domain, find that deep trees have a negative impact in Minipacman and an overall positive impact Ms. Pacman, Acrobot, Sokoban, and Go (see TTbll 13). While we did not detect quantitative effect in the other environments, qualitatively it appears as though very deep trees may cause worse performance in Hero.
Exploration vs. exploitation depth Figure 4p shows the strength of the policy prior as a result of manipulating the pUCT depth DUCT E 21, 6, oo} while keeping Deree oc and the search budget constant. Note that Duct and performing pure Monte-Carke sampling thereafter. We find DUct to have no effect in any environment except 9x9 Go (Talle 14); Anthony et al. 4 also found larger values of DUCT to be important for Hex. Thus, exploration-exploitatioo deep within the search tree does not seem to matter all except in the most challenging settings.
corresponds to only exploring with pUCT at the root node
Search budget FFiuure 4F shows the strength of the policy prior after training with different numbers of simulations, with Dtree = Do and DUCT to exploring different actions at the root node and then performing Monte-Carlo rollouts thereafter. We opted for these values as they correspond to a simpler form of planning, and our previous exper- iments showed that larger settings of DUcT made little difference. We find an overall strong effect of the number of simulations on performance in all environments (ablle I5). However, despite the overall positive effect of the search budget, too many simulations have detrimental effect in many
(except in Go, where DVCT = oo), corresponding

---

Published as a conference paper at ICLR 2021
Model
Simulator
Train
Test
Out-ct Disrributonn
Trai Scones
100
0
62s 3125
# Simulations
Simulations
Figure 6: Generalization to out-of-distribution mazes in Minipacman. All points are medians across seeds (normalized by the median scores in Table 10). with error bars showing min and max seeds. Colors indicate agents trained on different numbers of unique mazes. The dotted lines indicate the baseline. The maps on the right give examples of the types of mazes seen during train and test. In-distribution generalization is shown in Figure 12 (Appendix) as the behavior is similar.
environments, replicating work showing that some amount of planning can be beneficial, but too much can harm performance [e)...B4]. Additionally, the results with Ms. Pacman suggest that two simulations provide enough signal to learn well in some settings. It is possible that with further tuning, other environments might also learn effectively with smaller search budgets.
4.4 GENERALIZATION IN PLANNING
Model generalization to new search budgets Figure 5 shows the results of evaluating the baseline agents (Section 4.1) using up to 625 simulations. before, we find small but significant improve- ment in performance of 7.4 percentage points between full MuZero and agents which do not use search at all (t = p < 0.001, N = 70). Both Acrobot and Sokoban exhibit slightly better performance with more simulations, and although we did not perform experiments here with Go, Schrittwieser et al. [58] did and found a positive impact. However, Minipacman exhibits worse per- formance, and other environments show no overall effect (Table 16). The median reward obtained across environments at 625 simulations is also less than the baseline by a median of 4.7 percentage effect of compounding model errors.
points (L 5.71,, < 0.001, N 70), possibly indicating This suggests that for identical training and testing environment additional search may not always be the best use of computation; we speculate that it might be more worthwhile simply to perform additional Dyna-like training on already-observe data [see "MuZero Reanalyze", 58).
Policy and value generalization with a better model Planning with the simulator yields somewhat better results than planning with the learned model (Figure 5p). with all environments except Hero exhibiting positive rank correlations with the number of simulations (Table 17). Ms. Pacman, in particular, more than doubles in performance after 625 simulations However, across environments, 25, 125, and 625 simulations only increased performance over the baseline by a median of about percentage points. Thus, planning with better model may only produce small gains.
Model generalization to new planners We find dramatic differences between MCTS and BFS, with BFS exhibiting catastrophic drop in performance with any amount of search. This is true both when using the learned model (Figure II Appendix) and the simulator (Figure 5b), in which case the only learned component that is relied on is the value function. Consider the example of Sokoban, where the branching factor is five; therefore, five steps of BFS search means that actions at the root node are expanded, and the action will be selected performance of this agent substantially worse than just relying on the policy prior. This suggests mismatch between the value function and the policy prior, where low-probability (off-policy) actions are more likely to have high value errors, thus causing problems when expanded by BFS. Moreover, this problem not specific BFS: in sparse reward environments, any planner (such as random shooting) that relies on the value function without the policy prior will suffer from this effect. This result highlights that complex agent architectures involving multiple learned components, compounding error in the transition model is not the only source of error to be concerned about.
the with the highest value. Yet, the
However, using the simulator leaks information about the unobserved environment state, such as when the ghosts will stop being edible. Thus, these gains may overestimate what achievable by realistic model.
8

---

Published as a conference paper at ICLR 2021
Generalizing to new mazes We trained Minipacman agents on 5, 10, 100 unique mazes and then tested them on new mazes drawn either from the same distribution or a different distribution. results and Figure 12 the in-distribution results. Using the learned model, we see very slight gains in performance up to 25 simulations on both in-distribution and cut-of-distribution mazes, with sharp drop-off in performance after that reflecting compound- ing model errors in longer trajectories. The simulator allows for somewhat better performance, with greater improvements for small numbers of train mazes (t - -10.43, p < 0.001, .V 360, see also Table 18 and 19). indicating the ability of search to help with some amount of distribution shift when using an accurate model. However, as can be seen in the figure, this performance plateaus at a much lower value than what would be obtained by training the agent directly on the task. Moreover, reward using the simulator begins to decrease at 3125 simulations compared to at 125 simulations using the learned model (7 = -..5. , < 0.001, N1 = 20, N₂ 20), again indicating sensitivity to errors in the value function and policy prior. In fact, this is the same effect as can be seen in Figure 5p: the only reason the drop-off in performance is less drastic than with BFS is because the policy prior used by MCTS keeps the search more on-policy.
Figure 6 shows the uut-of-distributien
5 DISCUSSION
In this work, we explored the role of planning in MuZero (58) through a number of ablations and modifications. We sought to answer three questions: (1) In what ways does planning contribute to fi- nal performance? (2) What design choices within the planner contribute to stronger policy learning? (3) How well does planning support zero-shot generalization? In most environments, we find that (1) search is most useful in constructing targets for policy learning and for generating more infor- mative data distribution; (2) simpler and shallower planning is often as performant as more complex planning; and (3) search evaluation time only slightly improves zero-shot generalization, and even then only if the model is highly accurate. Although all our experiments were performed with MuZero, these results have implications for other MBRL algorithms due to the many connections between MuZero and other approaches (Section 1).
A major takeaway from this work is that while search is useful for learning, simple and shallow forms of planning may be sufficient. This has important implications in terms of computational efficiency: the algorithm with DUCT
can be implemented without trees and is thus far easier to parallelize than MCTS, and the algorithm with Dtree 1 can be implemented via model-free techniques [e.g.. [. suggesting that MBRL may not necessary at all for strong final performance in some domains. Moreover, given that search seems to provide minimal improvements at evalua- tion in many standard RL environments, it may be computationally prudent to avoid using search altogether at test time.
The result that deep or complex planning is not always needed suggests that many popular environ- ments used in MBRL may not be fully testing the ability of model- agents (or RL agents in general) to perform sophisticated reasoning. This may be true even for environments which seem intuitively require reasoning, such Sokoban. Indeed, out of all environments, only Acrobot and 9x9 Go strongly benefited from search at evaluation time. We therefore emphasize that for work which aims build flexible and generalizable model-based agents, important to evaluate on a diverse range of settings that stress different types of reasoning.
Our generalization experiments pose further puzzle for research on model-based reasoning. Even given a model with good generalization (e.g., the simulator), search in challenging environments is ineffective without a strong value function or policy to guide it. Indeed, our experiments demonstrate that if the value function and policy themselves do not generalize, then generalization to new settings will also suffer. But, if the value function and policy do generalize, then it unclear whether a model is even needed for generalization. We suggest that identifying good inductive biases for policies which capture something about the world dynamics [c.g.. 6 22]. as well as learning appropriate abstractions (30], may be more important than learning better models in driving generalization.
Overall, this work provides a new perspective on the contributions of search and planning in in- tegrated MBRL agents like MuZero. We note that our analysis has been limited to single-task and (mostly) obslyblbee,, deterministic environments, and see similar studies focusing on multi-task and more strongly oartiall and stochastic environments as important areas for future work.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We are grateful to Ivo Danihelka, Michal Valko, Jean-bastien Grill, Eszter Vértes, Matt Overlan, To- bias Pfaff, David Silver, Nate Kushman, Yuval Tassa, Greg Farquhar, Loic Matthey, Andre Saraiva, Florent Altché, and many others for helpful comments and feedback on this project.
REFERENCES
[1] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. In International Conference on Learning Representations (ICLR), 2018.
[2] Alekh Agarwal, Nan Jiang, and Sham M Kakade. Reinforcement learning: Theory and algo- rithms. Technical report, Technical Report, Department of Computer Science, University of Washington, 2019.
[3] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in Neural Information Processing Systems, pp. 3360-5370, 2017.
[4] Thomas Anthony, Robert Nishihara, Philipp Moritz, Tim Salimans, and John Schulman. Policy gradient search: Online planning and expert iteration without search trees. arXiv preprint arXiv:1904.088: 033-400000000.....0 2019.
[5] Kamyar Azizzadenesheli, Brandon Yang, Weitang Liu, Zachary C Lipton, and Animashree Anandkumar. Surprising negative results for generative adversarial tree search. arXiv preprint 1806::88000078800 2018.
[6] Victor Bapst, Alvaro Sanchez- Sanchez-Gonzaler Gonzalez, Carl Doersch, Kimberly L Stachenfeld. Pushmeet Kohli, Peter W Battaglia, and Jessica B Hamrick. Structured agents for physical construction. In International conference on machine learning (ICML), 2019.
[7] Petr Baudiš and Jean-loup Gailly. Pachi: State of the art open source go program. In Advances in computer games, pp. 24-38. Springer, 2011.
[8] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-2799 2013.
[9] Cameron B. Browne, Edward Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowl- ing, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of Monte Carlo tree search methods. IEEE Transactions Computational Intelligence and Al in Games, 4(1):1-43, 2012.
[10] Arunkumar Byravan, Jost Tobias Springenberg, Abbas Abdolmaleki, Roland Hafner, Michael Neunert, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Imagined value gradients: Model- policy optimization with tranferable latent dynamics models. In Con- ference on Robot Learning, pp. 566-589, 2020.
[11] Eduardo F Camacho and Carlos Bordons Alba. Model predictive control. Springer Science & Business Media, 2013.
[12] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754-4765, 2018.
[13] Rémi Coulom. Elficient selectivity and backup operators Monte-Carlo search. In Inter- national conference on computers and games, pp. 72-83. Springer, 2006.
[14] Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The helmholtz machine. Neural computation, 7(5):889-904, 1995.
[15] Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In International Conference machine learning (ICMI.,, pp. 465-472 2011
10

---

Published as a conference paper at ICLR 2021
[16] Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-basee deep reinforcement learning for Asion-based robotic control. arXiv 2018.
preprint 1912.
[17] Yonathan Efroni, Gal Dalal, Bruno Scherrer, and Shie Mannor. Multiple-step greedy policies in approximate and online reinforcement learning. In Advances in Neural Information Processing Systems, pp. 5238-52382 5247, 2018.
[18] Yonathan Efroni, Mohammad Ghavamzadeh, and Shie Mannor. Multi-step greedy and approx- imate real time dynamic programming. arXiv preprint arXiv: 0tti::9999.4233330 2019.
[19] Michael Fairbank. arXiv:0803.3539, 2008.
Reinforcement learning by value gradients.
arXiv preprint
[20] Jean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, loannis Antonoglou, and Rémi Munos. Monte-Carlo tree search regularized policy optimization. In International conference on machine learning (ICML), 2020.
[21] Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence principle for model-based reinforcement learning. Advances Neural Information Processing Systems, 33, 2020.
[22] Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sébastien Racanière, Théophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, and Timothy Lillicrap. An investigation of model-free planning. In International conference on machine learning (ICML), 2019.
[23] X. Guo, S. Singh, H. Lee, R. L. Lewis, and X. Wang. Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Rree Search Planning. In Advances in Neural Information Processing Systems, pp. 3338-3346, 2014.
[24] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, 2018.
[25] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Represen- tations (ICLR), 2020.
[26] Jessica B Hamrick. Analogues of mental simulation and imagination in deep learning. Current Opinion in Behavioral Sciences, 29:8 16, 2019.
[27] Jessica Hamrick, Victor Bapst, Alvaro Sanchez- Gonzalzz, Tobias Pfaff, Theophane Weber, Lars Buesing, and Peter W. Battaglia. Combining Q-learning and search with amortized value estimates. In International Conference on Learning Representations (ICLR), 2020.
[28] Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspired artificial intelligence. Neuron, 95(2):245-258, 2017.
[29] Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944 2952, 2015.
[30] Mark K. Ho, David Abel, Thomas L. Griffiths, and Michael L. Littman. The value of abstrac- tion. Current Opinion in Behavioral Sciences, 29:1 11-116, October 2019.
[31] Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al. Aeme: A research framework for distributed reinforcement learning. arXiv preprint 2xXv920006.0099, 2020.
[32] Zacharias Holland, Erin Talvitie, and Michael Bowling. The effect of planning shape on dyna-style planning in high-dimensional state spaces. arXiv preprint arXiv: ww0..000.6.05.. 2018.
[33] Ronald A Howard. Dynamic programming and markov processes. 1960.
11

---

Published as a conference paper at ICLR 2021
[34] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, pp. 12519-12530, 2019.
[35] Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon model accuracy. In Proceedings of the 2015 International Conference Autonomous Agents and Multiagent Systems, pp. 1181-1189, 2015.
[36] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Kon- rad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mo- hiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In International Conference on Learning Representations (ICLR), 2020.
[37] Ken Kansky, Tom Silver, David A Mély, Mohamed Eldawy, Miguel I. Lázzro-Grediilla, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, Scott Phoenix, and Dileep George. Schema networks: Zero-shot transfer with generative causal model of intuitive physics. In International confer- ence on machine learning (ICML), 2017.
[38] Levente Koesis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In European conference on machine learning, pp. 282-293. Springer, 2006.
[39] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations (ICLR), 2018.
[40] Michail Lagoudakis and Ronald Parr. Reinforcement learning as classification: Leveraging modern classifiers. In International Conference on Machine Learning (ICML), pp- 424-431, 2003.
[41] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[42] Mare Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upad- hyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafici. Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan- Davis. OpenSpiel: A framework for reinforcement learning in games. CoRR, tass/1908.094. 2019.
[43] Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071- 1 1079, 2014.
[44] Timothy P Lillicrap. Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
[45] Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model- based control. In Inter- national Conference on Learning Representations (ICLR), 2019.
[46] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations (ICLR), 2019.
[47] Thomas M Moerland, Joost Broekens, and Catholijn M Jonker. Model-basee reinforcement learning: A survey. arXiv preprint 20Xi 167 12, 2020.
[48] Igor Mordatch, Kendall Lowrey, Galen Andrew, Zoran Popovic, and Emanuel V Todorov. Interactive control of diverse complex characters with neural networks. In Advances Neural Information Processing Systems, pp. 3132-3140, 2015.
12

---

Published as a conference paper at ICLR 2021
[49] Rémi Munos. From bandits Monte-Carke tree search: The optimistic principle applied to optimization and planning. Foundations and Trends in Machine Learning, 7(1):1-130, 2014.
[50] Duy Nguyen-Tuong and Jan Peters. Model learning for robot control: survey. Cognitive processing, 12(4):319-34) 2011.
[51] Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, al. imagination-augmenntee augmented agents for deep reinforcement learning. In Advances in neural information processing systems, pp. 5690-5701, 2017.
[52] Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. In International conference on machine learning (ICML), 2020.
[53] Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence, (1(3):203-230, 2011.
[54] Bruno Scherrer. Approximate policy iteration schemes: comparison. In International Con- ference on Machine Learning, pp. 1314-1322, 2014.
[55] Jürgen Schmidhuber. Making the world differentiable: On using self-supervised fully recurrent eu al networks for dynamic reinforcement learning and planning in non-stationary environm nts. 1990.
[56) Jürgen Schmidhuber. Curious model-building control systems. In Proc. international joint conference on neural networks, pp. 1458-1463, 1991.
[57] Jürgen Schmidhuber. On learning to think: Algorithmic information theory for novel combina- tions of reinforcement learning controllers and recurrent neural world models. arXiv preprint arXiv:1511.09244 2015.
[58] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitr, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. arXiv preprint arXiv:1911.0822 2019.
[59] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust re- gion policy optimization. In International conference machine learning (ICML), pp. 1889- 1897, 2015.
[60] Ramanan Sekar, Oleh Rybkin, Kostas Danilidiis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International conference on machine learning (ICML), 2020.
[61] David Silver, Richard S Sutton, and Martin Müller. Sample-based learning and search with permanent and transient memories. In Proceedings of the 25th international conference Machine learning, pp. 968 188-975,22 2008
[62] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanc- tot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529 (7587):484 489, 2016.
[63] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
[64] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general rein- forcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362 164191:1140-1144 2018
13

---

Published as a conference paper at ICLR 2021
[65] Jost Tobias Springenberg. Nicolas Heess, Daniel Mankowitz, Josh Merel, Arunkumar Byravan, Abbas Abdolmaleki, Jackie Kay, Jonas Degrave, Julian Schrittwieser, Yuval Tassa, et al. Local search for policy iteration in continuous control. arXiv preprint arXfv:2010.05545, 2020.
[66] Wen Sun, Geoffrey Gordon, Byron Boots, and Bagnell. Dual policy iteration. In Advances in Neural Information Processing Systems, pp. 7059-7069, 2018.
[67] Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 60-):160-1,3 1991.
[68] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[69] Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy opti- mization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[70] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 180100000, 2018.
[71] Gerald Tesauro and Gregory R Galperin. On-line policy improvement using Monte-Carlo search. In Advances Neural Information Processing Systems, pp. 1068-1074, 1997.
[72] Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprint Stt::2005.098144 2020.
[73] Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems, pp. 14322- 14333, 2019.
[74] Tingwu Wang. Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shun- shi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based rein- forcement learning. arXiv preprint arXiv: 2019.
[75] Théophane Weber, Nicolas Heess, Lars Buesing, and David Silver. Credit assignment tech- niques in stochastic computation graphs. arXiv preprint 10X1v:1001.117000 2019.
[76]
Michael C Yip and David Camarillo. Model-less feedback control continuum manipula- tors in constrained environments. IEEE Transactions on Robotics, 30(4):880-889, 2014.
14

---

Published as a conference paper at ICLR 2021
A MUZERO ALGORITHM DETAILS
A.1 MUZERO PSEUDOCODE
Pseudocode for MuZero (58 is presented in Algorithmm 1 Following initialization of the weights and the empty circular replay buffer D, a learner and number of actors execute in parallel, reading from and sending data the replay buffer. In our experiments, the actors update their copies of the parameters 0 after every 500 learner steps. Our setup and description follow that of Schrittwieser
al. [58
Algorithm MuZero [58]
1: Initialize model μl and dataset D
2:
function ACTOR
3: while true do
4:
initialize episode
5:
while episode not finished do
6:
ho(....0)) π MCTS MCTS +- MCTS(s,,
7:
Alternative: Monte-Carle rollouts, BFS, etc. Alternative: sample action from ne
MCTS
8:
9:
pony o' + execute in environment
10:
Add o, a, rey, , , MCTS y MCTS to
11:
d'
12: end while
13: end while
14:
end function
15:
function LEARNER
while True do
17:
Sample batch of trajectories B from D
compute loss (Equation 8) on B
19:
Update 0 with gradient descent on L
end while
21:
end function
A.2 MODEL DETAILS
In MuZero, the model μi trained to directly predict three quantities for each future timestep k = 1...K K. These are the policy Th = MCTS ( (01+klo1.............. Ot, at...., 41++-1). the value ........ . a....... a+++-1) and the immediate reward function ub., R E | + 774444 +... ................................
yeav re, Rs "ttk where 7 MCTS is the policy used to select actions in the environment, euv is the ob- served reward, and 7 is the environment discount factor.
A.3 MCTS DETAILS
Pseudocode for MCTS is presented in Algorithm 2 MuZero uses the pUCT rule (53) for the search policy. The pUCT rule maximizes an upper confidence bound (38) to balance exploration and ex- ploitation during search. Specifically, the pUCT rule selects action
N
(cL + log
No + 02 +
a arg max a
Q(s,a)(+ ae(s,a)
1+ ((s
where N is the number of times has been selected during search, is the average cumulative discounted reward of a, and Cj C2 are constants that control the relative influence of Q and ne- Following Schrittwieser et al. (58), we set c₁ 1.25 and C2 19652 in experiments. At the root node alone, a small amount of Dirichlet noise added to the policy prior uy to encourage additional exploration at the root.
15

---

Published as a conference paper at ICLR 2021
Algorithm 2 MCTS in MuZero
1: function MCTSS state s model μ4,, number of simulations B)
2: initialize edge statistics {(N(s) a), Q(*", a))) to 0
3: for k 1... B do
4: r e' SEARCHANDEXPAND(6)))
5: BACKUP(r", s'
6: end for
7: return n MCTS and MCTS Equation 6 6and
8:
end function
9:
function SEARCHANDEXPAND(S)
10: while true do
11:
a f PUCT(s)
b choose action according to pUCT
r. 8 TRANSITION(S.
cached values if possible
13:
if (s. a)) = then
add node s' child of on edge initialize edge statistics {(N (s', a'), a'))) to compute Tg. vo + fo(s') 17: return r's''.
15:
end
19:
x- s'
end while
21:
end function
22: function BACKUP(r', ,')
23: for each edge on the path from to the root s do
24: update statistics N, Q with Equation 3 Dand
25: end for
end function
Within the search tree, each node has an associated hidden state s. For each action from 8 there an edge (s, a) on which the number of visits N a) and the current value estimate Q(s, a) stored. When a new node with state is created in the expansion step, the statistics for each edge are initialized as (NV a) 0, (s, a) 0}. The estimated policy prior (s a), reward rø, and state transition are also stored after being computed expansion since they are deterministic and can be cached. Thus, the model only needs to be evaluated once per simulation when the new leaf is added.
When backing up after expansion, the statistics on all of the edges from the leaf to the root are updated. Specifically, for k I...0, a bootstrapped k-step cumulative discounted reward estimate
2-1- G²
k+I+T + t
(2)
r-0
computed. The statistics for each edge (st,at) for 0...I in the simulation path updated as
Q(st a+) N(st,at Q(st, a²) + G*
(3)
N(sk,at) +1
Nit a²) N(s a +1.
(4)
To keep Q estimates bounded within [0, 1]. the Q estimates are first normalized as Q E [0, 1] before passing them the pUCT rule. The normalized estimates are computed
assh. (4 = Q( a²k - Q
(5)
Q Qmin
where Qmin (iinss..)TTre Q(s, a) and Qmax = max Q(s, are the minimum and maximum values observed in the search far.
16

---

Published as a conference paper at ICLR 2021
After all simulations complete, the policy x MCTS returned by MCTS is the visit count distribu- tion at the root parameterized by temperature
N(s a)
MCTS MCTS(a) E. ((s
(6)
During training, the temperature is set as a function of the number of learner update steps. Specifi- cally, the temperature is set and then decayed by a factor of 0.95 after every 5000 steps.
The value e MCTS returned by MCTS is the average discounted return over all simulations N(s a)
MCTS Σ
N (()".a).
A.4 TRAINING DETAILS
The MCTS policy is used to select an action dg MCTS which is then executed in the environment and a reward re observed. The model jointly trained to match targets constructed from the observed rewards and the MCTS policy and value for each future timestep k. The policy targets are simply the MCTS policies, while the value targets are the n-step bootstrapped discounted returns MCTS, For reward, value, and policy losses &", f" and c++1 + 1+2 + ................................ en + " P", respectivelly, the overall loss is then
(+n
((() er +
( 7ith
MCTT)) + c|
where c||0| 2 is an L2 regularization term. For the rewards, values, and policies, cross-entropy loss used for each of 8", 8", and 8
B FURTHER IMPLEMENTATION DETAILS
B.1 DEPTH-LIMITED MCTS
The depth-limited MCTS algorithm implemented by replacing the regular MCTS search and ex- pand subroutines with modified subroutine follows. The backup subroutine remains unchanged.
Algorithm Search and expand subroutine for depth-limited MCTS.
1:
function SEARCHANDEXPAND(CON state s", max UCT depth DUCT, max tree depth Diree)
2: =
3: while ls < Duree and sh is not a leaf do
b Search for leaf node
4:
<- SELECTACTION s k. DUCT)
++1 +- TRANSITION (
5:
6:
+- +
7:
end while
8:
if Dtree then
Expand unless maximum depth is reached
9:
- SELECTACTION Ducr)
10: 1++1 +- TRANSIIIONN(S""
11: Add s+1 tree
12: end
return k+1
14:
end function
15: function SELECTACTIONS kt DucT)
b The search policy
16: if k DUCT then
17:
- PU)T(s)
Choose action according to pUCT
18: else
20: end TT(-ss)
19:
> Sample from prior
21: return
22:
end function
17

---

Published as a conference paper at ICLR 2021
We initially tried varying Deree and Duct together Ms. Pacman and Minipacman. However, as we did not see any effect, we varied these variables separately for the remainder of our experiments in order to limit computation.
We did not run any experiments with DUcT = O, which corresponds to pure Monte-Carlo search. This is because this variant introduces confound: with DUcr = 0, the visit counts are longer informative and unsuitable for use as policy learning targets. To test DUCT = 0 would therefore also require modifying the learning target. Future work could test this by using the same MPO update described in the next section.
Additionally, we note that depth-limited MCTS will converge the BFS solution (Section B.4 in the limit of infinite simulations. For finite simulations, septh-limited MCTS interpolates between the policy prior and the BFS policy.
B.2 MUZERO VARIANTS
As described in Section 3, we implemented the following variants of MuZero:
One-Step.
This variant uses Dtree = 1 for learning, and acts by sampling from the policy prior. We also only train the model to predict one time step into the future (in all other variants, we train close as possible to a model-free version of MuZero, and could in principle be implemented solely with a Q-function.
predict five time steps into the future). This variant is therefore as
Learn. This variant uses Duree = oo for learning, and acts by sampling from the policy prior. It therefore gets the benefits of a deep tree search for learning, but not for acting.
for learning, acts during training by sampling from n MCTS (computed using Deree = x)), and acts at test time from the policy prior. This allows us to measure the impact of deep search on the distribution of data experienced during learning.
Data. This variant uses Dtree
This variant using Deree = do both for learning and for acting during train- ing; actions at test time are sampled from the policy prior. This variant thus benefits from search in two ways during learning, but uses no search at test time.
Learn+Data.
Learn-Data++Eaal This variant is equivalent to the original version of MuZero, using search (with Deree= = oo) for learning and for all action selection.
Another way to test the impact of search on learning and the data distribution would be to add secondary model-free loss function to MuZero, making it casier to fully ablate the model-based loss function in the One-Step and Data variants. While we leave this exploration to future work, mention that Hamrick et al. (27) investigated another member of the approximate policy iteration family which relies solely on Q-functions and which combines model-free loss with model-based loss. Thus, they were able to test the effect of purely model-free variant, as well as a "Data" variant, both of which underperformed the full version of their model.
B.3 REGULARIZED POLICY UPDATES (MPO) WITH MCTS
Past work has demonstrated that MuZero's 's policy targets suffer from degeneracies at low visit counts 27 220. To account for this, we modified the policy targets use an MPO-style update [ rather MPO
ε T exp (q MCTS/T/) /r). Here, q MCTS are the Q-values at the root node of the search tree; = 0.1 is temperature parameter; and we use MPO in place of MMCT in Equation Note that the Q-values for unvisited actions are set to zero; while it is in general poor estimate for the true Q-function, we found this choice to outperform setting the Q-function for unvisited actions to the value function. This is likely because unvisited actions are unlikely under the prior and perhaps ought not be reinforced unless good estimates are obtained through exploration. However, this choice leads to a biased MPO update; how to unbias it will be a topic of further research. Similarly, we chose the MPO update for its ease of implementation, but it is likely other forms of regularized policy gradient (e.g. TRPO [59], or more generally natural or mirror policy optimization [72 () would result in quantitively similar findings We also did not tune 5 for different environments; is likely that properly tuning it could further improve performance of the agent at small search budgets.
than the visit count distribution, similar to [20]: π i+
18

---

Published as a conference paper at ICLR 2021
B.4 BREADTH-FIRST SEARCH
In our generalization experiments, we replaced the MCTS search with a breadth-first search algo- rithm. BFS explores all children particular depth of the tree in an arbitrary order before pro- gressing deeper in the tree. Our implementation of BFS does not use the policy prior. Additionally, when performing backups, we compute the maximum over all values seen rather than averaging- Final actions are selected based on the highest Q-value after search, rather than highest visit count. Thus, this implementation of BFS is (I) maximally exploratory and (2) relies mostly on the value estimates vØ,s (especially when the simulator is used instead of the learned model).
C ENVIRONMENT AND ARCHITECTURE DETAILS
We evaluate on the following environments.
Minipacman: a toy version of Ms. Pacman with ghosts that move stochastically. We modified the version introduced by [22] such that the maze is procedurally generated on each episode. See Sectionn C.T for details.
Hero (Atari): sparse reward, visually complex video game. The goal is to navigate character through mine, clearing cave-ins, destroying enemies, and rescuing trapped miners.
Ms. Pacman (Atari): fast-paced, visually complex video game, The goal is to control Pacman to eat all the "food" in maze, while avoiding being eaten by ghosts.
Acrobot Sparse Swingup (Control Suite): diw-dimensional, yet challenging control task with sparse rewards. The task is to balance upright an under-actuated double pendulum.
Cheetah Run (Control Suite): six tix-dimensional control task, where the goal is to control the joints of a "cheetah" character to make run forward 2D plane.
Humanoid Stand (Control Suite): 21-dimensional control task, where the goal to control the joints of humanoid character to make it stand up.
Sokoban: difficult puzzle game that involves pushing boxes onto targets and in which incorrect moves can be unrecoverable [51].
9x9 Go: an casier version Go than the full 19x19 game, provided by [42], Evaluation is reported against Pachi 77, with 10 evaluations per (strong amateur play).
We now provide specific details on the network architectures and hyperparameters used for each environment. Unless specified and for layers where it is appropriate, all layers in the networks use padding 'SAME' and stride 1, and convolutions are 2-D with 3 kernels. All networks consist of an encoder hag a reppree dynamics function go, and prior function fo-
For most environments, the encoder hg is resnet composed of a number of segments. Each segment consists of convolution, a layer norm, number of residual blocks, and a ReLU, Each residual block contains a layer norm, ReLU, a convolution, a layer norm, a ReLU, and a final convolution. The input of the residual block is then added to the output of its final convolution.
For most environments, the dynamics network has the same structure as the encoder, with a different number of segments and residual blocks.
Finally, the prior function predicts three quantities and is composed of three separate networks: a policy head, value head, and reward head. The policy, value, and reward heads each consist of 1 x convolution followed by number of linear layers, with a ReLU between each pair of layers.
The hyperparameters in Table are shared across all environments except Go.
C.1 MINIPACMAN
Minipacman is toy version of Ms. Pacman. Unlike the Atari game, Minipacman also exhibits small amount of stochasticity in that the ghosts move using an epsilon-gred policy towards the agent.
19

---

Published as a conference paper at ICLR 2021
Table I Shared hyperparameters
Hyperparameters
Value
Dirichlet alpha
0.3
Exploration fraction
0.25
Exploration temperature
1.0
Temperature decay schedule
5 x 10
Temperature decay rate
0.95
Replay capacity
x 10
Min replay
10
Sequence length
C.1.1 MAZES
For Minipacman, altered the environment to support the use of both procedurally generated mazes ("in-ditrribution" mazes) and the standard maze couut-f-distributionn mazes), both of size 15x19. Figure 6 shows example mazes of both types. In all experiments except generalization, trained agents on an unlimited number the "in-distribution" mazes, and tested on other mazes also drawn from this set. For the generalization experiments, trained agents fixed set of either 5, 10, or 100 "in-distribution" mazes. Then, at evaluation time, we either tested on mazes drawn from the full in-distribution
from the out-of-distribttion set.
To generate the procedural mazes, we first used Prim's 'ii's algorithm to generate corridors. To make the maze more navigable, then randomly removed walls with probability of 0.3. The number of initial ghosts was sampled 9o ~ + Poisson(1) and this number increased by 94 ~ 0.25 + (0, 1) ghosts per level, such that the total number of ghosts at level was gi Lgo + 1)ga)- The number of pills was always set to 4. The default Minipacman maze was hand crafted to be similar the Ms. Pacman maze. In the default maze, there always one initial ghost (go- = 1) and
this number increases by every two levels (9a 0.5). We similarly set the number of pills to 4. In all mazes, the initial locations of the ghosts, pills, and Pacman is randomly chosen at the beginning of every episode.
C.1.2 NETWORK ARCHITECTURE
For Minipacman, the encoder has 2 segments, each with 64 channels and 2 residual blocks. The dynamics function has segment with residual blocks, all with 64 channels. The policy head has convolution with channels and one linear layer with channel per action (in Minipacman, this 5). The value head has a convolution with 32 channels and two linear layers with 64 and 601 channels. The reward network has the same structure as the value head.
All Minipacman experiments were run using 400 CPU-based actors and 1 NVIDIA V100 for the leamner.
Table 2: Hyperparameters for Minipacman
Hyperparameters
Value
Learning rate
10-3
Discount factor
0.97
Batch size
512
n-step return length
10
Replay samples per insert ratio
0.25
Learner steps
2 x 10a
Policy loss weight
1.
Value loss weight
0.3
Num simulations
10
Max steps per episode
600
20

---

Published as a conference paper at ICLR 2021
C.2
ATARI
PORER
3055
2710
Figure 7: (Left) Hero, (Right) Ms.mac
The Atari Learning Environment is a challenging benchmark of 57 classic Atari 2600 games played from pixel observations. We evaluate Ms. Pacman and Hero. Each observation consists of the 4 previous frames and action repeats is 4. Note that, as MuZero does not incorporate recurrence, this limited number of frames makes as when the ghosts turn from being edible to inedible).
aspects of the the environment partially observable (such
For Atari, the encoder consists of 4 segments with (64, 128, 128, 128) channels, each with 2 residual blocks, followed by segment with residual blocks with 128 channels. The dynamics network has I segment with residual blocks with 128 channels. The heads are the same as in Minipacman.
All Atari experiments were run using 1024 CPU-based actors and NVIDIA V100s for the learner.
Table 3: Hyperparameters for Atari
Hyperparameters
Value
Learning rate
10-3
Discount factor
0.995
Batch size
2048
n--tep return length
10
Replay samples per insert ratio
0.25
Learner steps
1.5 x 105
Policy loss weight
1.
Value loss weight
0.3
Num simulations
50
C.3 CONTROL SUITE
Figure 8: Control Suite environment: Left Acrobot Middle Cheetah Right Humanoid.
The DeepMind Control Suite 770 is widely used benchmark for control tasks in MuJoCo. We select high-dimensional environments Cheetah (Run), Acrobot (Swing-up Sparse) and Humanoid (Stand). We use the raw state observation input and follow the same procedure (20 69 to discretize the continuous action space. Each action dimension discretized into bins evenly spaced between [1
For Control Suite environments, we use a model based on that of 44 The encoder consists of linear layer with 300 channels, layer norm, tanh, linear layer with 200 channels, and an
21

---

Published as a conference paper at ICLR 2021
exponential linear unit (ELU). The dynamics function is simply linear layer with 200 channels followed by an ELU. The weights in the encoder and dynamics function are initialized uniformly. The policy head is factored policy head composed of linear layer with # dimensions x # bins channels. The factored policy head independently chooses an action for each dimension. The value and reward heads both composed of linear layer with 64 channels followed by ReLU and then linear layer with 2001 channels.
All Control Suite experiments were run using 1024 CPU-based actors and 2 second-generation (v2) Tensor Processing Units (TPUs) for the learner.
Table 4: Hyperparameters for control suite.
Hyperparameters
Cheetah (Run)
Acrobot (Swing-up Sparse)
Humanoid (Stand)
Learning rate
x 10-1
2.5 10
2.5 x 10-4
Discount factor
0.995
0.995
0.995
Batch size
1024
1024
1024
n-step return length
50
50
30
Replay samples per insert ratio
25.
15.
Learner steps
5 106
5 x 106
5 106
Policy loss weight
1.
1.
1.
Value loss weight
0.5
0.5
0.5
Num simulations
50
50
50
C.4
SOKOBAN
Figure 9: Sokoban environment.
Sokoban 51 is classic puzzle problem, where the agent's task is to push number of boxes onto target locations. In this environment many moves are irreversible as the boxes can only be pushed forward and hence the puzzle can become unsolvable if wrong moves made.
For Sokoban, the encoder is the same as that used for Minipacman with an additional 256-channel 1 1 convolutional layer the end. Instead of resnet, for the dynamics network for Sokoban we used a DRC(3, 1) convolutional LSTM [22]. For the heads, we use the same network as in Minipacman except that the convolutional layers in the policy, value, and reward heads have 32, 32, and 16 channels, respectively.
All Sokoban experiments were using 2048 CPU-based actors and 4 NVIDIA V100s for the leamnrr
C.5
9X9 Go
9x9 Go is a smaller version of the full 19x19 game. While the board is fully observed, the actions of the other player cannot be fully predicted, thus making the game stochastic from each players' perspective.
For the Go experiments, used different implementation of the MuZero algorithm due to easier interfacing with the Go environment. The main difference between the implementation used for other environments and the one for Go is the data pipeline. In the first implementation, actors and
22

---

Published as a conference paper at ICLR 2021
Table Hyperparameters for Sokoban
Hyperparameters
Value
Learning rate
10-3 3
Discount factor
0.99
Batch size
2048
n-step return length
10
Replay samples per insert ratio
0.4
Learner steps
3 x 10
Policy loss weight
1.
Value loss weight
0.3
Num simulations
25
learner communicate asynchronously through a replay buffer. In the one used for Go, there is no re- play buffer; instead, actors add their data to queue which the learner then consumes. Additionally, while the implementation of MuZero used in the other experiments trained the value function using MCTS (see Section 2). the one used here
n-step returns, 2 yenv 7T+22 + +........ n- env + y ottn
uses lambda returns, 2, = (1 A) E A Zt:tnn
The two player aspect of the game is handled entirely by having single player playing both moves, but using a discount of 1. The input to the agent last two states of the Go board and one color plane, where each state is encoded relative to color of the player with 3 planes, own stones, opponent's 's stones and empty stones.
The encoder consists of single convolution layer with 128 channels, followed by 6 residual 3 convolutional blocks with 128 channels (each block consists of two convolutional layers with a skip connection). The network is size-preserving so hidden states are of size 9x9. The transition model takes as input the last hidden state and the action encoded as a one-hot plane, and also consists in 6 residual convolutional blocks with 128 channels. The representation model consists in one 1 convolutional layer with 2 channels, followed by an MLP with single layer of 256 units.
Table 6: Hyperparameters for Go
Hyperparameters
Value
Learning rate
4 x 10 10-4
Discount factor
-1
Batch size
16
A
0.99
Learner steps
10a
Policy loss weight
Value loss weight
0.25
Num simulations
150
Dirichlet alpha
0.25
Exploration fraction
0.4
D ADDITIONAL RESULTS AND ANALYSIS
D.1 EXTENSIONS FIGURES IN MAIN PAPER
(contributions of search to performance) but split into separate groups and with error bars shown. Figure 11 shows the same information as Figure (effect of search at evaluation as function of the number of simulations) but using breadth-fing -irrst search with a learned model. Figure 12 shows the same information as Figure 6 (effect of search generalization to mazes) but for the in-distribution mazes instead of the out-o--distribution
Figure shows the same information Figure 3
23

---

Published as a conference paper at ICLR 2021
mazes. Figure 22 presents learning curves for Go for different values of Ducr and numbers of simulations.
Contributions of Search
Control
Games
Strategy
1.2
1.2
1.2
Acrobol
Norallzed Rewarrd
1.0
1.0
1.0
Cheetah
Humanold
0.8
Hero
0.6
PM
Mini PM
0.4-
0.4
Sokoban
0.2
0.2
0.-
Sxi Go
0.0
0.0
0.0
15
L+D
L+D+E
L-D
L+D+E
15
L-D
L+D-E
Figure 10: Contributions of the use planning to performance. breakdown containing the same
information as Figure 3 with error bars showing the maximum and minimum seeds.
BFS w/ Model Games
Acrobot
Norm. Reward
Control
Strategy
Chestah
Humenoid
Hero
PM
Simulations
Simuletions
25
# Simuletions
Soloban
Figure 11: Effect of search at evaluation as a function of the number of simulations for breadth- search (BFS) with the learned model. All colored lines show medians across seeds, with error bars indicating min and max seeds.
Model
Simulator
In-Ditriautto
625 3125
25 125 625
3125
Simulations
Simulations
Figure 12:
Effect of search on generalization to new in-distribution mazes in Minipacman. All points are medians across seeds, with error bars showing min and max seeds. Colors indicate agents trained on different numbers of unique mazes. The dotted lines indicate equivalent performance to the baseline. The maps on the right give examples of the types of mazes seen during train and test.
D.2 MUZERO WITH OBSERVATION DESERVATION-RECCOSSTTCCCCC RECONSTRUCTION
LOSS
As discussed above, planning in MuZero is performed entirely in hidden space. This hidden space has no direct semantic association with the environment state as its sole purpose is to en- able prediction of future rewards, policies, and values. This is in contrast, however, much other work on masel-based reinforcement learning, for which the model explicitly predicts environment states or observations. To investigate the effect of tying the model the environment, added an enservvation-rcoosttteete
loss to our MuZero implementation and re-ran a subset the experi- ments from Section 4.2 using this loss.
24

---

Published as a conference paper at ICLR 2021
D.2.1 OVERALL CONTRIBUTIONS OF PLANNING WITH RECONSTRUCTION LOSS
For two pixel-based environments (Minipacman and Sokoban), we added a binary cross-entropy loss between the true pixel observations and predicted future observations. Similarly, for the three state- based environments (Acrobot, Cheetah, and Humanoid), added an L2 loss between the true state observations and the predicted future states. We evaluated the effect of the reconstruction losses on these environments, for each of the "Learn," "Leam+Data," and DLearn+Data+Evvall MuZero variants of Section 4.2, Final performance results are shown in Figure 13 and learning curves are shown in Figure 14
Effect of Reconstruction by Search
Control
Games
Acrobet
1.00
1.00
3.00
Acrobateoos
0.75
Normalzes
Humanold
Humanoti
Mini
0.25
Mini Por
Sotoben
0.00
0.50
L-D+E-
1-D-T-
LIDEE
Figure 13: Contributions of planning to the performance of the "Learn (L)", "LearnDData (L+D)", and "Leann+Daaa+Evall (L+D+B)" variants when training with and without an observation- reconstruction loss. Variants trained with the reconstruction loss are denoted "+obs" in the legend. Variants without the reconstruction loss are the same as those shown in Figure 10
As shown by these figures, the performance of MuZero is relatively unchanged when using a recon- struction loss. The two main effects of the reconstruction loss, as shown by these figures, are (1) that learning consistently takes off faster when using the reconstruction loss (except in Humanoid) and (2) that the reconstruction loss allows the "Learn" variant of Acrobot to obtain reward. This latter point indicates that the model learned by "Learn" without the reconstruction loss is impoverished in way that causes compounding errors with forward credit assignment, discussed in van Hasselt et al. [73]. Overall, our results indicate that learning a model in observation space does not funda- mentally alter the role of planning in MuZero but it does improve the fidelity of the model, which can benefit planning. Given the connections between MuZero and MBRL in general, we postulate that this conclusion holds true for other MBRL methods well.
25

---

Published as a conference paper at ICLR 2021
Acrobot
Cheetah
600
800
600
Reward
400
Reward
400
200
200
0
0
1
2
3
0
1
2
3
Learner Steps
Learner Steps
Humanoid
Mini PM
800
300
600
Reward
Reward
200
400
100
200
0.0
0.5
1.0
Learner Steps
Learner Steps
Sokoban
1.0
0.8
Reward
L+obs
L+D
0.4
L+D+obs
L+D+E
0.2
L+D+E+obs
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Learner Steps
Figure 14: Learning curves for the "Learn (L)", "Learn+Data (L+D)", and "Leamn+Data+Fval (L+D+E)" variants when training with and without an observation-recoostrutiioo loss. Variants trained with the reconstruction loss are denoted "+obs" in the legend. The variants without the reconstruction loss are the same as those shown in Section D.3
D.2.2 ODSERVATION-
LOSS IMPLEMENTATION DETAILS
To implement the reconstruction loss, we added reconstruction function o% = pe (s²) to the model He that takes in the hidden state 9, and generates predicted observation ot: for the kth imagined step after timestep f. The predicted observation is then fed into additional term e ( oc44) in
26

---

Published as a conference paper at ICLR 2021
the MuZero loss (Equation 8). The parameters and architectures lightly tuned subset of these environments.
For the pixel-based environments (i.e., Minipacman and Sokoban), Pe is simple 3-layer convolu- tional network with 3 3 kernels, 128 channels per layer, and ReLU activations fceding into linear layer with 3 channels and sigmoid activation. The loss J is the binary cross-entropy loss.
Similarly, for the state- state-based environments (i.e., Acrobot, Cheetah, and Humanoid), po a simple feed- forwadd network with 3 linear layers of size [200, 100, |O]] and ELU activations, where [ is the size of the state vector. The output of this is fed into &,, which is an L2 loss.
D.3 BASELINE LEARNING CURVES
22 show the learning curves for each environment for each exper- iment in the main body. In all plots, shaded regions indicate minimum and maximum seeds, while lines show the median across seeds.
Figures 15
16
18
19
20
21
Acrobot
Acrcbct
600
500
Rewor
400
200
200
D+D+E
L+D
50
Acrobot
Acrcoct
300
300
200
Duci
10
100
100
2
Steps
Leamer Sleps
1e6
Figure 15: Learning curves for Acrobot.
27

---

Published as a conference paper at ICLR 2021
Cheetan
600
Dyse
L+D
200
0
Leamer Steps
Learner Stops
1es
Cheetan
500
600
DUeT
400
200
Learner Stops
Steps
Figure 16: Learning curves for Cheetah.
Humanoid
Humanoid
L+D L+D+E
200
1e6
Humanoid
800
Duci
Reward
400
400
10
200
200
Steps
Leamtr Sleps
1e6
Figure 17: Learning curves for Humanoid.
28

---

Published as a conference paper at ICLR 2021
Hero
30.00
Dyse
20000
30000
L+D
20000
10000
0.000
0.25
0.50
0.75
1.00
1.25
1,50
0.25
0.50
1.25
1.50
Loarner Staps
Learner Steps
165
Hero
40000
DUeT
Reward
Rewars
20000
0.00
0.28
0.50
0.75
1.00
1.25
1,50
Learner Stops
Learner Staps
Figure 18: Learning curves for Hero.
Mc. PM
Mis PM
40000
40000
00000
30000
20000
L+D
20000
L+D+E
10000
0
0.00
0.25
0.50
0.75
1.25
1,50
Steps
Stops
Ms. PM
Duci
Reward
10
50
0.75 Stepe
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Sleps
1e5
Figure 19: Learning curves for Ms. Pacman.
29

---

Published as a conference paper at ICLR 2021
Miri PM
500
200
200
Dyse
L+D
130
0.0
0.5
1.0
1.5
2.0
Leamer Steps
Learner Steps
1e5
Mini PM
300
300
200
200
DUTT
Reward
200
100
100
100
0.0
0.5
1.0
1.5
2.0
Learner Stops
Staps
Figure 20: Learning curves for Minipacman.
Sokoban
Soloban
Rauste
Reward
0.4
0.4
L+D
L+DHE
6.0
0.0
0.6
1.0
1.5
2.0
2.5
Stepe
Loamnrr Staps
Sokoban
0.8
0.6
Duci
0.4
10
0.2
50
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Steps
Learner Sleps
1e5
Figure 21: Learning curves for Sokoban.
30

---

Published as a conference paper at ICLR 2021
0.8
9x9 Go
0.8
Rawenn
0.4
Rewed
Dyse
L+D
0.1
0.0
0.2
0.4
0.6
0.
1.0
0.2
0.5
0.8
Leamer Staps
Leamer Steps
1e5
Search Budget
UCT depth
0.33
1.0
2.0
3.0
0.00
0.2
0.15
100
50.0
0.0
59199050.0
0.0
leatner papp
1ab
Figure 22: Learning curves for Go, including extra experiments on search budget and DUCT-
D.4 BASELINE VALUES
Table 7: Values obtained by the bascline vanilla MuZero agent (corresponding to the "leaan=-Data+Evva" agent in Figure 3), computed from the average the last 10% of scores seen during training. Shown are the median across seeds, as well as the worst and best seeds. Median values are used to normalize the results in Figure 3
Median
Worst Seed
Best Seed
Acrobot
620.07
590.66
632.83
Cheetah
885.57
868.04
904.64
Humanoid
787.98
712.33
863.27
Hero
29916.97
16104.65
37180.81
Ms. Pacman
45346.71
41840.45
50584.15
Minipacman
310.1
304.31
318.58
Sokoban
0.97
0.96
0.98
9x9 Go
0.72
0.7
0.77
Table 8: Values obtained by MuZero at the very start of training (i.e., with randomly initialized policy). Values are computed from the average of the first 1% of scores seen during training. Shown are the median across ten seeds, as well as the worst and best seeds. Median values are used to normalize the results in Figure
Median
Worst Seed
Best Seed
Acrobot
0.54
0.17
1.26
Cheetah
6.48
2.6
42.54
Humanoid
5.36
3.22
9,92
Hero
7.98
0.0
364.96
Ms. Pacman
247.93
169.99
315.68
Minipacman
6.8
3.71
10.4
Sokoban
0.0
0.0
0.0
9x9 Go
0.0
0.0
0.06
31

---

Published as a conference paper at ICLR 2021
Table 9: Values obtained by version of MuZero that uses search evaluation time (correspond- ing to the "Learn+Data"" agent in Figure 3). Shown are the median across ten seeds, as well as the worst and best seeds. Median values are used to normalize the results in Figure 4
Median
Worst Seed
Best Seed
Acrobot
202.66
139.01
290.64
Cheetah
839.59
591.52
896.61
Humanoid
692.05
426.25
799.85
Hero
29970.25
14059.2
37245.22
Ms. Paeman
41959.6
33780.17
49255.98
Minipacman
300.25
297.61
302.36
Sokoban
0.85
0.77
0.88
9x9 Go
0.4
0.34
0.44
Table 10: Values obtained by a baseline vanilla MuZero agent, evaluated offline from checkpoint saved at the very end of training. For each seed, values are the average over 50 (control tasks and Atari) or 1000 episodes (Minipacman and Sokoban). These values are used to normalize the results in innuieere Figure
Note that for Minipacman, the scores reported here are for agents that were both trained and tested on either the in-distribution mazes or the out-of- deddstiiibbtoonn mazes. Shown are the median across ten seeds, as well as the worst and best seeds.
Median
Worst Seed
Best Seed
Acrobot
558.18
366.84
625.72
Cheetah
896.56
806.24
905.91
Humamoid
790.8
704.95
867.8
Hero
32545.4
19350.8
37234.2
Ms. Pacman
45145.0
42776.6
54206.4
Minipacman (In-Distribution)
319.81
312.87
331.37
Minipacman (Out-of-
498.68
494.05
504.19
Sokoban
0.96
0.94
0.97
32

---

Published as a conference paper at ICLR 2021
D.5 OVERALL CONTRIBUTIONS PLANNING
Table 11: Values in Figure 3 Each column shows scores where 0 corresponds to the reward obtained by a randomly initialized agent (Table 8) and 100 corresponds to full MuZero ("Leam-Datta-Eval" Table 7)
Learn
Data
Learn+Data
Acrobot
-0.1
27.7
32.6
Cheetah
88.8
75.7
94.8
Humanoid
77.3
86.2
87.7
Hero
91.9
109.4
100.3
Ms. Pacman
46.0
57.7
92.5
Minipacman
67.6
104.9
96.8
Sokoban
69.4
0.0
88.1
9x9 Go
48.8
34.5
55.5
Median
68.5
66.7
90.3
Table 12: Effect of the different contributions of search, modeled as Reward Environmer: TrainUpdate TrainAct + TestAct over N = 400 data points, using the levels for each variable defined the table in Figure 3 This ANOVA indicates that both the environment, model-based learning. model-basee acting during training, and model-based acting during testing all significant predictors of reward. We did not detect interaction between model-baid based learn ing and model-based acting during learning.
Variable
Statistic
Strength of Evidence
Environment
F(7,388) 70.95, 0.001
TrainUpdate
F(1,388) = 175.98. < 0.001
#**
TrainAct
F(1,388 146.73.p < 0.001
TestAct
F(1,388) = 59.32, 0.001
TrainUpdate:Trainn..
381,388) 1.15,p 0.28
TrainUpdate (MB MF)
4.42, 0.001, N1 =240, N2 160
#
TrainAct (MB MF)
t= =5..1,,p < 0.001, Ni = =240. N₂ = 160
TestAct (MB MF)
t= =6..33 0.001, Ni =80,N, -320
D.6 PLANNING FOR LEARNING
33

---

Published as a conference paper at ICLR 2021
Table 13: Effect of tree depth, Pirees modeled as Reward Envi ronment log( (erre))) over
N = 375 data points. Where Duree = oC, we used the value for the maximum possible depth (i.e. the search budget). Top: this ANOVA indicates that both the environment and tree depth significant predictors of reward, and that there is an interaction between environment and tree depth. Bottom: individual Spearman rank correlations between reward and log( (rree for each environment. p-values are adjusted for multiple comparisons using the Bonferroni correction.
Variable
Statistic
Strength of Evidence
Environment
377,359) 25.23. < 0.001
8#8
log( (Dree))
F11,359)
22.95, 0.001
Environment:ogg (D Dree))
F7..359) 12.99. 0.001
#*8
Acrobot
P = 0.55, p < 0.001,
50
8#*
Cheetah
P = - 1.00, N 50
Humanoid
P = -0.01, = 1.00, = 50
Hero
P -0.27, 0.49, N 50
Ms. Pacman
P = 0.52, < 0.001, N = 50
Minipacman
P -0.88, 0.001, N 50
Sokoban
0.59, < 0.001, N 50
9x9 Go
P 0.96, 0.001, N 25
8*8
Effect of exploration vs. exploitation depth, DUCT, modeled as Reward ~ Envir Envirntment log( Ducr) over N 375 data points. Where Duct co, used the value for the maximum possible depth (i.e. the search budget). Top: this ANOVA indicates that neither the environment nor exploration vs. exploitation depth are significant predictors of reward. Bottom: individual Spearman rank correlations between reward and log DuCr) for cach environ- ment. p-values are adjusted for multiple comparisons using the Bonferroni correction. The main effects are primarily driven by Go.
Table 14:
Variable
Statistic
Strength of Evidence
Environment
F 359) 71.96, 0.001
log(Pucr)
F(1,359)
13.06, < 0.001
***
Environment:lo((Duerr Duct)
F57,359)
15,55, 0.001 #8#
Acrobot
P 0.15, p 1.00, N 50
Cheetah
-0.25, p 0.65, N 50
Humanoid
P -0.05, 1.00, N 50
Hero
-0.00, P 1.00, N 50
Ms. Pacman
P 0.26., P 0.56, N 50
Minipacman
-0.06, p = 1.00, N 50
Sokoban
P -0.02, P = 1.00, N 50
9x9 Go
0.52, 0.06, N = -25
34

---

Published as a conference paper at ICLR 2021
Table 15: Effect of the training search budget, B, the strength of the policy prior, modeled as Rewa rd ~ Envi ronment log( B) log( B)² over N 450 data points. Top: this ANOVA indicates that the environment and budget are significant predictors of reward, and that there is second-order effect of the search budget, indicating that performance goes down with too many simulations. Additionally, there is an interaction between environment and budget. Bottom: individual Spearman rank correlations between reward and logí B) for each enviromment. p-values are adjusted for multiple comparisons using the Bonferroni correction. Note that the correlation for Go does not include values for B >> 50 (and thus is largely flat, since Go does not learn for small values of B).
Variable
Statistie
Strength of Evidence
Environment
F(7,433) = 45.45. 0.001
$#8
log(B)
F(1,433
593.90,, 0.001
$88
l))(B)2
F(1,,33 = 121.70, < 0.001
Environment.log()) (B)
f(7,433)
11.64, 0.001
848
Acrobot
p= 0.76, p < 0.001, N 60
$#8
Cheetah
p 0.76, p < 0.001, N 60
Humanoid
P = 0.88,p<0...
60
$#8
Hero
P <.001, N 60
#48
Ms. Pacman
P = 1.61,p<0.000 0.001, = 60
$#8
Minipacman
P 0.70, <000p000000 N 60
#*8
Sokoban
P = 0.87 < 0.001, = 60
$*8
9x9 Go
P 0.48, p 0.05, N 30
35

---

Published as a conference paper at ICLR 2021
D.7 PLANNING FOR GENERALIZATION
Table 16: Effect the evaluation search budget, B, on generalization reward when using the learned model with MCTS, modeled as Reward ~ Environment log( over N 350 data points. Top: this ANOVA indicates that the environment and budget are significant predictors of reward, and that there is an interaction between environment and budget. Bottom: individual Spear- man rank correlations between reward and (B) for each environment. p-values are adjusted for multiple comparisons using the Bonferroni correction.
Variable
Statistic
Strength of Evidence
Environment
13.03, < 0.001
$*#
log(B)
F(1,336 9.31.p= 0.002
*
Environment:lg((B (B)
336, = 31.50, < 0.001
Acrobot
0.77, < 0.001, N = 50
Cheetah
P = 0.25, p = 0.38, = 50
Humanoid
P 0.32, 0.17, . 50
Hero
= -0.12. = 1.00, N = 50
Ms. Pacman
P -0.23, 0.73, N 50
Minipacman
- -0.39, = 0.03, N = 50
Sokoban
P 0.72, 0.001, N 50 8**
Table 17: Effect the evaluation search budget, B, on generalization reward when using the simulator with MCTS, modeled as Reward ~ Envi ronment log( [) over N = 350 data points. Top: this ANOVA indicates that the environment and budget are significant predictors of reward, and that there is interaction between environment and budget. Bottom: individual Spearman rank correlations between reward and log [) for each environment. p-values are adjusted for multiple comparisons using the Bonferroni correction.
Variable
Statistic
Strength of Evidence
Environment
F66, 336) 87.02, < 0.001
#*
log(B)
F1, 336) 271.62, < 0.001
8*8
Environment:log())
F66, 336) 89.38, < 0.001
8*8
Acrobot
== 0.82, p < 0.001, N 50
8*8
Cheetah
p 0.76, p < 0.001, N 50
Humanoid
P = 0.48, p 0.003, = 50
**
Hero
P 0.08, p 1.00, N 50
Ms. Pacman
P = 0.97, p < 0.001, N = 50
$#8
Minipacman
P 0.46, p 0.005, N = 50
Sokoban
P = 0.95, p < 0.001, = 50
#*8
Table 18: Rank correlations between the search budget, B, and generalization reward in Minipacman for different types of mazes and models. p-values are adjusted for multiple comparisons using the Bonferroni correction.
Scene Type
Model Type
Correlation
Strength of Evidence
In-distribution Out-of-distribttion
Learned model
p -0.54, 0.001, N -80
#$*
Learned model
= -0.54, < 0.001, N = -880
#$*
In-distribution
Simulator
P 0..5., 0.002, N -180
#*
dittr---istributionn
Simulator
P- -0.33, p 0.001, N = -180
36

---

Published as a conference paper at ICLR 2021
Table 19: Effect the evaluation search budget (B), the number of unique training mazes (M),, and test level on generalization reward in Minipacman when using the simulator with MCTS, modeled as Reward
Test Level over N 360 data points. This ANOVA indicates that the both the number of training mazes and the search budget are significant predictors of reward, and that there is an interaction between them.
log(M)
log(B)
Variable
Statistic
Strength of Evidence
Test Level
F(1, 11.23, p 0.001
S#$
log( M)
F(1, 355) =660.98, p 0.001
$**
log( B)
351, = -204.29, p < 0.001
$#8
log( ())
((1,355) = -108.80, p 0.001
37


Reviews:
[{"review_id": "zgzRcwaItum", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview of the paper: \nThis paper studies three empirically questions about the planning part in Muzero, an algorithm integrating direct learning, model-learning, and planning. These three questions, as written in the paper, are that 1) for what purposes is planning most useful? 2) what design choices in the search procedure contribute most to the learning process 3) does planning assist in generalization across variations of the environment. The paper answers all these three questions using experiments: 1) the major reason for the performance improvement using planning is maintaining a policy that approximates the policy found by the MCTS algorithm, 2) simpler and shallower planning is often as performant as more complex planning, 3)  search at evaluation time only slightly improves zero-shot generalization.\n\nComments:\nOverall I think the paper is not ready to publish. \n\nWhile the 3 questions asked in the paper are quite general and apply to many model-based algorithms, the paper provided general answers to them using empirical results only for the Muzero algorithm under deterministic environments, which doesn't look appropriate to me. The Muzero algorithm is different from many other planning algorithms, such as Dyna-style planning, MPC, value iteration, etc. And deterministic environments are easy cases compared with stochastic or even partial observable environments. I would suggest the authors rephrase the questions and answers to make them more specific so that the results in the paper can support conclusions well.\n\nWhile there are multiple things making me confused, I would like to highlight the following one as an example because that almost makes me think their answer to their first question is incorrect. The answer is drawn from the results shown in figure 3, which illustrates how important three design choices (following MCTS policy in both training and testing, following MCTS policy in training and prior policy in testing, and following prior policy in both training and testing) are in planning. These results could tell us how much more performance the algorithm achieves by following MCTS policy in training or testing, but they can not tell us how much more performance the algorithm achieves by updating the prior policy towards the MCTS policy compared with other approaches. That is, in all cases, the algorithm updates its prior policy towards the MCTS policy. \n\nThe other thing I feel not appropriate is, while the paper claimed that \"we systematically study the role of planning and its algorithmic design choices in a recent state-of-the-art MBRL algorithm, MuZero\", the planning part is not the only part that varies across different design choices. In particular, because the model parameters are learnable and all the variations of algorithms they tested have different updates to model parameters. The resulting learned models are different in these variations. Thus it is not appropriate to conclude that the performance difference between different variations is solely the result of planning. It might also come from differences in learned models.\n\nThanks for the author's detailed response.\n\nIn terms of the first question, I do appreciate the value of the paper as a nice empirical study of Muzero and other similar MBRL algorithms. Meanwhile, many MBRL algorithms are not like Muzero. For example, value-based planning algorithms don't maintain an explicitly parameterized policy. Therefore the conclusions here may not apply to all cases. Making its conclusions more precisely will not undermine the value of the paper, instead, it provides readers clearer results. I do see in the revised version, the authors changed their language in the discussion about the result. But maybe clearer results themselves are better.\n\nMy second concern is addressed in the updated version of the paper, with additional experiments. Cool!\n\nIn terms of the third question, after reading your response, I think there is a very interesting question. When we test planning algorithms, should we give the agent a fixed model and a fixed representation or fixed algorithms learning the model and the representation? After thinking for a while, I can see the advantages and disadvantages of both cases. So I would change my mind and agree with the authors that their choice of testing is valid. But I do hope this choice being mentioned in the paper because people like me would typically consider the other one.\n\nI would like to raise my score to 5.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QiyvsEq0QDc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper tries to disentangle the role of planning in model-based reinforcement learning with a number of different ablations and modifications to MuZero. Specifically, the authors analyze the overall contribution of planning by omitting planning from which it is originally used in MuZero, and investigate different planner settings that can drive performance. In addition, they check the generalization advantage of MBRL. Overall, the paper is well-written, and experiments are conducted appropriately. The results provide some insights that other researchers in the MBRL community can leverage for their future work. My major concern is the lack of direct ablation study that can clearly show the advantage of planning in providing a good learning signal. See the detailed comments below.\n\nComments:\n\n* To bolster the argument that planning contributes the most in providing a good learning signal, “(Data)” or “(Data+Eval)” ablation can be done:  training update is done with targets generated with MCTS of $D_{\\text{UCT}} = D_{\\text{tree}} = 1$ while the agent uses full MCTS during training rollouts. Since the current experiments do not ablate planning in training, the effect of planning is only implicitly examined.\n* In the extreme, model-free version of “(Data)” could be implemented in which $q$ is trained instead of $v$ while $v^{\\text{MCTS}}$ and $\\pi^{\\text{MCTS}}$ is replaced with $\\max q$ and $\\arg\\max q$. If this change is viable, it would also provide a chance to confirm the effect of planning for exploration only.\n* BFS is a too weak planner to compare with. How about random shooting methods?\n\nStyles & Typo:\n\n* Use “\\citet” when appropriate; when a citation is used as a subject in the sentence.\n* Line 3 of Algorithm 2 in Appendix: $k=1 \\dots S$ → $k=1 \\dots B$ ?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LRZ6cUR6HkR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "summary:\nThis paper analyzes the role of planning in the model-based reinforcement learning agent, based on evaluating MuZero on eight tasks (i.e. Ms.Pacman, Hero, Minipacman, Sokoban, 9x9Go, Acrobot, Cheetah, and Humanoid), which have discrete action spaces. The conducted experiments show three major implications: (1) Of the three parts in which search is used (i.e. search at evaluation time, search at training time for exploration, and using search result as a policy target), the role of serving as a policy improvement target was most substantial. (2) Deep tree search did not make a significant contribution to performance, and a simple Monte-Carlo rollout could be performant enough for MBRL. Also, a too small or too large search budget can be harmful to the performance of the MBRL agent. (3) Search at evaluation time was helpful for zero-shot generalization especially when the model is accurate.\n\n\npros:\n- The paper is well-written and well-organized. Hypotheses and the experiments are well-designed and seem thorough.\n\ncons:\n- The analyses are limited to MuZero that deals with discrete action space and to the deterministic environments. Since MuZero is a particular instance of MBRL, where only the reward and value prediction are performed in the latent-state space, it is unclear that the conclusions of the work can be applied to other instances of MBRL (e.g. MBRL methods dealing with dynamics model that operates on the original state space). I am not fully convinced that the results here can be generalized to other classes of MBRL.\n\n\nsome questions and comments:\n- It is interesting that the main benefit of the search is serving as a policy target. What are the advantages of policy improvement through search in MuZero compared to directly optimizing policy using analytic gradients in latent space like Dreamer (Hafner et al, 2020)?\n\n- In Figure 4c, why does the performance get worse as the number of simulations increases in Acrobot and Cheetah? Or in Figure 6 (Simulator, Green line), why the performance of (# Simulations = 3125) is worse than (# Simulations = 125)? This may not be explained as compounding model errors since the agent is using the exact simulator.\n\n- It would be have been great to see the learning curve even for the results of Section 4.3, similarly to Appendix D2 that presents the learning curve for the results of Section 4.2.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KCInErwXqKp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates how and why planning might be beneficial in model-based reinforcement learning settings. To that end, the authors ask three questions on planning in MBRL: (1) How does planning benefit MBRL agents? (2) Within planning, what choices drive performance? (3) To what extent does planning improve generalization? In order to answer these questions, the authors investigate the performance of MuZero in a variety of learning challenges while systematically ablating the algorithm to find how each part of the algorithm effects the overall performance.\n\nIn its current form the paper is marginally below the acceptance threshold.\n\nThe reasoning for my judgement is as follows:\nThe paper makes strong claims about how planning effects model-based RL algorithms based on their experimental results. However, the results are only based on five different seeds. Given the fundamental conclusions the authors formulate, the results are not statistically relevant enough. (For further intuition about the impact of a small number of seeds in DRL, please refer to ‘Deep Reinforcement Learning that Matters’, Figure 5 by Henderson 2017). The strong language in combination with the limited statistical relevance result in the experimental design not being sufficient.\n\nHaving that said, the paper itself is well written and provides important insights to the community. I would strongly encourage the authors to either adjust the language or (better) run additional experiments to strengthen the paper. With more runs for each of the experiments, I would recommend a clear accept.\n\nMinor comment: At the beginning of the second paragraph, the authors state ‘Many have suggested that models will play a key role in generally intelligent artificial agents’. The supporting papers are essentially just two different author-sets, making this another instance of too strong of a statement with too little foundation.\n\n-----------------------------------------------------------------------------------------\nAfter reading the authors response to all reviewers, I believe all questions to be sufficiently addressed. I will therefore (happily) raise my score.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the author replies, the additional experiments (more runs but also more ablations/baselines), and the updated paper. Also R2 is now largely satisfied (but seems to have been too late to post a public reply or to raise the score of the review).

The paper provides important insights in model-based RL and its connections to planning, by studying MuZero with systematic ablations. Hence a valuable contribution to the community. All (major) cons have been addressed in the revision.
Decision: Accept (Poster)