Online and Offline Reinforcement Learning by Planning with a Learned Model
Authors: Julian Schrittwieser
Thomas K Hubert
Amol Mandhane
Mohammadamin Barekatain
Ioannis Antonoglou
David Silver
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Reinforcement Learning
Deep Learning

Abstract: Learning efficiently from small amounts of data has long been the focus of model-based reinforcement learning, both for the online case when interacting with the environment, and the offline case when learning from a fixed dataset. However, to date no single unified algorithm could demonstrate state-of-the-art results for both settings.
In this work, we describe the Reanalyse algorithm, which uses model-based policy and value improvement operators to compute improved training targets for existing data points, allowing for efficient learning at data budgets varying by several orders of magnitude. We further show that Reanalyse can also be used to learn completely without environment interactions, as in the case of Offline Reinforcement Learning (Offline RL). Combining Reanalyse with the MuZero algorithm, we introduce MuZero Unplugged, a single unified algorithm for any data budget, including Offline RL. In contrast to previous work, our algorithm requires no special adaptations for the off-policy or Offline RL settings. MuZero Unplugged sets new state-of-the-art results for Atari in the standard 200 million frame online setting as well as in the RL Unplugged Offline RL benchmark.

Online and Offline Reinforcement Learning by Planning with a Learned Model
Julian Schrittwiesen DeepMind swj@google... com
Thomas Hubert'
Amol Mandhane DeepMind mandhane@google.com com
DeepMind
tdhubert@google.com
Mohammadamin Barekatain
loannis Antonoglou DeepMind University College London loannisa@google.o com
David Silver
DeepMind barekatain&google.c. com
DeepMind
University College London
davidsilver@googlle com
Abstract
Learning efficiently from small amounts of data has long been the focus of model- based reinforcement learning. both for the online case when interacting with the environment and the offline case when learning from fixed dataset. However, date no single unified algorithm has demonstrated state-of-the- results in both settings. In this work, describe the Reanalyse algorithm which uses model- based policy and value improvement operators to compute new improved training targets existing data points, allowing efficient learning for data budgets varying by several orders of magnitude. We further show that Reanalyse can also be used to learn entirely from demonstrations without any environment interactions, in the case of offline Reinforcement Learning (offline RL). Combining Reamalyse with the MuZero algorithm. we introduce MuZero Unplugged, single unified algorithm for any data budget, including offline RL. In contrast to previous work, our algorithm does not require any special adaptations for the off-policy or offline RL settings. MuZero Unplugged sets new state-of-the -ftthe-r-te-att results in the RL Unplugged offline RL benchmark as well as in the online RL. benehmark of Atari in the standard 200 million frame setting.
Introduction
Offline reinforcement learning holds the promise of learning useful policies from many existing real-world datasets in a wide range of important problems such as robotics, healthcare or education (Levine et al. 2020). Learning effectively from offline data is crucial for such tasks where interaction with the environment is costly or comes with safety concerns, but a large amount of logged and other offline data is often available.
wide varicty of effective reinforcement learning (RL) algorithms for the online case have been described, achieving impressive results in video games (Mnih al. 2015). robotic control (Akkaya et al. 2019) and many other problems. However, applying these online RL algorithms to offline data often remains challenging due to off-policy issues, with the best results in offline RL far obtained by specialised offline algorithms (Kumar et al. [2000] Wang et al.] 2020 Agarwal et al. (2020) At the same time, model-based reinforcement learning (RL) has long focused learning efficiently from little data, even going as far as learning completely within a model of the environment (Hafner et al. 2018 an approach ideally suited for offline RI.
'qqual contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

So far, these developments have been relatively independent, with no unified algorithm that could achieve state-of-tte the art results in both the online and offline settings.
In this paper, we describe the Reanalyse algorithm, a simple yet effective technique for policy and value improvement at any data budget, including the fully offline case. A preliminary version of Reanalyse was briefly introduced in the context of MuZero (Schrittwieser et al. 2020), but limited to data efficiency improvements in the discrete action case. Here, delve deeper into the algorithm and push its capabilities much further - ultimately to the point where most or all of the data is reanalysed.
Starting with the possible uses of Reanalyse, we show how it can be used for data efficient learning and offine RL, leading to MuZero Unplugged. We demonstrate its effectiveness for the online case through results on Atari and for the offline case through results on the RL Unplugged benchmark for Atari and DM Control.
2
Related Work
Recent work by Levine et al. (2020) provides a thorough review of offline RL literature and presents an excellent introduction to the subject. Much research has focused on regularising the value or policy learning to counteract off off-policy issues and learn only from high quality data. Critic-Ree Regularzee Regression (CRR) uses a critic to filter out bad actions and uses only good actions to train the policy (Wang et al., 2020). Random Ensemble Mixture (REM) regularises q-value estimation by using random convex combinations of ensemble members during training. and the ensemble mean during evaluation (Agarwal et al., 2020). Conservative Q-Learning (CQL) learns a conservative Q-function, used to lower bound the value of the current policy (Kumar et al., 2020). Pessimistic Offline Policy Optimization (POPO) also uses pessimistic value function for policy learning (He & Hou 2021).
Existing work has also demonstrated the promise of model-based RL. for offline learning (Matsushimal 2020 Argenson & Dulac-Amold 2020). but has often been restricted to tasks with low- dimensional action or state spaces, and has not been applied to visually more complex tasks such as Atari (Bellemare et al. 2013).
Model-Based Offline Reinforcement Learning (MOReL) implements a two-step procedure, first learning pessimistic MDP from offline data using Gaussian dynamics models, then a policy within this learned MDP (Kidambi et al.) 2020). Results are presented for state-based control tasks. Model- based Offline Policy Optimization (MOPO) penalises rewards by the uncertainty of the model dynamics to avoid distributional shift issues YYu et al.) 2020). Offline Reinforcement Learning from Images with Latent Space Models (LOMPO) extends MOPO to image based tasks (Rafailov et al. 2021). Results are reported on newly introduced datasets with image observations, which the authors aim to open-source in the near future. Deep Averagers with Costs MDP (DAC-MDP) (Shrestha et al.) 202T) builds non-parametric models from the offline data, solves these tabular MDPs using value iteration, then generalizes back the original MDP.
Most previous approaches primarily use the learned model for uncertainty estimation and to train a policy: they do not directly use the learned model for planning over action sequences. In contrast, our method focuses on using the learned model directly for policy and value improvement through planning both offline (when learning from data) and online (when interacting with environment). It requires no regularisation of the value or policy function either in the online or offline case, works well even in very high dimensional state spaces and is equally applicable both discrete and continuous action spaces.
Reanalyse is also qualitatively different from Dyna Suttonn 1991) in several important regards: it uses both value and policy rather than value function alone; and it also updates the state representation. In the specific case of MuZero Reanalyse it also performs tree search rather than single step lookahead used in Dyna.
A combination of MuZero Unplugged with regularisation approaches such as introduced in the previous work discussed above (Kidambi et al. 2020. Yu et al.) 2020) Rafailov et al. 2021 is possible; we leave such investigations for future work.

---

3 Reanalyse
Reanalyse takes advantage of model-based value and policy improvement operators to generate new value and policy training targets for given state (Algorithm []. In this work, we will use MuZero's 's Monte Carlo Tree Search (MCTS) planning algorithm combined with its learned model of the environment dynamics as the improvement operator
As the learned model and its predictions are updated and improved throughout training. Reanalyse can be repeatedly applied to the same state to generate better and better training targets. The improved training targets in turn are used improve the model and predictions, leading virtuous cycle of improvement.
Algorithm The Reanalyse algorithm. MuZero Unplugged instantiates representation, predict, dynamics with the MuZero network architecture; plan with MCTS; loss with the MuZero loss in eqn (1); and optimise with Adam.
for step <- O...N do
~ random(1:: :T)
s/ = representaation(n:::.)))
for < 0...k do
vi. Vi plantrepeseenaaion((uutttt
p. U predict(s}, dynamics(s},, ar4s, 0)
end for
= loss(h h (r.p. v, u, n, 1.
A optimise(1,0)
end for
To run MCTS and compute new targets for a training point, the representation function of MuZero maps the history h1 of observations, actions and rewards up to timestep t into an agent state or embedding st. The search over possible future action sequences then takes place entirely in this embedding space, by rolling the dynamics forward and applying prediction functions at every step. These predictions output the key quantities required by planning: the policy, value function and reward. The resulting MCTS statisties the root of the search tree visit counts for the actions and value estimate averaged over the tree are then used actions a are selected - instead the agent updates its model and prediction parameters based on the data it has already experienced.
training targets. During reanalysis, no
Specifically, MuZero Reanalyse jointly adjusts its parameters to repeatedly optimise the following loss at every time-step t, applied to a model that is unrolled O...K steps into the future,
((0)= = Σ 1 (
Σ (Z ) k=0
Σ I' (u)+ki)))
(1)
where vÊ, u/ and rt are respectively the policy, value and reward prediction produced by the k-step unrolled model. The respective targets for these predictions are drawn from the corresponding time-step t + k of the real trajectory: Titkk Vi+k the improved policy and value generated by the search, žt- = u++++1 +... + 7" Uu+k+n + 7t V=+4+k+ is an n-step return, and uz is the true reward.
The policy and value predictions are then updated towards the new training targets, in the same way they would be for targets computed based on environment interactions through minimising losses P. 1" and !". In other words, Reanalyse requires no changes on the part of the learner and can be implemented purely in terms of adapting the actors to generate improved targets based on stored data instead of environment interactions.
Since the actual MCTS procedure used to Reanalyse a state is the same as the one used to choose an action when interacting with an environment, it straightforward perform mix of both. We refer to this ratio between targets computed from direct interactions with the environment, and targets
Other model-based algorithms be used improvement operators well.
3

---

Reanalyse Fraction. %
99.8
98.6
99.2
98.4 96.9
93.8 87.5
75 50
Best FiL
Finel Reward
150k
Reanalyse
Median
Mcan
# Frames
100k
50.0%
1331.7%
4094.4%
2000M
95.0%
1006.4%
2856.2%
200M
50k
99.5%
126.6%
450.6%
20M
06
107
10 10P 1030
Figure 1: Reanalyse scaling in Atari. By varying the Reanalyse fraction alone, MuZero can learn efficiently data budgets differing by orders of magnitude. All other parameters held constant. Left: Final scores in Ms. Pac-Man for different Reanalyse fractions. Note the logarithmic x-axis: Linear improvements in score require exponentially more data, matching scaling laws such as described by (Kaplan et al. (020) for language models.
Right: Mean & median human normalised scores over 57 Atari games, by Reanalyse fraction.
computed by reanalysing existing data points the Reanalyse fraction. A Reanalyse fraction of 0% refers to training by only interacting with the environment, no Reanalyse stored data, whereas a fraction of 100% refers the fully offline case with environment interaction at all.
Since Reanalyse only uses stored data points and the learned model to compute improved targets, can be employed flexibly for many different purposes:
Data Efficiency. The simplest use of Reanalyse is improve data efficiency by repeatedly computing updated targets previously collected data throughout training. By scaling the Reanalyse fraction as described in Section 4 Iearning can be optimised for any data budget. For this purpose, the data be reanalysed Is sampled from the N most recent environment interactions; in the limit this includes all interactions throughout training.
Offiine RL. When increasing the Reanalyse fraction to 100%, learning takes place entirely from stored offline data as described in Section 5] without any interaction with the environ- ment. Offline data may be obtained from variety of sources, such as other agents, logged data from a heuristic control system or human examples.
Demonstrations. Reanalyse can be used to quickly bootstrap learning from demonstrations containing desirable behaviour that might otherwise be hard to discover collected for instance from humans while still interacting with the environment, learning from both sources of data at the same time. This is useful to skip past what might otherwise be hard exploration problems while still improving beyond the quality of the initial demonstrations.
Exploitation of good episodes. When using Reanalyse to improve data efficiency, Reanalyse is applied to the most recently collected data. If instead data is ordered by some other metric, such as episode reward, Reanalyse can be used to quickly learn from rare events, such as rewards observed in hard-exploration tasks. This variant is most useful in deterministic environments, as could otherwise bias the value estimates in stochastic environments.
In this paper, will focus on the data efficiency and offline RL cases. Remaining cases require no adjustments to the algorithm and only differ in the source of data to be reanalysed. Further combinations of the cases above are also possible, such as a mix of exploitation and data efficiency Reanalyse which we leave for future work.
The Reanalyse algorithm has some similarities to experience replay Lin 1992). Whereas replay performs multiple gradient descent updates for the same data point and target, Reanalyse uses model- based improvement operators to generate multiple training targets for the same data point. Reanalyse and replay have independent effects and can be combined to further improve data efficiency of learning; in fact we do for all experiments in this paper.
4 Reanalyse for Data Efficiency
By adjusting the ratio between targets computed from interactions with the environment and from stored trajectories (Reanalyse fraction), Reanalyse can be used to train MuZero at any desired data

---

Game
QR-DQN
REM
CQL(H)
MZ
Loss
Median
Mean
asterix (1%)
359.8
363.3
592.4
27220.5
BC
53.3 %
48.5 %
breakout
6.8
4.5
61.1
251.9
DQN
86.2
89.5
pong
-14.5
-20.8
19.3
-16.2
IQN
100.8 %
96.1 %
qbert
156.0
160.1
14012.0
6953.2
BCQ
107.5 %
120.0 %
seaquest
250.1
370.5
779.4
4964.0
REM
107.9 %
113.5 %
asterix (10%)
1293.9
3912.3
156.3
40554.0
CRR (ours)
155.6 %
271.2 %
breakout
61.8
56,9
269.3
485.8
b MuZero BC
54.0 %
46.9 %
pong
12.7
9.5
18.5
15.6
MuZero Unplugged
265.3
595.5
qbert
9420.5
5800.0
13855.6
16817.9
seaquest 353.1
3643.5
3674.1
8556.3
Table 1: RL Unplugged Atari benchmark.
Left: Overall results. Mean and median normalised scores over the 46 Atari games of the RL Unplugged benchmark. a) Baseline algorithms.. CRR results are for our own reimplementation, other results are from (Gulcehre al.) 2020). b) Results using the MuZero network architecture. Behaviour cloning (BC) with the MuZero network replicated the baseline BC results from a), con- firming correct import of the dataset and evaluation settings. Critic Regularized Regression (CRR) (Wang et al. 2020) significantly improved performance of the policy. MuZero Unplugged training with Reanalyse loss and MCTS for action selection led to overall best performance.
Right: Low-data setting. QR-DQN (Dabney al. 2018), REM (Agarwal et al.) 2020), CQL(H) (Kumar et al. 2020) and MuZero Unplugged results when trained on only 1% (top, million frames) or 10% (bottom, 20 million frames) of Atari data. QR-DQN and REM results from (Agarwal et al. 2020 MuZero Unplugged performance improves consistently when trained on more data.
The total amount of computation for each training run (number of updates on the learner and number of scarches the actors) is held constant.
budget, as shown in Figure I
As training progresses, the policy produced by MCTS with the latest network weights will increasingly differ from the policy originally used to generate the trajectories that are being reanalysed. This can bias the state distribution used for training as well as some of the training targets:
The policy prediction Pt for a state 8, is always updated towards the MCTS statistics n, for that same state. In this way, the policy can be learned completely independently from the trajectory; no off-policy issues can arise.
The reward prediction only depends on the state and the action that was taken from this state and not affected by off-policy issues as such. However, if the state distribution is very biased - in the extreme an action may never be observed - the reward function will be unable to learn the correct reward prediction for these cases, limiting the maximum policy improvement step.
The situation for the value function depends the choice of training target; when using an n-step TD return such as in Atari (n = 5), the target depends on the trajectory and off ooflpo issues can potentially arise. Whether this is an issue depends how different the data distribution is from the policy that is being learned. Empirically, we observed that the gain from bootstrapping with the actually observed environment rewards seems outweigh any harm from being off off-policy. We speculate that the bias introduced by early bootstrapping may be larger than the bias introduced by off off-policy targets, as also seen in prior work (Vinyals et al. 2019).
MuZero Unplugged: Offline RL with Reanalyse
We obtain MuZero Unplugged, offline version of MuZero, by adjusting the Reanalyse fraction to 100% - learning without any environment interactions, purely from stored trajectories. In contrast to previous work, we perform off-policy corrections adjustments to the value and policy learning: the exact same algorithm applies to both the online and offline case.
We used the RL Unplugged (Gulcehre et al. 2020) benchmark dataset for all offline RL experiments in this paper. To demonstrate the generality of the approach, we report results for both discrete and continuous action spaces well state and pixel based data, specifically:
5

---

Loss Unroll
supervised
CRR
Reanalyse 5
5
5
policy
60.6
61.4
54.0
155.6
203.2
value
92.2
105.0
159.2
153.0
239.9
MCTS
137.3
169.7
172.5
265.3
Table 2: Median score in RL Unplugged Atari: ablations of action selection and training loss. Median normalized scores over the 46 Atari games from RL Unplugged.
Rows of the table correspond to different action selection methods: sampling according to the policy probabilities, selecting the action with the highest value or selecting according to MCTS visit counts. Columns correspond to different number of unroll steps of the MuZero learned model and different losses. The leftmost three columns use the action from the training data as a supervised policy target, the rightmost two columns the CRR and the Reanalyse loss respectively. For the case of 0 unroll steps, an action-value head is used to predict action values, instead of the state-value predicted by the normal model. All columns use 5-step TD bootstrap towards target network the value target. For all action selection methods, Reanalyse loss led to the best performance; for all losses, MCTS action selection also led to the best performance. Overall, the combination of MCTS action selection and Reanalyse loss the MuZero Unplugged algorithm led to the best results.
DM Control Suite, different tasks, number of frames varies by task (Table B) Continuous action space with 1 to 21 dimensions, state observations.
Atari, 46 games with 200M frames each. Discrete action space, pixel observations, stochas- ticity through sticky actions (Machado et al. 2017).
MuZero Unplugged was highly effective in either setting, outperforming baseline algorithms in Atari [T well the DM Control Suite (Table We performed no tuning of hyperparameters for these experiments, instead using the same hyperparameter values as for the online RL case
(Schrittwieser al. 2020 Hubert et al. 2021).
To add another strong bascline for the Atari benchmark, we also implemented Critic Regularized Regression (CRR), recent offline RL algorithm (Wang al. 2020). For the critic value required by CRR we used the value head of MuZero model, trained by 5-step TD with respect to a target network, as in previous work (Schrittwieser et al. 2020) and the same used for MuZero Unplugged. Using CRR to train the policy head led to improved results in Atari (Table DA. CRR), matching results reported for continuous action tasks, but did not reach the same performance as MuZero Unplugged.
Performance of MuZero Unplugged was robust across the whole range of 46 Atari games the RL Unplugged benchmark, reaching the same or better performance the DQN policy used to generate the data in 44 games, and slightly worse performance in only 2 games (Figure 2). Improvements in performance with respect to the training data were considerable, exceeding 20 times increase in score in several games.
To examine the performance of MuZero Unplugged in detail and ascertain the contributions of action selection methods and training losses, we also performed a set of ablations (Tables and 7 based on the Atari dataset. We chose Atari because the large number of diverse levels enables robust performance estimates and its discrete action space allows us to cleanly disentangle the contributions of value and policy predictions as well as planning with MCTS. In contrast, for continuous action spaces such as in the DM Control suite, the contributions of policy and value are entangled, as the value function can only evaluate actions already sampled from the policy.
For our ablations, we considered three possible action selection methods: Sampling actions according to the policy network probabilities, selecting the action with the maximum value, or selecting actions based on the MCTS visit count distribution (rows of Table We also considered different losses and network architectures: the lefumost three columns use variants of the MuZero learned model with (no model at all), or 5 steps of model unroll, all trained using the supervised behaviour cloning policy target and 5-step TD value target based on target network. The next column used CRR to train the policy. The last column used the the MCTS visit count distribution from the Reanalyse loss. These ablations allow us to separately measure the contribution of MCTS at training time (rightmost

---

column) and evaluation time (bottom row), with the combination of MCTS at evaluation time and Reanalyse loss (bottom right cell) corresponding to MuZero Unplugged.
As expected, the policy prediction was insensitive to the choice of model depth, but benefited from an improved training target: the CRR loss significantly improved results. Best results were obtained when using the rich MCTS visit count distribution from the Reanalyse loss as training target (top row of Table 2
When selecting actions according to the value estimate for each action (middle row of Table 2 the depth of the learned model surprisingly important. The difference between estimating q-values (0-step model) and state-values (1-step model) was small, with both attaining results similar to the IQN baseline (Table [Ta) expected, since all of these results use distributional value prediction. However, learning a full 5-step model led to a big improvement even though only 1-step value predictions were used for evaluation. We speculate that learning full 5-step model is beneficial because regularises the network representation and acts as a useful auxiliary loss
Baselines
MuZero BC Unplugged
Task
# dims # episodes
BC
D4PG
BRAC
RABM
cartpole.swingup
1 40
386.0
856.0
869.0
798.0
143.7
343.3
finger.turn hard
2
500
238.0
714.0
227.0
433.0
308.8
405.0
fish.swim
5
200
444.0
180.0
222.0
504.0
542.8
585.4
manipulator. inseet ball
5
15000
385.0
154.0
55.6
409.0
412.7
557.0
manipulator.inser.te
5
1,000
279.0
50.4
49.5
290.0
309.9
432.7
walker.stand
6
200
386.0
930.0
829.0
689.0
444.4
759.8
walker. walk cheetah.run
6
200
380.0
549.0
786.0
651.0
496.3
901.5
6 300
408.0
308.0
539.0
304.0
592.9
798.9
humanoid.run
21
3000
382.0
1.7
9.6
303.0
408.5
633.4
mean
365.3
415.9
398.5
486.8
406.7
601.9
Table 3: Results for DM Control benchmark from RL Unplugged. Mean final score on DM Control tasks, as well as mean score across all tasks. First three columns indicate task, action dimensonality and dataset size, subsequent four columns reproduce baseline results from (Gulcehre) ct al. 2020). Final columns show performance of Behaviour Cloning (BC) with the MuZero network and results for MuZero Unplugged. As the data sets for the DM Control tasks are very small and vary a hundredfold between tasks, to keep the number of model parameters per datapoint constant and
prevent memorisation, we scaled the neural network according to channels datapoints For an layers
ablation of network size see Table
Keeping the 5-step model but changing the loss for the policy head, we observed that CRR had no effect the quality of the value prediction for action selection, while the richer MCTS visit count distribution from the Reanalyse loss led to another big improvement. Even though the policy head is not used when selecting actions according to the maximum 1-step value, we hypothesise that the auxiliary loss has a strong regularising effect and further improved the internal representation of the model. This matches the results of (Silver al. (017) that training single combined network to estimate both policy and value led to improved value prediction accuracy.
Finally, using MCTS to select actions at evaluation time (bottom of Table improved results no matter which loss was used at training time, with best results obtained when using MCTS for both training and evaluation the full MuZero Unplugged algorithm.
We also verified that our training setup correctly interpreted the offline data and reproduced the baseline performance when using the same loss: Using the actions played in the training data as
'These results suggest that an n-step model can also be used as an auxiliary loss to imprave the performance of otherwise model-free algorithms. For value- value-based algorithms without explicit policy prediction, the distribution used for action selection be used the target for the policy loss of the model.
*We spent a surprisingly large amount of time tracking down action space mismatches, data discrepancies and compression artefacts. We recommend that any offline RL paper should first reproduce baseline results for the chosen dataset before attempting modifications and improvements the algorithms.
7

---

MuZero
Task
CRR
BC
Unplugged
cartpole.swingup
664.0
501.8
594.3
fnner.turn_han
714.0
333.8
759.0
fish.swim
517.0
556.8
681.6
manipulator.insert..au
625.0
465.6
659.2
cunnipulator.insrt..po
387.0
325.9
556.0
walker.stand
797.0
473.3
887.2
walker
901.0
637.9
949.5
cheetah.run humanoid.run
577.0
765.3
869.9
586.0
416.5
643.1
mean
640.9
497.4
733.3
Table Comparison of MuZero Unplugged and CRR. Results for CRR (Wang al., 2020) were reported by selecting the checkpoint with the highest mean reward from each training run. Since this does not follow the offline policy selection guidelines from RL Unplugged and is therefore not directly comparable to the baseline results, we compared to it separately. The same highest mean reward evaluation scheme used in CRR was used for MuZero Unplugged results in this table as well. All other tables report results at the end of training.
supervised policy target to train policy head using cross-entropy loss and sampling from it for evaluation (Table T] policy BC a and b) reproduced the behaviour cloning (BC) baseline results.
Offline RL and Continuous Action Spaces
An important motivation for offline RL is the application real-world systems such as robotics, which often have continuous and high-dimensional action spaces. To investigate the applicability of MuZero Unplugged to this setting, we used the DM Control Suite dataset from the RL Unplugged dataset. DM Control is a collection of physics based benchmark tasks (Tassa et al. 2018)) with variety robotic bodies of different action and state dimensionalities (Table
In order to use planning and Reanalyse with continuous action spaces, we used the sample based search extension of MuZero introduced by (Hubert al.. 202T). This extension uses policy head to produce a set candidate actions to search over, where the MCTS considers only the sampled actions instead of fully enumerating the action space. Finally, the policy updated towards the search distribution only at the sampled actions.
When applying Reanalyse for data efficiency improvements to data generated by the agent itself, no modifications are required to use sample based search and Reanalyse together. In offline RL or when reanalysing demonstrations from source other than the agent itself, the policy that generated the actions making up the dataset is often quite different from the one learned by MuZero Unplugged, and unlikely to sample the same actions, at least at the beginning of training. Since in this case the MCTS (and by extension, Reanalyse) can only consider actions that have been sampled from the policy, it would be unlikely to learn about the actions contained in the dataset, and thus unable to sample them from the policy in the future. This effect is most pronounced in very high dimensional action spaces.
To prevent this issue, we explicitly included the action from the trajectory being reanalysed in the sample of actions searched over at the root of the MCTS tree. This serves the same purpose the Dirichlet exploration noise used in standard MuZero encouraging the MCTS to explore actions would not otherwise consider. For the prior of the injected action therefore use the same value as for the Dirichlet probability mass, 25%, though the algorithm is not sensitive to the exact value. This step is redundant for discrete action spaces (such in Atari) where the policy already always produces prior for all possible actions.
We compared the performance of MuZero Unplugged to offline RL algorithms from the literature such as D4PG (Barth-Maron et al. 2018), BRAC (Wu al. 2019) and RABM (Siegel al. 2020
Gulcehre et al. 2020 (Table 3h, well the recent Critic Regularized Regression (CRR) (Wang

---

et al. 2020) algorithm (Table 4 shown separately as CRR was evaluated by selecting the maximum performance throughout training and results are thus not comparable to the other baselines).
We first measured the performance of Behaviour Cloning (BC) when implemented using the MuZero network ensure used the offline dataset correctly and that it matches the evaluation environment. Overall performance indeed approximately matches the BC baseline (Table B).
MuZero Unplugged outperformed baseline algorithms both in individual tasks and for the mean return averaged across all tasks. It did best in difficult high-dimensional tasks such as humanoid. or the manipulator tasks, classified as "hard" by (Wang et al. 2020), compared to "easy" for the other tasks. Performance in the simplest tasks, especially cartpole, was somewhat lower - primarily due to the very small dataset leading to overfitting of the learned model and value function throughout training: in cartpole, performance of the best checkpoint (Figure() was much hetter than performance the end of training (Figure 3 Additional regularisation techniques such as dropout (Hinton et al.) 2012) could be employed to prevent this. We leave this for future work since we are primarily interested in performance on complex tasks that consider most representative of real-world problems.
7 Limitations
MuZero Unplugged uses a deterministic model, potentially limiting its performance in stochastic or partially observed environments. The learned model is single time-step model, which may limit the time horizon of planning. MuZero Unplugged also does not employ explicit forms of regularizations: combination with existing methods from the literature may improve its performance very small datasets.
The MCTS improvement operator in MuZero Unplugged requires suitable value function; in environments where value learning is very difficult this may limit the magnitude of the improvement obtained by Reanalyse.
8 Conclusions
In this paper we have investigated the Reanalyse algorithm and its applications to both data efficient online RL at any data budget and completely offline RL. We combined Reanalyse with MuZero to obtain MuZero Unplugged, unified model-based RL algorithm that achieved new state of the in both online and offline reinforcement learning. Specifically, MuZero Unplugged outperformed prior baselines in the Atari Learning Environment both using a standard online budget of 200 million frames and other data budgets spanning multiple orders of magnitude. Furthermore, MuZero Unplugged also outperformed offline baselines in the RL Unplugged benchmark for Atari and continuous control. Unlike previous approaches, MuZero Unplugged uses the same algorithm for multiple regimes without any special treatment for off-policy offline data.
This work represents a further step towards the vision of a single algorithm that can address a wide range of reinforcement learning applications, extending the capabilities of model-based planning algorithms to encompass new dimensions such as online and offline learning, using discrete and continuous action spaces, across pixel and state- state-based observation spaces, in addition to the wide array of challenging planning tasks addressed by prior work (Silver et al.] 2018).
Acknowledgements
We would like to thank Caglar Gulcchre for providing very detailed feedback and helpful suggestions to improve the paper.
All of the work in this paper was funded by DeepMind.
'Using the mean is appropriate in DM Control as the return for all tasks is in [0, 1000). with the return for the optimal policy close 1000. Therefore no outlier can dominate the mean; this unlike the situation in Atari where scores wildly varying magnitude require usage the median. 'The amount of information contained in the dataset is the product of the number of episodes and the size of the state and action space.

---

References
Agarwal, R., Schuurmans, D., and Norouzi, M. An optimistic perspective on offline reinforeement learning. In International Conference Machine Learning, pp. 104-114. PMLR, 2020.
Akkaya, I., Andrychowicz, M., Chociej. M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plap- pert, M., Powell, G., Ribas, R., et al. Solving rubik's s cube with a robot hand. arXiv preprint arXiv: 19X000//10.0111 2019.
Argenson, A. and Dulac-Amold, G. Model-based offline planning. 2020.
Ba, L.,, Kiros, R., and Hinton, G. E. Layer normalization, 2016.
Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., TB, D., Muldal, A., Heess, N., and Lillicrap. T. Distributed Distributional Deterministic Policy Gradients, 2018.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The Arcade Learning Environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A.. VanderPlas, J., Mitteemma---iilee S., and Zhang, Q. JAX: composable transformations of Python+Nummy programs, 2018. URL http::ttttt:tii
www com/google// jaax
Dabney, W., Rowland, M., Bellemare, M. G., and Munos, R. Distributional reinforcement learning with quantile regression. In AAAI, 2018.
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
Google, 2018. Cloud TPU. http::////oooudoogggeeecoo
Accessed: 2019.
Gulcehre. C., Wang, Z., Novikov, A., Paine, T. L., Colmenarejo. G., Zolna, K., Agarwal, R., Merel, J., Mankowitz, D., Paduraru, C., Dula--ulloooo Arnold, G., Li. J., Norouzi, M., Hoffman, M.. Nachum, O., Tucker, G., Heess, N., and de Freitas, N. RL Unplugged: Benchmarks for Offline Reinforcement Learning. 2020. URL https: pattts://arxxxr //wxxvvvvv...//////v//ax///x/av/ org/ pdf 2006 13888
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 18110555 2018.
He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. CoRR, abs/1603.05027, 2016. URL
He, Q. and Hou, X. POPO: Pessimistic Offline Policy Optimization, 2021.
Hennigan, T.. Cai, Norman, T., and Babuschkin, I. Haiku: Sonnet for JAX, 2020. URL
http://github com/ deepminde demdddddmmmma hm-hai
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Hinton, G. E.,, Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. Improving neural networks by preventing co-adaptation feature detectors. arXiv preprint arXiv: 2012.
Hubert, T., Schritwieser,, Antonoglou, Barekatain, M., Schmitt, and Silver, D. Learning and Planning in Complex Action Spaces. arXiv e-prints, April 2021.
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint 1:X.v:/611.05330 2016.
Kaplan, J., McCandlish, S., Henighan, T., Brown, B., Chess, B., Child, R., Gray, S., Radford, A.,
Wu, J., and Amodei, D. Scaling laws for neural language models, 2020.
10

---

Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. MOReL Modeel -lode--Based Offline Reinforcement Learning, 2020.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. CoRR, abs/1412.6980) 2015.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative Q-learning for Offine Reinforcement Learning, 2020.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020,
Lin, L.JJ. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning. 8(3-4):293 321, 1992.
Loshchilov, I. and Hutter, F. Fixing weight decay regularization in adam. CoRR, abs/ 1711005101 2017. URL http:////arx http://...wggaa/////11/ 1711105111111111
Machado, M., Bellemare, M., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. Revisiting the Arcade Learning Enviromment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61, 09 2017. doi: 10.161 3/1113//irr5699
Matsushima, T., Furuta, H., Matsuo, Y., Nachum, O., and Gu, S. Deployment-efficienl reinforcement learning via model-based offline optimization, 2020.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Ried- miller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Rafailov, R., Yu, T., Rajeswaran, A., and Finn, C. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154-1168. PMLR, 2021.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. In International Conference on Learning Representations, Puerto Rico, 2016.
Schmitt, S., Hessel, M., and Simonyan, K. Off-policy actor-critic with shared experience replay. arXiv preprint urXiv:/909) 1583, 2019.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel. T., Lillicrap, T. P., and Silver, D. Mastering Atari, Go, Chess and Shogi by Planning with Learned Model. Nature, 588(7839):604-607) 2020.
Shrestha, A. K., Lee, S., Tadepalli, P., and Fern, A. Deepaveragers: Offline reinforcement learning by solving derived par-meraei (mdp)s. s. In International Conference on Learning Representations, 2021. httplatttte://peerrreeii
Siegel, N. Y., Springenberg, J. T., Berkenkamp. F. Abdolmaleki, A., Neunert, M.. Lampe, T., Hafner, R., and Riedmiller, M. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002. 208332002.08336, 2020.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang. A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap. T., Hui, F., Sifre, L., van den Driessche, G., Gracpel, T., and Hassabis, D. Mastering the game of Go without human knowledge. Nature, 550:354-359, October 2017.
Silver, D., Hubert, T., Schritwieseer, J., Antonoglou, I., Lai, M., Guez, A., Lanetot, M., Sifre, I.., Kumaran, D., Graepel, T., et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, (0206419):114011111 2018.
Sutton, R. S. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 160) 1991.
Tassa, Y., Doron, Y., Muldal, A., Frez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancy, A., Lillicrap, T., and Riedmiller, M. Deepmind control suite, 2018.

---

Vinyals, O., Babuschkin, I., Czarnecki, W. M.. Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, pp. 1-5, 2019.
Wang, Z., Novikov, A., Zolna, K., Merel, J. S., Springenberg. J. T., Reed, S. E., Shahriari, B., Siegel, N.. Gulcehre, C., Heess, N., et al. Critic Regularized Regression. Advances in Neural Information Processing Systems, 33, 2020.
Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint ar arxiv:9911 11361, 2019.
Yu, T.. Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. MOPO; Model-based Offline Policy Optimization, 2020.
12


Reviews:
[{"review_id": "rnlE9hZPHwW", "reviewer": "Reviewer_WqPe", "summary": "In this paper the authors focus on the Reanalyze algorithm introduced in MuZero [1] and show its applicability to improving sample efficiency, as well as the offline/batch RL setting with no modification to the underlying algorithm (i.e., it can be online and offline).\n\nThey show that Dyna style re-use of previous data points provides the ability to bootstrap policy and value estimations using improved versions of themselves, thus allowing for better estimation of these quantities thanks to having a latent-space model (similar to VPN [2]) to facilitate imaginary rollouts under MCTS. They show state of the art performance in several offline RL benchmarks as a result.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* The paper is fairly well written, and whilst the core idea is relatively simple (e.g., reapply the update steps from MuZero to previously sampled trajectories), it does a good job of delving into its inner workings and investigating how it compares to some existing approaches/losses that don’t inherently need a model (e.g., BC and CRR).\n* The results seem relatively superlative, especially in the offline Atari experiments, where I believe the results are significantly better than those reported previously. Similarly, the DM Control performance appears when taken as a whole.\n* The overall approach is transparent, and the authors have been clear regarding limitations as well as their experiment methodology. For instance, regarding the performance figures shown, they make it clear when they are cherry-picking offline checkpoints for online performance in their tables, and provide full training curves in the appendix to show the degradation phenomenon that can occur with increased training. Whilst this may seem like a minor point, in my experience these details often made very opaque in offline RL (for the obvious reasons), so it is refreshing to see the opposite here. Other examples are the extensive pseudocode in the appendix, as well as baselining a BC version of their MuZero against an explicit BC model to affirm their implementation is valid.\n\nCons:\n* One concern I have is the lack of novelty in this paper. Whilst this is the first time (to my knowledge) a MuZero-style algorithm has been applied to the offline setting, most of the important algorithmic details have been covered in the past, such as MuZero itself, and the Reanalyze algorithm, which in and of itself is simply an extension of Dyna [3] (which is not cited) to their algorithmic approach..\n* Another concern is the lack of generality in the derived insights. It has been known for some time that model-based methods with planning can improve sample efficiency and perform well on the offline domain [5,6], so the key takeaway is that this also applies to MCTS+function approx.+model-based methods. My main issue is that approaches like MuZero are definitely not the predominant approach due to their high computational requirements (144 TPU Hours per single Atari game!), and I am struggling to see what can be gleaned from this paper to more generic model based approaches.\n* The baselines in the continuous control experiments seems lacking. For instance, there are other model-based methods that perform well on similar benchmarks (e.g., [5,6,7,8]) that are not compared with here; I believe the open source code is available for at least one of these. On a similar note, the results should be significance tested to demonstrate statistical power rather than just highlighting the largest number.\n\nComments/Nits:\n* L65: “they do not directly use the learned model for planning over action sequences.”; I don’t think this is strictly true, since MBOP [5] does use the model for planning.\n* L203: “value target based on a target network for the.”: unfinished sentence?\n\nConclusion:\nOverall this paper shows the importance of trajectory reuse in model-based methods by applying it to MuZero (calling it Reanalyze), and shows that the resultant method provides excellent performance both fully online and fully offline, achieving state-of-the-art performance in an offline Atari benchmark. Furthermore, the scientific approach is generally good and the paper is fairly easy to follow.\n\nHowever, I have concerns regarding the novelty of this work, as well as the applicability of the insights to a wider audience, as this work is so focussed on investigating the finer details of the MuZero algorithm. Furthermore, I believe more benchmarks should be provided for the continuous control experiments. Taken together, I therefore recommend a weak accept, but would like to see if the authors can provide more general insights, as well as more benchmarks for the DM Control section.\n\n[1] Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model, Schrittwieser et al.\n\n[2] Value Prediction Network, Oh et al.\n\n[3] Dyna, an integrated architecture for learning, planning, and reacting, Sutton\n\n[4] Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models, Chua et al.\n\n[5] Model-Based Offline Planning, Argenson and Dulac-Arnold\n\n[6] MOReL: Model-Based Offline Reinforcement Learning, Kidambi et al.\n\n[7] MOPO: Model-based Offline Policy Optimization, Yu et al.\n\n[8] COMBO: Conservative Offline Model-Based Policy Optimization, Yu et al.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No issues.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "NuBJT31STwc", "reviewer": "Reviewer_zEEy", "summary": "This work proposes the Reanalyse algorithm which uses model-based policy and value improvement operators to compute new improved training targets on existing data points, allowing efficient learning for data budgets varying by several orders of magnitude. The Reanalyse can also be used to learn entirely from demonstrations without any environment interactions, as in the\ncase of offline Reinforcement Learning (offline RL). The MuZero Unplugged algorithm provides a single unified algorithm for any data budget, including offline RL.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The Reanalyse (or Muzero unplugged) algorithm shows strong empirical performances on the Atari games and 9 different tasks in the DM Control Suite. By repeatedly computing updated targets on previously collected data throughout training, the algorithm is able to largely improve the data efficiency comparing to previous methods. It can switch between online and offline settings without extra adjustment via choosing different reanalyze fractions. \n\nGiven all those advantages, I still have some questions. First, the Reanalyse algorithm seems very similar to the one that is described in the original MuZero paper (which diminishes the contribution). I can understand that the previous version can only improve data efficiency for discrete action space. Other than that, could you comment if there are any other differences? Also, for the look-ahead loss function $l_t(\\theta)$ defined in (1), there is no $l_2$ regularity term comparing to the original MuZero paper. How could this possibly help? \n\nGiven the comprehensive empirical study provided, I would like to set my score to weak accept for the moment.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "18 Hours."}, "extra_text": {}}, {"review_id": "JARWvD84Isi", "reviewer": "Reviewer_zAF9", "summary": "The submission investigates the application of MuZero to offline RL and mixed online/offline RL using the Reanalyse algorithm introduced in MuZero. It presents experimental results for Atari with different online data proportions and for the RL Unplugged benchmark.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. While the submission does not make any novel insight or propose any novel algorithm beyond a more detailed look into the performance of the Reanalyse procedure introduced in MuZero, its empirical results are strong, despite not making any offline RL specific adaptions. Given the level of recent interest in offline RL, the having the performance of MuZero benchmarked is a valuable contribution. Overall, it is a strong submission, and I don't have any major criticisms.\n\n### Small comments\n\nLine 4 could demonstrate -> has demonstrated\n\nLine 44 “A lot of research” informal\n\nOn line 96 and 97 v should be \\nu\n\nIs the submission suggesting that the logarithmic scaling observed in Figure 1 holds in generality across environments?\n\nLine 215 use em dash\n\nLine 215 Table 2a -> Table 1a\n\nLine 271 use em dash\n\n### Reproducibility\n\nIt would be nice if the submission could additionally include the pseudocode in the appendix as a text file to make it easier for others to work from.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BaNrYWEvYGD", "reviewer": "Reviewer_7BJj", "summary": "The authors explore in depth the reanalyze method specified in muzero[2] originally, in the context of data efficiency and offline RL. The reanalyze fraction can be changed to adapt for different dataset budgets. Both discrete and continuous variants of the muzero algorithm are explored for fully offline settings using RL Unplugged dataset/benchmarks. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\nS1: Overall, the proposed method is straightforward, intuitive, and outperforms baselines on most of the challenging tasks.\n\n\nS2: The experimental results consist of both online and offline settings across discrete and continuous action spaces. \n\nS3: Addition of Limitations section and pseudocode in the supplementary materials both are very  welcome updates.\n\nS4: The ablations study (Appendix table 9) on network sizes are welcome additions. It would be wonderful if a pointer to this was added in the description of table 3. \n\nS5: The presentation quality is high. Figures are and tables clear/readable. Writing is clear.\n\n\n**Overall** I am quite happy with the relative improvements of the paper and believe that the community will overall benefit from the provided insights and results. \n\t\t\n\t\t\n**Minor Clarity Issues/ Comments:**\n\nCL1: Line 52-55 may not hold true in light of new publications (While the overall notion of these kind of work being sparse is true, I feel it is worth mentioning other similar works). [1] [3] \n\n** References:**\n\n[1] Rafailov, Rafael et al. “Offline Reinforcement Learning from Images with Latent Space Models.” L4DC (2021).\n\n[2] Schrittwieser, Julian et al. “Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model.” Nature 588 7839 (2020): 604-609 .\n\n[3] Shrestha, Aayam et al. “DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs.” ArXiv abs/2010.08891 (2020): n. pag.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have done a good job responding to the reviewers' questions. The reviewers are in consensus that the paper is a worthy contribution to the empirically aspects of offline reinforcement learning.

The authors are encouraged to include some discussion in relationship to the recent advances in the theory of offline RL, e.g., those from the recent workshops:

1. ICML RL Theory workshop:  https://lyang36.github.io/icml2021_rltheory/#papers
2. NeurIPS'20 Offline RL workshop:  https://offline-rl-neurips.github.io/

and the references therein. Some of the papers there might provide interesting theoretical insight into why model-based approaches are the way to go for offline RL.



Decision: Accept (Spotlight)