Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents ROSMO, a model-based offline RL algorithm based on MuZero Unplugged (MZU). ROSMO improves MZU in the offline RL setting in two main ways. First, it only performs one-step rollout for advantage estimation, rather than MCTS with multi-step rollouts. Second, it uses advantage-based policy improvement with advantage-filtered behavior cloning. ROSMO outperforms MZU in a low-data regime and stochastic transition and is less sensitive to network parameters. In the Atari benchmark, ROSMO outperforms offline RL baseline algorithms.

strength_and_weaknesses: [Strengths]
1. The paper is well-written and easy to follow. The modification to MZU for offline RL is well-motivated, and it makes the algorithm simpler yet more effective in terms of computation cost and performance.
2. ROSMO outperforms baseline algorithms in the experiments.


[Weaknesses]
1. Technical novelty is a bit limited. The short-horizon rollout for MBRL has already been widely used (e.g., [1]), and the behavior regularization is also very similar to the one in previous works (e.g. [2,3]). I am curious to see the result when only (11) is used for the policy improvement while not using (9), since it seems (11) can also serve as a policy improvement.
2. There are missing comparison with model-based offline RL algorithms in the experiments, e.g. [4,5,6].
3. In (1), there is no explicit definition of each term, i.e. $\ell^r$, $\ell^v$, and $\ell^p$.
4. In Figure 4d, it seems OneStep is not yet converged. It would be great to provide the result with a longer number of steps to see whether ROSMO clearly outperforms OneStep at convergence. In the current result of Figure 4d, the conclusion is unclear to me.
5. It would be great to include the result of baseline algorithms in Figure 2. Why is ROSMO only being compared with MZU?
6. In section 4.1.(3), why is MZU more sensitive to the parametrization of dynamics? It would be great to provide some possible reasons.


[1] Janner et al., When to Trust Your Model: Model-Based Policy Optimization, 2019

[2] Oh et al., Self-Imitation Learning, 2018

[3] Wang et al., Critic Regularized Regression, 2020

[4] Yu et al., COMBO: Conservative Offline Model-Based Policy Optimization, 2021

[5] Kidambi et al., MOReL : Model-Based Offline Reinforcement Learning, 2020

[6] Yu et al., MOPO: Model-based Offline Policy Optimization, 2020


clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written and easy to follow. The technical novelty seems to be limited, given that each component of the improvements to MZU is not new.
summary_of_the_review: The paper provides modifications to MuZero Unplugged for offline RL, which makes the algorithm simpler yet more effective and this is a nice contribution. Still, the technical novelty is a bit limited, and it would be great to have more offline RL baselines in the experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces ROSMO, a simpler and better version of Muzero Unplugged for offline policy optimization. In the offline setting, Muzero Unplugged can suffer from limited data-coverage, inaccurate models and an expensive compute budget. Instead of relying on an expensive MCTS and suffering from out of distribution errors as well as compounding of errors, ROSMO resorts to a one step lookahead approach to learn the same components as the one needed by Muzero Unplugged (policy and value targets) and a behavior cloning regularization to prevent going into an unsafe regions. ROSMO outperforms Muzero Unplugged on the Bsuite and Atari benchmarks. The authors also provide an ablation study.
strength_and_weaknesses: The paper is well written and well organized. It was easy to read and understand. I liked that the authors recalls the ideas behind Muzero Unplugged in section 2.2 (in a more general way) and the MCTS policy improvement used by Muzero Unplugged in section 2.3, I think this eases the reading of the paper for people not very familiar with Muzero Unplugged.

Muzero Unplugged is a not so simple algorithm to implement and has no official open sourced implementation as far as I know. I agree with the authors write that "Muzero Unplugged is sophisticated". Having a simpler, yet as effective (and even better) algorithm is a great contribution for the community, especially if the implementation is to be made open sources after acceptance of the paper.

Empirical evaluations of Muzero Unplugged and ROSMO as well as the ablation study show the pertinence and the superiority of the ROSMO algorithm.

It would be better to say that having a one-step update might not be the only way to solve the compounding of errors as well as the fact that Muzero can go in unsafe regions. For instance having a model with its own uncertainty could be another direction to avoid unsafe regions and planning too much into the future. I think that planning (with MCTS or a different search algorithm) could make better targets if the uncertainty is used correctly. I agree that this would make a more complicated algorithm and for that it's good to have the ROSMO algorithm as it is simpler and will provide a good baseline when developing other algorithms.

The part on the stochastic transitions in section 4.1 is the least clear for me: why not add noise in the observations? why is it connected to the compounding of errors?

Designing simpler algorithms is a nice contribution to the community and for this reason I would be in favor of accepting the paper. 

It is a pity that the authors are not using the D4RL benchmark which has become the default benchmark for offline RL and for which performances of MOPO/CQP/Morel etc... are available. It would have been nice to compare the ROSMO algorithm to these algorithms. Muzero unplugged is said to be the state of the art but was not compared to these algorithms to the best of my knowledge which are uncertainty based.
clarity,_quality,_novelty_and_reproducibility: The author provide a detailed pseudo code in the appendix and will release their code if the paper is accepted.

Question:
- In the pseudo code, the author use K as the number of unroll steps. This K is only used for the ablation study to compare the performance between K=1 and K=5 and the ROSMO paper is using K=1 by default hence the "one-step" update? Is that correct? Same question for the loss in equation (1), ROSMO is using K=1, it is just for the ablation that K=5 is used?

- p5: v_t+n = f_theta'(s_t+n), I think it should be f_theta', v?
- p5: pi_prior is the prior policy (often realized by the target network). In practice in the paper I assume it is always the target network that is used for the prior policy?
- p7: Muzero is insensitive to search depth (Hamrick et al.) : I think the conclusion is a bit more nuanced than this. They say that "simple and shallow forms of planning **may** be sufficient". Moreover "Indeed, out of all our environments, only Acrobot and 9x9 Go strongly benefited from search at evaluation time. We therefore emphasize that for work which aims to build flexible and generalizable model-based agents, it is important to evaluate on a diverse range of settings that stress different types of reasoning"
- p7 (last paragraph): Figure 2(b) -> Figure 2(c)
summary_of_the_review: Overall I think this is a nice paper and good contribution to the community. A comparison to other model-based algorithms for offline RL on the D4RL would have been a big plus (Morel/MOPO, ...).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies offline model-based RL, specifically identified several shortcomings of the previously proposed MuZero Unplugged. The proposed ROSMO (Regularized One-Step Model-based Offline RL) incorporates (i) one-step lookahead instead of MCTS planning, (ii) behavior regularization. 

Extensive experiments and ablation studies are conducted on the Atari benchmark and achieve impressive results, both in terms of performance and runtime efficiency. 
strength_and_weaknesses: **Strengths**
- Empirical results are pretty extensive and strong across the board. 
- Sec 4.1 ablations help the reader understand what each modeling component addresses. 

**Weaknesses**
- MOPO/MOReL: "both of them focus on state-based control tasks and hence are not compared here" - could you clarify what this mean? 
clarity,_quality,_novelty_and_reproducibility: Clarity: this paper is clear. 

Quality: technical details are provided, experimental details are also rather extensively documented. 

Novelty: ideas used in this paper are pretty straightforward and are standard ingredients of offline RL (one-step lookahead is kind of analogous to Q-learning (?), behavior regularization also commonly used in past work), 

Reproducibility: authors promised to open-source code upon publication. 
summary_of_the_review: Overall the paper is written quite clearly and empirical results are indeed strong, however my slight concern is where this contribution lies within the literature, perhaps a dedicated related works section would help - which should include more discussion on previous model-based offline RL approaches in "Offline RL Tutorial by Levine et al. 2020".
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present a novel method for model based off-policy RL that instead of performing monte-carlo-tree-search to identify a good continuation value of each action, it only performs a one-step advantage learning. The benefits are computational, but also in terms of robustness to model mis-specification and noisy dynamics. They show that their method outperforms other methods on the BSuite benchmark suite. 

There is valid criticism that the elements invoked in this new method are not novel and that there was not sufficient comparison with other model-based approaches in the experiments, and only with mu-zero. However, the majority of reviewers value the practical aspect of the contribution and combining existing elements into a practically working algorithm is non-trivial. 

Moreover, the reviewers have offered extensive responses to the reviewer criticism and have updated their manuscript with new experiments comparing to the aforementioned model-based approaches that were not there in the original submission; showing superior performance. Moreover, there was some criticism on the benchmark suite being used (compared e.g. to DR4L), to which the authors again provided satisfactory response. Therefore the main concerns have been addressed in the revision
justification_for_why_not_higher_score: The method is novel and practical but uses existing methodological elements.
justification_for_why_not_lower_score: The method provides great empirical performance on realistic datasets.
Decision: Accept: poster