Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper identified three key ingredients to speed up the learning process for MBRL models. Specifically, the authors, building on TD-MPC, proposed a framework with policy pretraining, seeding, and finetuning with interactive learning, and showed great sample-efficiency improvement on a set of challenging visual controlling tasks.
strength_and_weaknesses: ### Strength
- The authors have provided a thorough comparison with prior work and have shown significant improvement on visuo-motor control tasks. Ablation studies also confirm the importance of each proposed training phase.

### Weakness
- minor
    - Could the authors please elaborate more on your motivation? How is the expert demonstration helping the exploration bottleneck of model-based RL?
    - TD-MPC uses real data for policy learning and conducts decision time planning. Meanwhile, other MBRL like Dreamer-V2 learns the policy in imagination rollouts, do you think the proposed framework can also benefit Dreamer-V2?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The results might be hard to reproduce.
summary_of_the_review: This paper identified three key ingredients to speed up the learning process for MBRL models. Experiments have shown significant improvement on a set of challenging control tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper demonstrates that a small number of expert demonstrations could assist model-based RL to learn faster. Model-based RL algorithms suffer from low sample efficiency, especially in complex environments due to exploration challenges. The proposed method alleviates this problem by continually reusing demonstrations to mix in desired state-action distributions. The proposed framework consists of three phases: (1) policy pretraining on a handful of demonstrations, (2) seeding phase where pretrained policy, with added exploration, is used to collect a dataset from the environment, in order to train world model and critic, (3) finetuning policy interactively with data from all phases. Evaluation on visual manipulation tasks with sparse rewards and locomotion tasks with dense rewards indicate the effectiveness of the proposed method.
strength_and_weaknesses: Strength:
1. The motivation for improving sample efficiency is clearly explained.
2. The paper presents experiments that indicated the effectiveness of the proposed method and provides extensive ablation studies on the importance of each designed phase.
3. The paper is well-written and easy to follow.
Weakness:
1. The idea of combining expert demonstrations with RL lacks novelty, plenty of works have implemented this before.
2. Under an unlimited sampling budget, will the proposed method and other baselines eventually converge at the same level?
clarity,_quality,_novelty_and_reproducibility: - Clarity: The proposed method and evaluation are clearly clarified.
- Quality: The paper is well-written and structured.
- Novelty And Reproducibility: The idea of incorporating expert demonstrations into RL is now general, and thus lacks novelty. This work can be easily reproduced with a public database.
summary_of_the_review: The evaluation of the proposed method is convincing. However, the main idea is not novel enough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes several techniques to improve the sample efficiency of the model-based RL. Specifically, demonstrations are leveraged in a more clever way: first training the policy, then training the world models and the critic with pre-trained policy, finally training the policy in a model-based manner. Experiments are conducted on three task suites with necessary ablation study.
strength_and_weaknesses: Strength:

The paper has clear writing with necessary explanations about the motivations. A good summary of previous related works is provided. The experiments are solid and thorough in terms of the number of tasks. Ablation studies are also conducted to prove the effectiveness of the proposed techniques.

Weakness:

Fig. 2(c) is unclear. The caption says the step (c) collects data using the model, however, the action $a$ is sent back to the environment. I assume this means interaction with the real environment also happens to collect data. So is the model only used for value estimation in TD-learning? If so, since TD-learning only requires the value of the next state, why are three steps plotted here? Please clearly indicate whether each $r$, $a$ and $z$ are from the prediction model or the rollout trajectories.

What is the policy gradient? It seems a majority of the model updates in phase 2 and 3 are using TD-MPC method, however, the policy gradient as an important component in TD-MPC is not clearly explained. Only in Section 2 the paper briefly mentions that ‘The policy … and is optimized to maximize temporally weighted Q-values’. How specifically is the temporally weighted Q-values implemented and what’s the explicit form of the policy gradient?

Why $\pi$ at the seeding phase and $\Pi$ at the third phase? It’s not clear to me why the policies use different notations in the second and third phase. Please explain it in the paragraph.

Lack of experiments. For experiments, the most important baseline methods to compare for the proposed algorithm should be those falling in the category of model-based RL+IL setting as in Table 2. However, only the TD-MPC method is compared in the main results. The MWM and Dreamer-v2 should also be compared in the experiments across three task suites. The paper only reports partial results in Fig. 12. Moreover, given that the results in one (Meta-World) of the two environments show similar performances across three methods, it does not prove the strong performance gain of the proposed method. 

clarity,_quality,_novelty_and_reproducibility: Some details about the algorithm need to be explained well, as said in the Weakness part. 

The quality of experiments and writing is good.

I generally think the proposed algorithm is short in novelty. The oversampling of demonstration is a trivial technique. And the idea of training the critic and prediction models are quite straightforward. Moreover, there is a lack of theoretical insight into the algorithm.

summary_of_the_review: I think the paper proposes an effective method for improving the model-based RL, but the idea is not novel enough. Also, a lack of important experimental comparison is another major drawback. Details about the algorithm need to be modified and clarified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates a staged approach to model-based RL with behaviour demonstrations, exploring the effect of each of these stages. Initially, demonstration data is used to train a behaviour cloning policy. This policy is then used with some exploration to gather data for model learning. Finally model and policy fine tuning takes place using environment interaction. Data augmentation is applied, alongside demonstration sampling, which is used to ensure the policy does not deviate significantly from the initial BC mode. Results show that this approach (MoDem) outperforms a number of other acronyms on a number of simulated benchmarks (dm control, meta world, adroit), and that exploration with the BC model seems to be key to this success. 
strength_and_weaknesses: Strengths:

The paper is clearly written with decent ablations, and the 3 stage approach proposed provides clear improvements over a number of methods (seemingly strong baselines) on relatively challenging benchmarks.

Weaknesses:

While I appreciate the exhaustive empirical study of the approach, the findings are not unexpected, in light of prior work. I see nothing technically wrong with this work, but nothing groundbreaking either.

I think it would be valuable to reflect on the relationship of the proposed work to Dagger [1], which at its core follows a similar idea, albeit without the model-based rl aspects.

Some minor comments. 

Figure 1. "solves 21 hard robotics tasks..." Could you define solve? How can we solve manipulation, pick and place or locomotion? Does a 53% success rate constitute "solve"? Perhaps a better, more precise description is needed. 

EQ 1 - can you define H as the planning horizon here? it is mentioned much later, but nice to have it here already.

Fig 4 - why is the oracle model from state so poor?

Refs:
[1] Ross, Gordon and Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, with good ablations and a seemingly reproducible approach.  

There has been prior work highlighting the benefits of pre-training policies with demonstration data [1], the importance of exploration in RL [3] and the need to balance policies against demos to avoid losing inductive biases [2], which limits the novelty and importance of the contributions made by this paper. At first glance the proposed approach seems somewhat obvious, but I appreciate the fact that the authors took the time to explore this question in detail. 

[1] Ross, Gordon and Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011
[2] Peters,  M¨ulling and Altun, Relative entropy policy search AAAI 2012
[3] Ecoffet,Huizinga, Lehman, Stanley, Clune, Go-Explore: a New Approach for Hard-Exploration Problems, 
summary_of_the_review: These results are potentially interesting to SOTA chasers in the environments tested and the paper describes a relatively clear and seemingly reproducible approach to do well in these settings. For this reason, I rate the paper as marginally above the acceptance threshold. I do not rate this higher, as I do not think the strength of contribution or novelty of this work warrants this.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The paper proposes a new approach for speeding up model-based RL. Three phases are designed where a small set of demonstrations is used in multiple ways. The approach is validated with robotic tasks.

Strengths:
- Well written paper
- The approach can be applied to a range of algorithms
- Ablation studies sowing the need of the setup
- Extensive evaluation

Weaknesses:
- The approach is somewhat obvious (at least in hindsight)
- The method is developed based on intuition and validated with extensive experiments, i.e., more a heuristic method
- A comparison to sota MBLR methods would still be interesting
justification_for_why_not_higher_score: The paper presents interesting insights from extensive evaluation. The paper is a bit more on the "how to make it work" side than "fundamental new idea".
justification_for_why_not_lower_score: All reviewers argue for acceptance.
Decision: Accept: poster