Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an approach (called MAMF) to augment a model-free reinforcement learning algorithm (namely TD3) with a learned dynamics model. The learned model is used to sample imaginary transitions from actually observed states, which are then used to train the critic. The imaginary transitions are filtered based on two criteria: 1) higher Q values, and 2) higher uncertainty. Experimental results in simulation show that this method outperforms prior methods in simulated manipulation tasks, achieving higher performance and better sample efficiency.

strengths_and_weaknesses: Strengths:

The paper is clear and easy to read, and has the relevant references (to the best of my knowledge). It also clarifies the differences with other closely related prior approaches well. Experiments are performed on many diverse environments, showing the effectiveness of the approach across domains and problem types. Ablation studies are also presented for various aspects of the algorithm, which is useful to see. I believe a key part of what makes this method works well is that the learned model is only used for extrapolating a single time-step into the future, limiting the amount of bias introduced by an inaccurate model. Another good feature is that the proposed strategy is not necessarily limited to augmenting TD3: it could (in theory) be applied to any model-free RL algorithm, although this is not tested in the paper. This definitely extends the applicability of the algorithm.

Weaknesses:

I believe the main weakness of this paper is probably the lack of real-world robot experiments. However, I believe this is mitigated by: 1) extensive experiments in a variety of simulation domains, and 2) the generality of the approach to any model-free RL algorithm, not just TD3.

Another potential issue which could make real-world application more challenging is the difficulty of training dynamics models for high-dimensional state/observation spaces such as camera images.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This is a good paper, which experimentally evaluates a model-based augmentation to model-free RL algorithms. While there are some weaknesses and limitations when considering real-world application, it nonetheless seems like a useful strategy to consider, and I would consider this a good contribution to the RL community.
issues: I couldn't find/think of anything that needed revision.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an algorithm that combines model based and model free RL for control. Specifically the authors propose a learning framework where a learned model is used, after being pre-trained, to roll-out imaginary transitions that are used to update a critic in order to eventually facilitate exploration in the real environment to solve a task. The authors present experimental evaluation of the proposed algorithms on a variety of simulated environments and compares to a number of baselines. 
strengths_and_weaknesses: Strength: The paper is written clearly and is easy to follow, the algorithm is also presented well and it would be in principle possible to reimplement it. The direction of combining model based and model free is relevant and interesting and looking at the model as a possible avenue for doing exploration is also an interesting direction. The results overall seem promising and show that the method works. 

Weaknesses: I am not super convinced by the results, compared to TD3 there might not be a clear benefit as the model was also already pre-trained. The final performance is mostly comparable to TD3. I would not be so worried about this itself, however since you don't show any transfer/generalisation experiments learning a model without reusing it, where TD3 actually also performs well, seems a bit of an overhead to me.  

I think the paper would be very strong if you could show that you can reuse the model across tasks, or are you doing this already but did not make clear in the paper? 

I think that for that it would also be important to clarify the pre-training of the model more, and if you pre-trained for each experiment a new model. 

- robot experiments are missing 

Questions:

- could you give a better intuition or understanding of why the initialisation of the bias help? Also how did you come up with this equation, what why were you expecting this would help? 

- How much data did you use to pre-train and how close is this data to the experimental domain?

- Why did you not compare to a purely model based RL algorithm like this one https://arxiv.org/pdf/1912.01603.pdf ?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: There are a couple of questions that need clarification but I'm willing to adjust my assessment during the authors respond period.
issues: see above
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies improving the sample efficiency of model-free DRLs using a learning transition model. The proposed method uses a learning model to generate imaginary transitions, which are then used to update the critic. The authors applied the proposed method to various simulated robot tasks and demonstrated that it outperformed other methods with high sample efficiency.
strengths_and_weaknesses: Strengths:
* When the proposed method is applied to TD3, sample efficiency and performance are improved simultaneously
* Comparison with other methods on various robot simulation tasks
* An ablation study of the proposed method is conducted

Weaknesses:
* The proposed method is a general method to improve DRL sample efficiency. It is not specialized for the complex manipulation tasks in the title. The association between the proposed method and complex manipulation is not strong.
* No comparison in real robot task
* As the authors describe in their limitations, the sample efficiency is not high enough to learn complex manipulations on a real robot. The combination of sim2real and the proposed method is given as an example as a solution; however, the proposed method would not be necessary if a simulator could be used.
* No comparison of applying the proposed method to Soft Actor Critic (SAC), which is more sample efficient than TD3.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The approach is general enough to apply to a variety of tasks. Simulation results show high sample efficiency and performance for various robot tasks.
issues: It would be necessary to explain how the proposed method can be used to train real robot tasks by comparing it with previous sample-efficient methods.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a deep reinforcement learning approach to improve the sample efficiency of an actor-critic method. It adds random noise to action values to create imaginary transitions. To address the model bias from using the transitions, it introduces high Q-value initialization of critics and two filtering methods that favor transitions with high target Q-values and high uncertainty in the Q-value estimates. They compare their method with different variants of TD3.
strengths_and_weaknesses: **Strengths:**
* This paper presents several interesting tricks to counteract the model bias of using imaginary transitions. 
* The paper is well structured and written.
* Figures are clear and useful for understanding the results.

**Weaknesses:**
* Since the proposed method pre-trains a dynamics model, it is unfair not to show the pre-training steps in the experimental results (Figures 2 and 3) when compared to the baselines. The proposed method's sample efficiency could be overestimated as the number of pre-training steps is not taken into account.
* The experimental results in Figure 2 will look more convincing if non-TD3 algorithms are used for comparison, such as SAC. Figure 2 looks more like an ablation than a baselines comparison since TD3 and MA-TD3 are implementations without some of the components in your method. 
* Lack of real robot experiments.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper presents an interesting method to improve the sample efficiency of an actor-critic method. However, two issues need to be addressed. First, the proposed method pre-trains a dynamics model in the environment, but the number of pre-training steps is not shown in the experimental results, resulting in an unfair comparison. Second, most baselines (TD3 and MA-TD3) in Figure 2 are implementations without some of the proposed method's components, making it more like an ablation study than a baselines comparison. The authors should consider adding state-of-the-art non-TD3 methods in Figure 2. Additionally, having real robot experiments will certainly improve the paper.
issues: Results:
* [5.2.1] It will be clearer if there is an additional sentence that describes the details of MVE, similar to what you did for MA-BDDPG.

Conclusion:
* The sentence “one key novelty is to generate imaginary data with exploratory actions” is not true. Some prior works also add Gaussian noise to the action values, such as [1]. 

Typos:
* Line 46: “see sec:result”.
* Line 179: there is an extra “to”.

**References:**

[1] Hado van Hasselt and Marco A. Wiering. Reinforcement Learning in Continuous Action Spaces, 2007.
review_title: null

Meta-review: metareview: Strengths:
+ Reviewers agree that the method proposed is technically sound and well explained.
+ An extensive set of experiments in simulation seems to indicate a sample efficiency and performance advantage over the TD3 baseline.
+ The method is general (applicable to more than manipulation tasks, and applicable on top of more algorithms than TD3)

Weaknesses
- There are no experiments on real robots.
- The pre-training time is not considered in the comparisons, which seems unfair if the main point is improved sample efficiency. The authors have addressed and clarified this aspect in discussion.
- Re-using the learned model on a different task would highlight the advantages of this method, but that aspect is not tackled. The authors did attempt to investigate this aspect in the discussion period, but with limited results. This remains a key area for improvement for future studies.
- Showing performance improvements on top of a different algorithm than TD3 (e.g. SAC) would improve the paper, but that aspect is not tackled. Again, the authors attempted to address this, but in the limited time available during discussion the results were mixed. This also remains a key area for improvement for future studies.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)