Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This is a system paper where DreamerV2 [Hafner 2020] was applied to real-world tasks on four different robots: A1 Quadruped walking, UR5 visual pick and place, XArm visual pick and place, and Sphero navigation. The algorithm was trained directly on the hardware and showed great performance under the same set of hyperparameters.
strengths_and_weaknesses: ### Strengths:

1. Implementation of RL on robotics hardware

2. Great performance across 4 robots and tasks, under the same hyperparameters


### Weaknesses:

1. The author claimed software infrastructure as one contribution but it does not seem to have been included in the submission hence I can not properly assess this contribution. 

2. Some details were missing that would otherwise help better understand the work:
  - (reproducibility)Line 176 mentioned the use of low-pass filter but the details were not given.
  - (reproducibility)What do the lines and the shading in the learning curves represent?
  - How does the wall clock time (1hr and 10 mins) translate to the number of samples? What was the compute resource used?
  - The manipulation experiments seem to use a green background for Xarm and a background with a mixture of colors for the UR5. Is there any particular reason for the difference in the setup? 

3. Additional ablation experiments that would be interesting:
  - How was the reward function in Eq. 5 chosen? If the authors had put some efforts into reward shaping, it would be great to see an ablation study on that. If not, could the authors clarify?
  - Line 231 and Appendix A.3 showed the adaptation to lighting conditions. This is quite interesting and I’m curious to see what happens when the lighting switches back to nighttime again (after the adaptation). Could this ablation be added?

4. Minor:
  - what is sg() in Eq. 4? It should be explained.
  - Line 125 typo: reperameterization -> reparameterization

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I applaud the authors for validating deep RL algorithms on real robots. 
Running Dreamer on real-world robots is novel to my best knowledge and is potentially of high impact. 
My main concerns are some clarifications and missing ablation studies that I think will strengthen the work. I will be happy to raise my score if my concerns are properly addressed.

issues: Please see the weakness list for questions to be addressed.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper showcases that learning of sufficiently accurate world models for the real world is possible. In particular, the Dreamer framework is applied to four real tasks that are distinct in their particular nature. Learning is performed without using a simulator or demonstrations. 
strengths_and_weaknesses: # Strengths
- impressive performance on real robots (e.g., learn to stand up and walk on real system in 1h).
- All tasks are solved using the same algorithm and hyperparameters.
- The paper is well written despite occasional typos.
# Weaknesses
- no or very little algorithmic advances. The paper uses the existing Dreamer framework and feels more like a report in large parts of the paper. It would be nice to understand what the authors needed to do to make the Dreamer framework run on real systems compared to running it in simulated environments. Are (3) and (4) modifications to the previous work that needed to be performed for the real system?
- Contributions: Although I acknowledge that the paper is the first to make Dreamer run on real robots and the open-source argument, I doubt that this paper pioneers to learn to walk in 1h and reaches previously unseen performance on the visual pick and place task. For the letter task, the work by Sergey Levine’s group presented immense results over the past years. The results in the paper are indeed impressive, but I feel the contributions are not formulated carefully. One of the biggest contributions I see is that all tasks have been solved by a single algorithm and constant hyperparameters, which is not mentioned in this list. It would be nice if the authors would clarify if the hyperparameters of behavior learning part (Figure 3 (b)) are also the same for each task. 
- The paper claims to “approach human performance” in the visual pick and place task, which implies that the robot is better at picking the objects than a human. However, in Section 3 it is mentioned that the robot is teleoperated by a human, which is a much easier baseline to beat. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper reports impressive results, but seems not to include any algorithmic advances. Moreover, the paper seems to partially overstate the performance: (i) the argument to match human performance in the pick and place task leaves room for misinterpretations and (ii) the results in the Sphero Navigation task only match the performance of the baseline.
issues: It would be interesting to understand the contributions of this paper better. Why exactly are the empirical results new, since similar performances have been reported before? Please also comment on the weaknesses listed before. If addressed appropriately, I'm happy to increase the score.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, the authors applied the Dreamer algorithm directly on real robots to learn locomotion, manipulation and navigation tasks. Except for the parallel execution of data collection and controller learning, the paper made little change to Dreamer and applied it as-is. Nevertheless, the tasks were trained within a reasonable time budget. The authors therefore demonstrated the feasibility of applying a model-based RL algorithm for online learning on physical robots and established baselines for future methods.
strengths_and_weaknesses: Strengths
* The idea of applying Dreamer directly on robots to learn real-world world model is novel.
* The results cover a variety of robotic tasks. They encourage future research that explore/improve model-based RL methods directly on physical robots and serve as baselines.

Weaknesses
* Lack of details to support data efficiency. While wall clock time comparison is an important criteria, the number of trajectories/data used for learning should also be put to check.
* Lack of hardware setup info. While training a locomotion controller on a physical robot within 1 hour is impressive, the readers may also want to know the spec of the hardware that allowed this.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well written and easy to follow. Although the technical innovation is mainly applying Dreamer directly to physical robots, the results are encouraging, and will likely inspire more future works along the same direction. The reviewer believes the paper can be improved if some details and additional analysis (see Issues below) are added.
issues: * Was the parallel data collection and learning also applied to the baselines? The reviewer assumes this is not the case due to the nature of the baselines. If not, Dreamer may have consumed more data. What is the amount of data used for training in Dreamer and the baselines?
* Can the authors add the specification of the hardware used for learning?
* DrQv2 seems promising too. If data augmentation were applied to the baselines in other experiments (e.g. the pick and place tasks), what would the performance gaps look like? How narrow would they be?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper evaluates the Dreamer algorithm by training directly on the hardware on 4 tasks.

strengths_and_weaknesses: Pros:
- real world training on tasks that require use of multiple sensors.
- Dreamer manages to learn the pick and place task in the sparse reward setting
- Impressive recovering result for the quadruped robot

Cons:

One concern is the missing details for the experiments, as it is an application paper (no new method is proposed).
For instance, it is said that a low-pass filter was used but there is no information about the cut-off frequency (and how it was chosen).
Could this low-pass filter be avoided as in [2] (where training is also directly done on hardware)?
Was a history given as input, or how was the Markovian assumption not violated (see [1, 2])?

Details are also missing regarding how was the success measured, both for picking and Sphero navigation task. For instance, for the sphero task, it seems that the system can compute the distance to the target. If so, then learning probably doesn't make sense for that task.

One other concern is that the picking and sphero task are not visually challenging. Objects are easy to distinguish (good contrast with the background) and are not cluttered. The main challenge for the picking task comes from the sparse reward.

The difference between the two picking tasks is unclear. There are on two different robots, but both uses vision and proprioception and both are using objects that are relatively soft.

The recovery strategy learned is good, but the walking gait is far from something desirable (for instance, the knees touch the ground while walking), what happens if the agent is trained longer?

The baselines for the picking tasks seem unfair. One should at least provided a pre-trained (frozen) encoder to Rainbow as in [3] (where they also learn directly on hardware). For the sphero task, despite being model-free, the well-chosen baseline is competitive with the model-based approach.

Several times the paper claims to be close to "human-level performance", but this claim is misleading. It corresponds to a human operator that had access to the robot for 20 minutes.

Dreamer is meant to be data-efficient, but it sill takes 9h for the picking task...


## Remarks/Questions

- typo l224 "the the"
- the following references that train RL agent directly on real hardware seem to be missing (even from the appendix) and are probably relevant for that paper: [1, 2, 3]
- for future work, authors should consider TQC [4] (SAC with distributional critic) as model-free baseline for continuous control
- I would not spend too much time on presenting the Dreamer method and leave more space for the experiments/discussion/related work


[1] Learning to Walk via Deep Reinforcement Learning (RSS19)

[2] Smooth Exploration for Robotic Reinforcement Learning (CoRL21)

[3] Learning to Drive in a Day (ICRA19)

[4] Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (ICML20)
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I would currently recommend the paper for rejection, more experimental details are needed.

issues: - see issues raised in the "weakness" section
- it is said that the "infrastructure will be released" but no link is provided nor is it in the supplementary material
- for the sphero navigation: are observation stacked?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper demonstrates that the previously introduced Dreamer algorithm can be applied to learning control tasks directly on real robots, without simulators. The authors successfully train policies for three different tasks: quadruped locomotion, pick and place with a robotic arm, and simple navigation of a 2-wheeled vehicle. 
The Dreamer algorithm learns a world model and then uses the model to train the control policy. The purpose of this setup is to increase the sample efficiency by having the policy interact with the model instead of the real world.

strengths_and_weaknesses: The main strength of the paper are the impressive results. To date, it is probably the most convincing example of a control policy trained in real life. The fact all tasks can be solved with the same hyperparameters and minimal tuning is also remarkable. Furthermore, the fact that vision based tasks can be learned live on a real robot greatly reduces the burden training such behaviours. It removes the need of complex sensor/noise modelling needed for sim2real transfer (this fact could be emphasised more in the paper). Finally, the paper is well written, structured clearly, and the supplementary video is a great addition.

On the other hand, the scientific novelty of this work is limited. While it is common (and important) to apply existing algorithms to real world tasks, such works should emphasise the main takeaways and potential modifications which were necessary to achieve the new results. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: While there is no algorithmic novelty in this paper, the results set a new baseline for live learning on real robots. I would recommend accepting the paper if the authors extend the results section with more insights and a proper study of sample efficiency.
issues: Contributions presented in the introduction are repetitive and can be summarized by the first point ‘Dreamer on Robots’

The sections ‘World Model Learning’ and ‘Actor-Critic Learning’ seem to be repeating [1]. There is no need to repeat this description. Instead, it would be more interesting to focus on the differences, if any.

More details should be provided about the “use of an asynchronous actor and learner” since this is one of the modifications necessary to achieve results in real life.

From my understanding, using a world model should mostly improve sample efficiency. However, the paper does not compare the efficiency of the proposed approach with more common RL algorithms such as SAC, PPO, etc.. (such an experiment could be done in simulation). It would be interesting to see how many samples these algorithms need to reach a similar performance.

The introduction claims that the world model removes the need for state estimation. However, in the quadruped example, the projected linear velocity is used to compute rewards. Isn’t it coming from the state estimator of the robot? Similarly, the pick and place task uses Forward and inverse kinematics which requires a model of the robot (of course a simple model). Finally, how is the reward of tasks 3.2, 3.3, and 3.4 computed? Don’t they require a detection pipeline?
This doesn’t hinder the usefulness of the proposed approach, but the authors should clarify that some simple models are still needed.

What new information can the reader get from experiment 3.3 compared to 3.2?

Why does SAC suffer from dead-locked leg configuration while your method doesn’t? Is this simply due to random chance or is there an explanation?

Minor issues: 
In the description of figure 3, it is not clear what are the two neural networks.
Cite the work you are referring to in 3.1
The paper mentions multiple times four tasks on four robots. Three tasks on four robots seem more appropriate 
Typos:
 reconstructions -> reconstructs (figure 3)
Which inapplicable -> which is inapplicable

[1]: D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv
285 preprint arXiv:2010.02193, 2020.
review_title: null

Meta-review: metareview: This paper demonstrated the application of the Dreamer framework to four real robot tasks. It verified that RL with world models could be applied in the real world without using simulators and demonstration data, and better than model-free RL. 

Strengths: 

- Impressive performance on real robots 

- Four tasks are solved by the same algorithm with the same hyperparameters

- Well-written and organized paper 

Weaknesses: 

- No novelty in the method 

- Missing details and insights for the experiments 

- Lack of qualitative evaluation in sample efficiency compared to other RL. 

----- Post rebuttal -----

In the rebuttal period, some reviewers’ concerns were resolved and turned into rather positive, and all the reviewers almost agreed that this paper should be accepted. Therefore, I conclude that the paper should be accepted. I strongly encourage the authors to take all the reviewers’ suggestions for the final version of the paper, especially clarifying the statement on the non-standard learning curves made with only one run for a deep RL algorithm to avoid any misleading to the readers.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)