DayDreamer: World Models for Physical Robot Learning
Authors: Philipp Wu
Alejandro Escontrela
Danijar Hafner
Pieter Abbeel
Ken Goldberg
Venue: CoRL 2022 Poster (2022)
Keywords: reinforcement learning
model-based reinforcement learning
world models
locomotion
sample-efficiency

Abstract: To solve tasks in complex environments, robots need to learn from experience. Deep reinforcement learning is a common approach to robot learning but requires a large amount of trial and error to learn, limiting its deployment in the physical world. As a consequence, many advances in robot learning rely on simulators. On the other hand, learning inside of simulators fails to capture the complexity of the real world, is prone to simulator inaccuracies, and the resulting behaviors do not adapt to changes in the world. The Dreamer algorithm has recently shown great promise for learning from small amounts of interaction by planning within a learned world model, outperforming pure reinforcement learning in video games. Learning a world model to predict the outcomes of potential actions enables planning in imagination, reducing the amount of trial and error needed in the real environment. However, it is unknown whether Dreamer can facilitate faster learning on physical robots. In this paper, we apply Dreamer to 4 robots to learn online and directly in the real world, without any simulators. Dreamer trains a quadruped robot to roll off its back, stand up, and walk from scratch and without resets in only 1 hour. We then push the robot and find that Dreamer adapts within 10 minutes to withstand perturbations or quickly roll over and stand back up. On two different robotic arms, Dreamer learns to pick and place objects from camera images and sparse rewards, approaching human-level teleoperation performance. On a wheeled robot, Dreamer learns to navigate to a goal position purely from camera images, automatically resolving ambiguity about the robot orientation. Using the same hyperparameters across all experiments, we find that Dreamer is capable of online learning in the real world, which establishes a strong baseline. We release our infrastructure for future applications of world models to robot learning.


Reviews:
[{"review_id": "w3Y21a_ZWKL", "reviewer": "Reviewer_BYdE", "summary": "This paper demonstrates that the previously introduced Dreamer algorithm can be applied to learning control tasks directly on real robots, without simulators. The authors successfully train policies for three different tasks: quadruped locomotion, pick and place with a robotic arm, and simple navigation of a 2-wheeled vehicle. \nThe Dreamer algorithm learns a world model and then uses the model to train the control policy. The purpose of this setup is to increase the sample efficiency by having the policy interact with the model instead of the real world.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper are the impressive results. To date, it is probably the most convincing example of a control policy trained in real life. The fact all tasks can be solved with the same hyperparameters and minimal tuning is also remarkable. Furthermore, the fact that vision based tasks can be learned live on a real robot greatly reduces the burden training such behaviours. It removes the need of complex sensor/noise modelling needed for sim2real transfer (this fact could be emphasised more in the paper). Finally, the paper is well written, structured clearly, and the supplementary video is a great addition.\n\nOn the other hand, the scientific novelty of this work is limited. While it is common (and important) to apply existing algorithms to real world tasks, such works should emphasise the main takeaways and potential modifications which were necessary to achieve the new results. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While there is no algorithmic novelty in this paper, the results set a new baseline for live learning on real robots. I would recommend accepting the paper if the authors extend the results section with more insights and a proper study of sample efficiency.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Contributions presented in the introduction are repetitive and can be summarized by the first point ‘Dreamer on Robots’\n\nThe sections ‘World Model Learning’ and ‘Actor-Critic Learning’ seem to be repeating [1]. There is no need to repeat this description. Instead, it would be more interesting to focus on the differences, if any.\n\nMore details should be provided about the “use of an asynchronous actor and learner” since this is one of the modifications necessary to achieve results in real life.\n\nFrom my understanding, using a world model should mostly improve sample efficiency. However, the paper does not compare the efficiency of the proposed approach with more common RL algorithms such as SAC, PPO, etc.. (such an experiment could be done in simulation). It would be interesting to see how many samples these algorithms need to reach a similar performance.\n\nThe introduction claims that the world model removes the need for state estimation. However, in the quadruped example, the projected linear velocity is used to compute rewards. Isn’t it coming from the state estimator of the robot? Similarly, the pick and place task uses Forward and inverse kinematics which requires a model of the robot (of course a simple model). Finally, how is the reward of tasks 3.2, 3.3, and 3.4 computed? Don’t they require a detection pipeline?\nThis doesn’t hinder the usefulness of the proposed approach, but the authors should clarify that some simple models are still needed.\n\nWhat new information can the reader get from experiment 3.3 compared to 3.2?\n\nWhy does SAC suffer from dead-locked leg configuration while your method doesn’t? Is this simply due to random chance or is there an explanation?\n\nMinor issues: \nIn the description of figure 3, it is not clear what are the two neural networks.\nCite the work you are referring to in 3.1\nThe paper mentions multiple times four tasks on four robots. Three tasks on four robots seem more appropriate \nTypos:\n reconstructions -> reconstructs (figure 3)\nWhich inapplicable -> which is inapplicable\n\n[1]: D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv\n285 preprint arXiv:2010.02193, 2020.\n"}}, {"review_id": "a7uCbXi63qD", "reviewer": "Reviewer_niYi", "summary": "This is a system paper where DreamerV2 [Hafner 2020] was applied to real-world tasks on four different robots: A1 Quadruped walking, UR5 visual pick and place, XArm visual pick and place, and Sphero navigation. The algorithm was trained directly on the hardware and showed great performance under the same set of hyperparameters.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n1. Implementation of RL on robotics hardware\n\n2. Great performance across 4 robots and tasks, under the same hyperparameters\n\n\n### Weaknesses:\n\n1. The author claimed software infrastructure as one contribution but it does not seem to have been included in the submission hence I can not properly assess this contribution. \n\n2. Some details were missing that would otherwise help better understand the work:\n  - (reproducibility)Line 176 mentioned the use of low-pass filter but the details were not given.\n  - (reproducibility)What do the lines and the shading in the learning curves represent?\n  - How does the wall clock time (1hr and 10 mins) translate to the number of samples? What was the compute resource used?\n  - The manipulation experiments seem to use a green background for Xarm and a background with a mixture of colors for the UR5. Is there any particular reason for the difference in the setup? \n\n3. Additional ablation experiments that would be interesting:\n  - How was the reward function in Eq. 5 chosen? If the authors had put some efforts into reward shaping, it would be great to see an ablation study on that. If not, could the authors clarify?\n  - Line 231 and Appendix A.3 showed the adaptation to lighting conditions. This is quite interesting and I’m curious to see what happens when the lighting switches back to nighttime again (after the adaptation). Could this ablation be added?\n\n4. Minor:\n  - what is sg() in Eq. 4? It should be explained.\n  - Line 125 typo: reperameterization -> reparameterization\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I applaud the authors for validating deep RL algorithms on real robots. \nRunning Dreamer on real-world robots is novel to my best knowledge and is potentially of high impact. \nMy main concerns are some clarifications and missing ablation studies that I think will strengthen the work. I will be happy to raise my score if my concerns are properly addressed.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see the weakness list for questions to be addressed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "PTRWbthnISN", "reviewer": "Reviewer_HuH8", "summary": "In this paper, the authors applied the Dreamer algorithm directly on real robots to learn locomotion, manipulation and navigation tasks. Except for the parallel execution of data collection and controller learning, the paper made little change to Dreamer and applied it as-is. Nevertheless, the tasks were trained within a reasonable time budget. The authors therefore demonstrated the feasibility of applying a model-based RL algorithm for online learning on physical robots and established baselines for future methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The idea of applying Dreamer directly on robots to learn real-world world model is novel.\n* The results cover a variety of robotic tasks. They encourage future research that explore/improve model-based RL methods directly on physical robots and serve as baselines.\n\nWeaknesses\n* Lack of details to support data efficiency. While wall clock time comparison is an important criteria, the number of trajectories/data used for learning should also be put to check.\n* Lack of hardware setup info. While training a locomotion controller on a physical robot within 1 hour is impressive, the readers may also want to know the spec of the hardware that allowed this.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and easy to follow. Although the technical innovation is mainly applying Dreamer directly to physical robots, the results are encouraging, and will likely inspire more future works along the same direction. The reviewer believes the paper can be improved if some details and additional analysis (see Issues below) are added.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* Was the parallel data collection and learning also applied to the baselines? The reviewer assumes this is not the case due to the nature of the baselines. If not, Dreamer may have consumed more data. What is the amount of data used for training in Dreamer and the baselines?\n* Can the authors add the specification of the hardware used for learning?\n* DrQv2 seems promising too. If data augmentation were applied to the baselines in other experiments (e.g. the pick and place tasks), what would the performance gaps look like? How narrow would they be?"}}, {"review_id": "297yfaip6gn", "reviewer": "Reviewer_4BTT", "summary": "The paper showcases that learning of sufficiently accurate world models for the real world is possible. In particular, the Dreamer framework is applied to four real tasks that are distinct in their particular nature. Learning is performed without using a simulator or demonstrations. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- impressive performance on real robots (e.g., learn to stand up and walk on real system in 1h).\n- All tasks are solved using the same algorithm and hyperparameters.\n- The paper is well written despite occasional typos.\n# Weaknesses\n- no or very little algorithmic advances. The paper uses the existing Dreamer framework and feels more like a report in large parts of the paper. It would be nice to understand what the authors needed to do to make the Dreamer framework run on real systems compared to running it in simulated environments. Are (3) and (4) modifications to the previous work that needed to be performed for the real system?\n- Contributions: Although I acknowledge that the paper is the first to make Dreamer run on real robots and the open-source argument, I doubt that this paper pioneers to learn to walk in 1h and reaches previously unseen performance on the visual pick and place task. For the letter task, the work by Sergey Levine’s group presented immense results over the past years. The results in the paper are indeed impressive, but I feel the contributions are not formulated carefully. One of the biggest contributions I see is that all tasks have been solved by a single algorithm and constant hyperparameters, which is not mentioned in this list. It would be nice if the authors would clarify if the hyperparameters of behavior learning part (Figure 3 (b)) are also the same for each task. \n- The paper claims to “approach human performance” in the visual pick and place task, which implies that the robot is better at picking the objects than a human. However, in Section 3 it is mentioned that the robot is teleoperated by a human, which is a much easier baseline to beat. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper reports impressive results, but seems not to include any algorithmic advances. Moreover, the paper seems to partially overstate the performance: (i) the argument to match human performance in the pick and place task leaves room for misinterpretations and (ii) the results in the Sphero Navigation task only match the performance of the baseline.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "It would be interesting to understand the contributions of this paper better. Why exactly are the empirical results new, since similar performances have been reported before? Please also comment on the weaknesses listed before. If addressed appropriately, I'm happy to increase the score."}}, {"review_id": "-KyyFYFdvO5", "reviewer": "Reviewer_VhL5", "summary": "The paper evaluates the Dreamer algorithm by training directly on the hardware on 4 tasks.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- real world training on tasks that require use of multiple sensors.\n- Dreamer manages to learn the pick and place task in the sparse reward setting\n- Impressive recovering result for the quadruped robot\n\nCons:\n\nOne concern is the missing details for the experiments, as it is an application paper (no new method is proposed).\nFor instance, it is said that a low-pass filter was used but there is no information about the cut-off frequency (and how it was chosen).\nCould this low-pass filter be avoided as in [2] (where training is also directly done on hardware)?\nWas a history given as input, or how was the Markovian assumption not violated (see [1, 2])?\n\nDetails are also missing regarding how was the success measured, both for picking and Sphero navigation task. For instance, for the sphero task, it seems that the system can compute the distance to the target. If so, then learning probably doesn't make sense for that task.\n\nOne other concern is that the picking and sphero task are not visually challenging. Objects are easy to distinguish (good contrast with the background) and are not cluttered. The main challenge for the picking task comes from the sparse reward.\n\nThe difference between the two picking tasks is unclear. There are on two different robots, but both uses vision and proprioception and both are using objects that are relatively soft.\n\nThe recovery strategy learned is good, but the walking gait is far from something desirable (for instance, the knees touch the ground while walking), what happens if the agent is trained longer?\n\nThe baselines for the picking tasks seem unfair. One should at least provided a pre-trained (frozen) encoder to Rainbow as in [3] (where they also learn directly on hardware). For the sphero task, despite being model-free, the well-chosen baseline is competitive with the model-based approach.\n\nSeveral times the paper claims to be close to \"human-level performance\", but this claim is misleading. It corresponds to a human operator that had access to the robot for 20 minutes.\n\nDreamer is meant to be data-efficient, but it sill takes 9h for the picking task...\n\n\n## Remarks/Questions\n\n- typo l224 \"the the\"\n- the following references that train RL agent directly on real hardware seem to be missing (even from the appendix) and are probably relevant for that paper: [1, 2, 3]\n- for future work, authors should consider TQC [4] (SAC with distributional critic) as model-free baseline for continuous control\n- I would not spend too much time on presenting the Dreamer method and leave more space for the experiments/discussion/related work\n\n\n[1] Learning to Walk via Deep Reinforcement Learning (RSS19)\n\n[2] Smooth Exploration for Robotic Reinforcement Learning (CoRL21)\n\n[3] Learning to Drive in a Day (ICRA19)\n\n[4] Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (ICML20)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would currently recommend the paper for rejection, more experimental details are needed.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- see issues raised in the \"weakness\" section\n- it is said that the \"infrastructure will be released\" but no link is provided nor is it in the supplementary material\n- for the sphero navigation: are observation stacked?"}}]

Author rebuttal:

Decision: Accept (poster)