RoboTube: Learning Household Manipulation from Human Videos with Simulated Twin Environments
Authors: Haoyu Xiong
Haoyuan Fu
Jieyi Zhang
Chen Bao
Qiang Zhang
Yongxi Huang
Wenqiang Xu
Animesh Garg
Cewu Lu
Venue: CoRL 2022 Oral (2022)
Keywords: Learning from Videos
Video Demonstration Dataset
Real2Sim
Self-supervised Reward Learning
Robotic Simulation Benchmark

Abstract: We aim to build a useful, reproducible, democratized benchmark for learning household robotic manipulation from human videos. To realize this goal, a diverse, high-quality human video dataset curated specifically for robots is desired. To evaluate the learning progress, a simulated twin environment that resembles the appearance and the dynamics of the physical world would help roboticists and AI researchers validate their algorithms convincingly and efficiently before testing on a real robot. Hence, we present RoboTube, a human video dataset, and its digital twins for learning various robotic manipulation tasks. RoboTube video dataset contains 5{,}000 video demonstrations recorded with multi-view RGB-D cameras of human-performing everyday household tasks including manipulation of rigid objects, articulated objects, deformable objects, and bimanual manipulation. RT-sim, as the simulated twin environments, consists of 3D scanned, photo-realistic objects, minimizing the visual domain gap between the physical world and the simulated environment. After extensively benchmarking existing methods in the field of robot learning from videos, the empirical results suggest that knowledge and models learned from the RoboTube video dataset can be deployed, benchmarked, and reproduced in RT-sim and be transferred to a real robot. We hope RoboTube can lower the barrier to robotics research for beginners while facilitating reproducible research in the community. More experiments and videos can be found in the supplementary materials and on the website: https://sites.google.com/view/robotube


Reviews:
[{"review_id": "uAs-_SUmJca", "reviewer": "Reviewer_fnSc", "summary": "This paper presents a suitable dataset (RoboTube) for robot learning from videos. The dataset contains 5,000 video demonstrations recorded  with multi-view RGB-D cameras of human performing everyday household tasks. In contrast to existing video dataset for robot learning, RoboTube includes complex manipulation of different categories of objects such as rigid objects, articulated objects and deformable objects. The larger database of objects in the dataset promises to support a large range of diverse manipulation behaviours.\nRobotube dataset is used to  evaluate a robot learning pipeline from human videos. To evaluate the learning process and benchmark different learning algorithm, a simulated twin environment  (RT-Sim) is developed. The simulator uses 3D scanned, photo realistic objects for minimizing the visual domain gap  between real and simulated environments.\nThe efficacy of the pipeline through sim to real transfer is validated on real experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper addresses an important problem in robotics. The area and significance of research is very relevant to the robotics community. I agree that a suitable dataset for robot learning from human videos is missing and is needed.\n\nThe dataset has larger variety of object models and bigger database of videos. The simulated twin environment RT-Sim is able to compare different methods and also transfer the learning to real world.\n\nWeakness:\n\nThe significance of the dataset with respect to the others in the state-of-the-art is not very clear. The comparison of features of the dataset with other existing ones on Table 1 does show that the dataset has larger database with diverse object models. However, the deficiencies mentioned is Line 32-50 is not well supported. \nI do not agree that the manipulation tasks in RoboTube are long spectrum as compared to other dataset which is mentioned as short spectrum.\nAlso the data diversity and relevance of the RoboTube dataset is not measured using suitable metrics.\nThere already exists simulation environments like Robosuite and RLBench. I could not appreciate the development of  a new environment RT-Sim for benchmarking. I think either its significance is not well measured.\nThe link for the dataset pages is not updated. It would have been better to inspect the dataset.\n\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the topic is interesting and a robotics focus dataset is developed for robot learning from videos.\nHowever, I missed some comparative assessment in the paper and therfore its significance with respect to the state-of-the-art is not very clear.\nI would want to see my concerns address for better recommendation.\n\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Show the significance of  the proposed dataset with respect to others in Table 1 as how it addresses the deficiences mentioned in line 32-50.\nUpdate: Satisfactory after revision\n\nAlso the significance of RT-Sim with respect to other simulators e.g., Robosuite, RLBench ... \n"}}, {"review_id": "tFmMuWpRAMG", "reviewer": "Reviewer_ZLYq", "summary": "The work presents a manipulation dataset consisting of two major parts. 1. Human demonstrated tasks, recorded via egocentric and stand mounted RGB cameras 2. A digital simulation of the tasks, called RT-sim. The dataset is intended for RL algorithms to achieve human-like manipulation with robotic systems. The paper demonstrates the application of some initial RL algorithms to the dataset. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is on the whole well written and gives compelling arguments for the benefits of this dataset over existing datasets. There is some slight repetition here. \n\nThe tasks included in the dataset and interesting, though after watching the video supplement I was expecting there to be more tasks or object variations.\n\nA strength is that the paper includes application of RL algorithms to the dataset, rather than simply saying that this is the intention of the work.\n\nA major weakness is my opinion is the use of arbitrary objects. If the authors had used the YCB object set then users of the dataset would have been able to aquire physical objects in addition to digital models. Clearly the YCB object set would not address all of the tasks mentioned here (e.g. there is no drawer or cupboard), but it would have been nice to be able to use the same mug for testing on physical robots. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend that this work be accepted for publication. I think the dataset will be a useful resource and the authors provide nice examples on its use. It is a pity that reproducable physical objects were not used. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Minor weaknesses / typos\n- Details of the robot used in section 5.3 would be welcome.  \n- Please expand the various acryonyms when first used. This was done for GC in section 5.1, but not AVID, LbW, TCN etc. This applies to other sections also. It is likely that readers will not be familiar with all acroynms and a general idea can be gained from such expansions without breaking the flow of reading. \n- 'concert live' should be 'live concert'\n- In the Mug pouring task description the authors write that there are '20 tiny balls'. Tiny is not very technical, and I would personally imagine something tiny to be around 1mm or less in diameter. Please remove the word tiny and include the diameter of the balls.  "}}, {"review_id": "i8C41JKKTPu", "reviewer": "Reviewer_wYij", "summary": "This paper presents a human household manipulation benchmark with associated digital twins and a focus on robot learning. The 5000 video clips cover tasks from articulated objects to deformable objects with multi-views and depth inputs, and the visual fidelity of the simulation is high and potentially useful for sim-to-real. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\nThe contribution of a large / diverse egocentric human manipulation video dataset has critical importance and good timing for the research community. Humans can provide many demonstrations for manipulation tasks and is a good way to scale up offline data.\n\nCompared to existing benchmarks, depth and multi-view information are provided, which is important for 3D vision and spatial reasoning. Negative samples and simulated twins also open more research topics such as reinforcement learning in simulations, and avoid requiring excessive real-world experiments or only vision tasks for using this video benchmark. \n\nThe benchmark has a focus on robotics by design such as cluttered / clean scenes as well as simulating different robot arms and diverse poses, and its tasks are much more interesting than pick-and-place while not being too complicated as the full process of cooking a dish. \n\nWeakness:\n\nThere are several arguments made in the paper but I think it would be great to have more convincing evidence.\n\nThe dynamics of simulations are claimed to resemble the real world but I can’t seem to find details for some system identification procedures during the benchmark curation. For instance, do all the shapes share similar physics parameters such as mass and frictions?\nSimilarly, to map the human arms to robot arms as well as the full scene, it seems some retargeting functions are needed and the human arms and some object pose / shape labels are needed, are these information also contained in the benchmarks and how accurate they are for cluttered scenes? It seems to me that Unity might have good content creation ability and rendering performance, but its physics engine might not have the enough fidelity to simulation contact-rich tasks in robotics.\n\nA philosophical question. I find it interesting that the paper is taking a “real to sim to real” approach to tackle robotic tasks. I am curious about authors’ thoughts on the difficulty of building a world model compared to solving tasks for robotics. In other words, I agree that a simulator allows for benchmarking and democratizing researching progress, but are sim-to-real or real-to-sim problems necessary in the long run?\nAnother key usage of human demonstration datasets such as Ego4D is for representation pretraining, are there supporting results for better representation learned from a more focused robotic benchmark? Are there analysis of the multi-modality of the demonstrations?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a novel and very relevant simulation and video benchmark for robotics. The designs such as multi-view, depth, task complexity, and cluttered scenes are important yet missing from benchmarks in other communities. There are still several weaknesses regarding experiments, missing details, and limitation discussions, and would love to hear about the response from the authors. Overall I recommend acceptance of this paper. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The usage of unity simulator certainly provides better rendering quality, but the physics fidelity, especially contacts, are important for robotics and might not be as accurate. Moreover, I wonder about the speed of the simulation and the ease of usage for general robotic researchers.\n\nI am not sure if I understand figure 5 correctly. Is the optimal reward function continuous and smooth along the time axis, and there should be a consistency for both real and sim? If so, it seems that none of the approaches are doing very well and no consistent win for either method? For instance, TCN seems to completely fail for simple tasks as cabinet, but works for complex task such as cloth folding. It’s interesting to think that in this case, simulation is used for test generalization and real-world data is used as training. \n\nI also have confusions for table 2. It looks like that the designed environment reward is way much better than the learned rewards? I think this is reasonable and suggest more experiments on the representation or policy learning rather than inverse RL. \n\nThe details for the real-world experiments are not clear to me. What is the policy trained on, is it simulator with a learned reward? If the learned reward is good and the task is not that complicated, I am curious how well a simple MPC shooting method would work as a baseline. \n"}}]

Author rebuttal:

Decision: Accept (oral)