Cross-Domain Transfer via Semantic Skill Imitation
Authors: Karl Pertsch
Ruta Desai
Vikash Kumar
Franziska Meier
Joseph J Lim
Dhruv Batra
Akshara Rai
Venue: CoRL 2022 Poster (2022)
Keywords: Reinforcement Learning
Imitation
Transfer Learning

Abstract: We propose an approach for semantic imitation, which uses demonstrations from a source domain, e.g. human videos, to accelerate reinforcement learning (RL) in a different target domain, e.g. a robotic manipulator in a simulated kitchen. Instead of imitating low-level actions like joint velocities, our approach imitates the sequence of demonstrated semantic skills like "opening the microwave" or "turning on the stove". This allows us to transfer demonstrations across environments (e.g. real-world to simulated kitchen) and agent embodiments (e.g. bimanual human demonstration to robotic arm). 
We evaluate on three challenging cross-domain learning problems and match the performance of demonstration-accelerated RL approaches that require in-domain demonstrations. In a simulated kitchen environment, our approach learns long-horizon robot manipulation tasks, using less than 3 minutes of human video demonstrations from a real-world kitchen. This enables scaling robot learning via the reuse of demonstrations, e.g. collected as human videos, for learning in any number of target domains.


Reviews:
[{"review_id": "hOXF5DrbsPT", "reviewer": "Reviewer_YQ7o", "summary": "This paper proposed a skill transfer method which exploits learned semantic correspondence in order to transfer skills from source domains to novel target domains. Although the means of executing actions to achieve skills might be very different in terms of low level control, the proposed method can transfer the skill from source domain to the target domain with high performance. The method was verified in two simulation environments and with real human video demonstration data. The method was compared against SOTA methods that do not use semantic information and shown to exhibit superior performance.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Especially, generalization using real human video demonstrations and automatic extraction of skill labels are very impressive.\n- The paper is well-written but a heavy read. A lot of details were only provided in the Appendix, which makes reading the paper challenging. Additionally, dropping the notation for the source/target domain reduces readability.\n- The assumption of predefined skill annotations is a very strong assumption. Yet, if the demonstrations/dataset is from robot motions, it is acceptable because these movements are programmed/learned for some purpose, which might be used as the label of the corresponding segment. Additionally, the authors convincingly showed the use of action recognition models in automatic labeling. What is left is the assumption of having \"the same set of semantic skills\" in between different (source and target) domains. The authors might discuss what happens if the semantic skill set is not exactly the same but similar.\n- Related to the previous comment, in order to match the semantic skills, one way might be to focus on the changes the robot creates in its environment. Even though the changes are not exactly the same with different embodiments and arrangements, still the relative changes created on objects by the end effector of the robot would be similar across different domains/environments.\n- The paper was shown to have superior performance against baselines that do not use semantic information. I am curious how it would perform if such information is included in other skill transfer methods, for example, in ACNMP which is a demo+RL method that can efficiently transfer skills between different embodiments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The approach is novel, the method is sound, and the methods are verified rigorously in simulation experiments and real data. The paper works on a well-motivated problem. It opens a possibility for a scale-up if we can indeed extract information from demonstrations available in other domains.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "In Eq. 4. I did not understand how aligning p_t(k | s^T) and p_s(k | s^S) helps transfer from source to target since s^T and s^S are not paired. In a sense, we align these two distributions but not necessarily align if they output the same k for semantically same s^T and s^S. I think Eq. 5 partially helps for this issue by also considering multiple timesteps, but I am not sure if this is generic. A clarification helps."}}, {"review_id": "dZM6ah1BYVp", "reviewer": "Reviewer_UjLs", "summary": "This paper presents an approach for imitation across cross-domain based on semantic skills.\nThe approach can utilize demonstrations across substantially differing domains to accelerate the learning of long-horizon tasks.  This approach can be used to leverage human video demonstrations for teaching new tasks to the robot  while being robust to noise in training and demonstration data.\nThe approach is tested across two simulated maze and kitchen environments as well as from real world videos of human to a simulated robot. The results show that the approach can accelerate learning from cross domain demonstrations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is very well written and organized. The motivation and problem setup is very clear.\nThe approach presents a promising way to use large scale video datasets for behavior learning in robotics.\n\nWeakness:\nThe following points could be addressed:\nWhat are the computational challenge of learning cross domain skill? e.g. Time for  episodes?\nHow would it transfer to real robot manipulation from a human video demonstration?\n\nUpdate:\nThe complexity of algorithm is well addressed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper has interesting ideas for scalable and generalizable robot learning with relevant experimental results.\nI would recommend accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Analysis of time complexity  and insights about human video to real world manipulation.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Xh7jYphpt0-", "reviewer": "Reviewer_95b2", "summary": "This paper focuses on imitation learning of high-level semantic skills where the source and target domain are different. The paper propose \"STAR\", a model dealing with cross-domain knowledge of long-horizon skills, based on an assumption that the low-level skills are already learned from the given dataset in both source and target domains. The work outperforms several baseline methods in the aspect of learning speed, result metrics, and no need for demonstrations in the target domain.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The motivation and problems to be solved are well explained.\n- Good presentation of related works and their limitations.\n- The assumptions in the problem formulation are properly made and do not violate common sense.\n- Table 1 in the Appendix helps understand different models and their training objectives.\n- The experiment results look very interesting and promising, especially the last experiment on learning semantic skills from human demonstrations in a video. The current model solved some of the drawbacks of the related works. \n- The limitation section is well written, letting me know that the authors share the same concern as mine, i.e. the existence of pre-defined skills and corresponding data in all the domains.\n\nCons:\n- The approach and model in section 4.1 are not well formulated. From line 113 to line 139, I could not follow the techniques and causality the authors wanted to explain.\n- The role of the latent representation z in the current model is not clear. In line 129, the author gives an example saying \"z capture the robot hand orientation\", but z should have some other meaning, such as a high-level execution description of the task. \n- Why a tuple (k,z) can represent a sequence of H steps?\n- Model and math lack details and the explanation is not sufficient to help understand the model, e.g., for Eq. 1, I do not understand how did the authors reach this learning objective, and the content from line 135 to line 139 cannot help me understand it, neither the reasons of \"capturing the discrete aspects of manipulation skills\". \n- The model formulation since Section 4.2 becomes easier to understand. However, the entire model seems very complicated. It has a lot of submodules that need rich experience to get trained, which decreases my confidence in reproducing the result by other researchers and allowing the people beyond the authors' group to extend this work. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I feel very difficult to determine if this paper should be recommended or not. Thus, I would like to hear other reviewers' opinions. My difficulties come from: (1) I am not familiar with the related works. (2) Most of the presentation of the paper seems correct and the experiments look promising, but the entire model is very complicated. In my opinion, it will be quite hard to be used or extended by other researchers. So I do not feel confident to evaluate the contribution and influence of this paper. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I can capture the main idea and the majority of the model formation of this paper. Due to the lack of knowledge of the related works, however, I cannot easily judge the correctness of the technical details. \nBut I think the content between line 127 and line 139 can be improved, as I got confused when I read it. \n\n- Give more details to help the understand Eq. 1 and the reasons to support the claims from line 136 to line 139. If the proofs are from related works, please explicitly pointed out where (paper, section, eq, etc.) can I find them.\n\n- Line 129, gives a more proper description of the latent representation z instead of \"z captures the robot hand orientation\", which may also explain why \"A single tuple (k, z) represents a sequence of H steps\"\n\n- Why can this inference network help learn the low-level policy network?\n\nOther question:\n- What is the difference between the proposed method and a method which is predicting the order of executing skills? E.g. in Figure 4 middle, given some demonstrations, the prediction result is \"2, 3, 1, 4\", and then the robot just executes the basic skills following the order \"skill2\"->\"skill3\"->\"skill1\"->\"skill4\", as the low-level skills are already learned. \n\nTypo:\n- Table 1 in Appendix, the training obj. should be Eq(1), instead of Eq.(2)."}}, {"review_id": "7Lmi4XWyCTE", "reviewer": "Reviewer_8M4B", "summary": "This paper proposes a method to leverage demonstrations collected in a different instance of an environment, to accelerate the learning process of the same tasks in a new instace (e.g. two different kitchens). They do so by assuming the existence of different datasets, one with demonstrations of the full task at a high-level, in a different and non-accessible environment, and a task agnostic dataset collected on the current environment, where the agent can interact with the objects.\nThe general method is interesting and effective, although there are many moving parts and I wonder if it couldn't be made simpler.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stengths:\n- The paper tackles an interesting problem: bootstraping reinforcement learning with demonstrations of a long horizon tasks that come from a different instance of the same type of environment. Results show that the method can indeed accelerate the learning phase, and surpasses the performance of many baselines.\n- The experiments section is well designed and shows several ablations that help convincing the reader of the benefits of the method.\n- Using a real world video of a person solving those tasks can indeed show the potential of such methods to scale to larger datasets composed of humans solving tasks. This data is plentiful online and could possibly enable exponential growth of the capabilities of the robot.\nWeaknesses:\n- The method is convoluted and has many moving parts. There are several losses that are based on regularising the policy with a prior learned on the existing datasets. These parts could possibly be swapped for normal supervised learning, making it simpler and possibly without sacrificing performance.\n- It is unclear if some baselines fail at a high-level or low-level. In line 244 the authors specify that a baseline mostly fails because the low level policies fail, but the plan is correct. This should be better investigated, as learning low level policies should be easier, and generally not the main point of the paper.\n- The task is relatively linear in execution, and therefore, as they also show, simply learning or emulating the sequence of high level action can achieve good performance. As specified before, it is unclear if such benchmarks fail at a high-level (difficult to solve) or low-level (easy to solve). Figure 7 shows that if some skills are missing from the demonstration, these techniques indeed fail. However, I believe it would still be possible to learn a good planner with a larger dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I suggest to accept the paper in the current form: the idea is interesting, and the problem they tackle is fundamental for the robot learning community. However, I would suggest the authors to minimise the number of moving parts of their algorithm, that can dramatically hurt reproducibility, and show, possibly with more complex robot environments, the benefits of this method for high-level planning than a BC-like planner", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "As described above.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)