Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations
Authors: Chenhao Li
Marin Vlastelica
Sebastian Blaes
Jonas Frey
Felix Grimminger
Georg Martius
Venue: CoRL 2022 Oral (2022)
Keywords: Adversarial
Imitation Learning
Legged Robots

Abstract: Learning agile skills is one of the main challenges in robotics. To this end, reinforcement learning approaches have achieved impressive results. These methods require explicit task information in terms of a reward function or an expert that can be queried in simulation to provide a target control output, which limits their applicability. In this work, we propose a generative adversarial method for inferring reward functions from partial and potentially physically incompatible demonstrations for successful skill acquirement where reference or expert demonstrations are not easily accessible. Moreover, we show that by using a Wasserstein GAN formulation and transitions from demonstrations with rough and partial information as input, we are able to extract policies that are robust and capable of imitating demonstrated behaviors. Finally, the obtained skills such as a backflip are tested on an agile quadruped robot called Solo 8 and present faithful replication of hand-held human demonstrations.


Reviews:
[{"review_id": "Zwwl82-Gdyo", "reviewer": "Reviewer_2hjT", "summary": "This paper presents an imitation learning algorithm, named WASABI, that enables quadruped robots to learn from rough and partial expert demonstration using generative adversarial learning. Instead of using task-specific reward functions, as with many robotics learning approaches, the proposed approach can infer rewards from demonstration using GAN. In particular, the authors incorporate task-agnostic regularization terms and adversarial rewards to enable the robot to learn stable and agile locomotion behaviors. Moreover, this approach can learn only from partial and physically incompatible demonstrations (e.g., a human operator holds and moves the robot's trunk in a specific fashion). The authors also showed simulation and experimental results of a quadruped robot learning four challenging tasks. The papers show that WASABI outperforms the least-square GAN (LSGAN) in most of the tasks (3 out of 4) in terms of motion similarity (measured by DTW distance). ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n\n- Able to learn from limited demonstration without defining task-specific reward functions, which often need to be designed and tuned by hand.\n- Replacing LSGAN loss with Wasserstein loss improves the efficiency in discriminating the reference and the generated motions.\n- Able to prevent mode collapse using gradient penalty term in the loss function.\n- Demonstrating the performance of the proposed approach in hardware experiments\n\n**Weakness:**\n\n- The final reward function also includes a regularization reward, which seems to be specific to the robot. How much does this term contribute to the total reward?\n- The observation space does not include the position/orientation of the robot base (it only includes linear and angular velocities). Will the demonstration speed of the human operator would affect the final learned locomotion?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this work presented in the paper is well developed. In robot learning (especially when it involves hardware), the reward design or shaping is often heuristic and requires manual tuning for different tasks. WASABI, which uses generative adversarial learning, provides a principled means to design rewards for robot learning. The demonstration of hardware shows the promising potential of the proposed approach. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please explain the reasoning behind the choice of observation speed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "W3sxzIcULA", "reviewer": "Reviewer_V4vA", "summary": "Authors present a novel generative adversarial method named WASABI for inferring reward functions from partial and potentially physically incompatible demonstrations. WASABI simultaneously trains Wasserstein GAN to distinguish between real and generated observations, and policy which uses output of the discriminator as a reward. This approach allows to use partial demonstrations which don't contain full information about the state. Authors experimentally compare performance of WGAN and LSGAN for reward inference and experimentally validate the proposed method in simulation and on a quadruped robot training complex skills such as a backflip. \n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes an extension of Generative adversarial imitation learning method (GAIL) method. Wasserstein GAN formulation allows to use the discriminator output as a reward function directly only with simple normalization. This method allows to train a policy for complex behaviour from partially observed expert demonstrations. The paper is  well written and enjoyable to read. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It would be interesting to see ablation on how much expert data needed to train a discriminator ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "It seems important to mention how much expert data was used to train discriminator for each of the tasks ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Gw4DBj9A2W", "reviewer": "Reviewer_V8R4", "summary": "In this paper, the authors proposed a GAN-based framework to train agile locomotion policies using partial demonstrations. Similar to AMP (Adversarial Motion Prior) work, the authors use Wasserstein-GAN to discriminate between the expert demonstrations and the learned policies, to generate the reward signal for reinforcement learning. Different from previous works, the demonstrations in this work are not from experts (dogs for examples). Instead, only rough base movement demonstrations are provided for the learner policy to figure out dynamically feasible motions that can realize the base trajectories. The authors has demonstrated successful deployment of the learned policies on the hardware with zero-shot sim-to-real transfer. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper:\n\n1, The authors show that it is possible to only provide base trajectories as demonstrations to train imitation policies for quadruped robots. This opens many new potentials for quadruped robots, since expert demonstrations are often hard to obtain. \n\n2, There is a good ablation study and analysis on the different GAN losses (least square vs Wasserstein)\n\n3, The learned policy can transfer to the real hardware on various agile tasks including back flipping. \n\nThe weaknesses of the paper:\n\n1, the delta between AMP and this work is not quite large, especially since AMP has also been tested on hardware recently. The main differences seem to be (1) whether or not to include the joint angles as part of the discriminator observations for reward generation and (2) use of WGAN vs LSGAN. \n\n2, Not enough baselines to compare with. I think it will be great to compare AMP with the proposed method on the selected tasks. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "See the strengths and weakness section. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "Here are some additional comments, in no particular order:\n\n1, There are also other AMP extension works on hardware to refer to, such as this one: “Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions”\n\n2, Running mean and variance for the discriminator: Are they learnable parameters of a gaussian output?\n\n3, Tabel 2 bottom line, why only two columns?\n\n4, Action space for the policy is better to be included in  the main text. Also, please list the joint PD gains in the supplementary text.\n\n5, Have the authors tried to use different observation spaces for the control policy than the discriminator? For example, base linear velocity is often not needed for the control policy to function properly. And in this way the policy may transfer to the real hardware without motion capture. \n\n6, Why is there no SOLOSTANDUP task on the real robot? Does it transfer or not?\n\n7, Figure 4A and 4B are not quite clear, maybe due to the color coding. \n"}}, {"review_id": "2O56aEqCjwA", "reviewer": "Reviewer_Hona", "summary": "The paper proposed an adversarial imitation learning method to learn agile motor skills of a quadrupedal robot from partial demonstrations.\nThe main contributions of the paper are following two points:\n1. The authors introduced the Wasserstein loss to the adversarial reward learning.\n2. The authors showed objective and reward functions to acquire the agile motor behaviors from the rough and partial demonstrations.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The authors successfully demonstrated that the proposed method, Wasserstein Adversarial Behavior Imitation (WASABI), could learn the agile behaviors from the handcrafted partial state information.\n* The authors successfully showed that the agile behaviors of not only the simulated but also real quadrupedal robot could be generated with WASABI by transferring the learned policies.\n\nWeakness:\n* Although the experimental results are fascinating, the paper seems difficult to read. Especially, readers may need to read the original paper of adversarial motion priors (AMP) [Peng et. al., TOG, 2021] to understand the learning process of the adversarial reward learning and policy training. To improve this, for example, I recommend that the authors make a new section to explain the original AMP method as a preliminary in the paper.\n* The ablation study of the proposed reward function Eq. (6) is missing.\n\nMinor comments:\n* Since the backflip behavior of the real quadrupedal robot is impressive, it would be better if the authors show snapshots of the backflip on the paper like Figure 1 (left). Figure 1 (right) is hard to see.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The contribution of this paper is sufficient: strength in both originality of the approach and thoroughness of results. The experiments are performed in not only simulation but also real environments. However, the clarity of the presentation is insufficient. The paper should be self-contained.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* I suggest the authors to improve the paper structure so that, ideally, the paper becomes understandable without reading the original AMP paper.\n* I suggest the authors to perform the ablation study for the proposed reward function Eq. (6) to show that both $r^{T}$ and $r^{R}$ are important.\n"}}]

Author rebuttal:

Decision: Accept (oral)