Learning Generalizable Dexterous Manipulation from Human Grasp Affordance
Authors: Yueh-Hua Wu
Jiashun Wang
Xiaolong Wang
Venue: CoRL 2022 Poster (2022)
Keywords: Generalized policy learning
Dexterous manipulation
Affordance model
Reinforcement learning
Imitation learning

Abstract: Dexterous manipulation with a multi-finger hand is one of the most challenging problems in robotics. While recent progress in imitation learning has largely improved the sample efficiency compared to Reinforcement Learning, the learned policy can hardly generalize to manipulate novel objects, given limited expert demonstrations. In this paper, we propose to learn dexterous manipulation using large-scale demonstrations with diverse 3D objects in a category, which are generated from a human grasp affordance model. This generalizes the policy to novel object instances within the same category. To train the policy, we propose a novel imitation learning objective jointly with a geometric representation learning objective using our demonstrations. By experimenting with relocating diverse objects in simulation, we show that our approach outperforms baselines with a large margin when manipulating novel objects. We also ablate the importance of 3D object representation learning for manipulation. We include videos and code on the project website: https://kristery.github.io/ILAD/ .


Reviews:
[{"review_id": "iW7WO028UCC", "reviewer": "Reviewer_nXCy", "summary": "This paper mainly studies the problem of dexterous grasping. It first generates grasp demonstrations from a human grasp affordance model. Using the generated demonstrations it conduct demonstration-augmented RL to learn dexterous relocation. The proposed method also encode the point cloud observation with a PointNet and pre-train and finetune the PointNet with behavior cloning tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.  The proposed data generation pipeline (both grasping pose and approaching trajectories) is useful.\n2. Pre-training and finetuning visual encoder with behavior cloning task is an interesting idea.\n3. The performance of the proposed method is very impressive, surpassing the baselines with a significant margin.\n\nWeakness:\n1. The proposed task, relocation, consists of three phases: approaching, grasping, and lifting. The proposed method for demonstration generation seems to already handle the first two phases. For the last phase of lifting and placing it in a target pose, I think it can be implemented with some simple hard code. From the videos, I can see the proposed method mostly translates the object to the desired location and does not handle the orientation properly, so probably it's an even simpler task. The authors should include a baseline with the demonstration generation method combined with a hard-coded lifting policy and evaluate the result.\n2. The benefit of pre-training is not demonstrated in the experiment results. The baseline with joint learning but without pre-training, the PointNet should be included. It would be stronger if the author could compare it with other point cloud network pre-training methods like PointContrast.\n\nMinors:\n1. Fig 3 is a little confusing: \n  a. the arrangement of the legend of the figure looks like robot state <--> share, object information goes into every T epoch and point cloud embedding goes into every epoch. Leave more margin between the columns.\n  b. it looks like the demonstration is the input to the MLP. Adding a line between demonstration and the output actions to show demonstration is also the supervision would make it clearer.\n2. It would be great if the authors could try if the proposed method can generalize to objects of different categories other than the training ones.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'm actually voting for borderline. The results in the simulation are pretty impressive, but I don’t think the argument stands on its ground if we can get the same result with the method that generates the demonstrations. If the authors can address my issues, I'm very likely to change my opinion.\n\n-----\n\nPost-rebuttal:\n\nThe author addressed my major issue and demonstrate the necessity of learning in the problem. So I change my recommendation from WR to WA.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "1. Include a baseline with the demonstration generation method and show the learned method can outperform it. Also analyze the reason behind.\n2. Include a baseline without PointNet pre-training.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "JS8AF0gcO31", "reviewer": "Reviewer_p8H4", "summary": "This paper focuses on the problem of grasping categories of objects with dexterous hands. The paper is built on top of DAPG. The main contribution of the paper is to add a new term to DAPG so that it focuses more on the data that it does not work well. Cause without that term, the policy learns to ignore the difficult objects and focus on the objects that can be easily manipulated. The architecture of the model is pointnet++ for processing the point cloud along with object pose and joint states. All the experiments are done in simulation and the generalization is only considered for different instances within the same category.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem of learning from human demonstration for dexterous manipulation is important\n\nWeaknesses:\n- Very limited novelty: Apart from the second term of eq. (2), there is nothing novel about the method.\n- There are no real world experiments.\n- It seems like the policies are trained for each category separately. So different policies are needed for different categories. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Limited novelty and no real robot experiments are my biggest concern for this paper.\n\nPost Rebuttal: I appreciate the authors for the new results on all the categories and held out test categories. I improved my rating.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses above.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "DiZSCZeguaMc", "reviewer": "Reviewer_WeMh", "summary": "This paper presents a pipeline for a robot hand to grasp onto different categories of objects and move the object to a target pose. The target grasp of the hand on the object is generated by an affordance model GraspCVAE, which was then reached using CEM for planning. The next step is to perform imitation learning based on DAPG using demonstrations generated from the previous planning algorithm. The policy encourages learning from trajectories that is hard to reproduce by dynamically assigning a normalized likelihood weight during training. A neural network is trained to predict the advantage function that is applied to the partial demonstration. PointNet is used to encode object shape, which is jointly trained with the imitation learning MLP.  The proposed method was compared with multiple baselines, and the ablation study shows the advantage of joint learning. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n* Extensive experiments to demonstrate the effectiveness of the proposed method in simulation. \n\n**Weaknesses:**\n* No real world demonstration. \n* Clarity or the quality of writing needs improvement. (see “Issues” section for details)\n* The scope claimed is not sufficiently backed by what’s demonstrated in the paper. For example the word “manipulation” is not in any of the technical sections. (see “Issues” section for details)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The main contribution of the paper is to propose a method to generate actions for grasping object in certain grasp pose and manipulating the grasped object to a target pose. The generated method has a high success rate during the experiment and has demonstrated certain generalizability to objects within the same category. The reviewer believes this work provides a novel approach to object grasping (fetching) than dexterous manipulation as claimed in the title and introduction. Nevertheless, it does showcase algorithmic innovations and demonstrated certain level of success. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "* Figure 4: “JT” in the legends and “JL” in the caption.\n* Line 155: $\\\\rho_{\\\\pi_{\\\\theta}}$ instead of $D_{\\\\pi_{\\\\theta}}$ in the equation above. \n* The equation between line 151 and 152 is missing the number. \n* Figure 3. Since “Behavior cloning” is a type of “Imitation learning”, the labeling can be confusing and makes the figure difficult to understand. \n* The user study on naturalness demonstrated the benefit of affordance model (a prior work used in this pipeline) rather than that of the proposed method, which does not seem to be very useful in this paper.\n* It’s a bit of stretch to use the phrase “dexterous manipulation” in the paper. The action generated by the hands are mostly two stages: (1) grasp onto the object, and (2) move the object to a target pose. The proposed policy is more or less learning a grasp pose of the hand. While the hand does manipulate the object to a final pose. During this stage there’s no relative motion between the object and the fingers, and such manipulation can be achieved through simple inverse kinematics. \n* Please try to have a more elaborated limitation section rather than just focusing on explaining why there is no real-world experiment. \n"}}]

Author rebuttal:

Decision: Accept (poster)