NeuralGrasps: Learning Implicit Representations for Grasps of Multiple Robotic Hands
Authors: Ninad Khargonkar
Neil Song
Zesheng Xu
B Prabhakaran
Yu Xiang
Venue: CoRL 2022 Poster (2022)
Keywords: Robot Grasping
Neural Implicit Representations
Grasp Transfer
Grasping Contact Modeling
6D Object Pose Estimation

Abstract: We introduce a neural implicit representation for grasps of objects from multiple robotic hands. Different grasps across multiple robotic hands are encoded into a shared latent space. Each latent vector is learned to decode to the 3D shape of an object and the 3D shape of a robotic hand in a grasping pose in terms of the signed distance functions of the two 3D shapes. In addition, the distance metric in the latent space is learned to preserve the similarity between grasps across different robotic hands, where the similarity of grasps is defined according to contact regions of the robotic hands. This property enables our method to transfer grasps between different grippers including a human hand, and grasp transfer has the potential to share grasping skills between robots and enable robots to learn grasping skills from humans. Furthermore, the encoded signed distance functions of objects and grasps in our implicit representation can be used for 6D object pose estimation with grasping contact optimization from partial point clouds, which enables robotic grasping in the real world.


Reviews:
[{"review_id": "YQqlCuGQmFk", "reviewer": "Reviewer_hW79", "summary": "In this work, the authors proposed a new method that leverages 3D SDF-based implicit representation to learn an embedding for object grasps. The learned embedding can index various grasps of the same object but using different grippers, e.g. the two-finger parallel gripper, three or four-finger ones, and even the human hand, so that grasp transfers from one gripper to another or from human demonstration to robot execution are possible. Experiments are done using 7 YCB shapes and 5 different grippers. Simulated grasps are collected for training and real-world/real-robot experiments are presented.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem of grasp transfer across different robot grippers is important, novel, interesting, and challenging.\n- The core idea of learning SDF-based 3D implicit representations for the object and gripper geometry jointly is quite new and interesting.\n- The results of grasp transfers from human hand demonstrations to robot hand execution are interesting and good.\n\nWeaknesses:\n- The method learns one individual embedding for each input shape, and learns different embeddings for different shapes. Although the authors discussed this limitation and mentioned this is a future work, I think this is a quite big limitation, since it prevents the method from being used for a novel shape other than the 7 YCB objects. One needs to get the full 3D model, run the simulated grasps data collection process, and train all networks again. Firstly, it's not already possible to get a full 3D model. Secondly, if you got it, there are plenty of ways to analytically compute good grasps given a gripper, right?\n- While I think the joint SDF learning for the object and the gripper is interesting, I don't quite get the motivations of learning a latnet embedding of grasps. The embedding is supervised using a heuristic of grasp similarity using different grippers -- the contacts over the object. Then, why do we need to learn such latent embedding? Why not directly compare the explicit contact heatmaps? The problem exaggerates when I find that the paper only performs grasp retrieval. You can compare the explicit contact heatmaps to do the retrieval, right? Can the authors show that you can solve for the grasp poses instead of retrieving one given a new gripper?\n- While I agree that it's hard to define the similarity between two grasps using different grippers and I think the idea of using contact maps is reasonable, I think that many transferred grasps are quite different, for example, the grasp poses can be quite different while the contact regions are similar. I want to see more justifications for only using the contact maps as the criterion, and not using gripper pose similarity. Say, the easiest thing one can do is to compute the gripper geometry similarity? or the gripper skeleton chamfer distance?\n- I don't think the authors can claim the data contribution as a main contribution since the grasps are synthetically generated.\n- In Table 2, why \"near GT\" is similar to \"near Z\"? Also, as mentioned in Line 252, \"has similar scores\". This exaggerates my concern if learning a latent embedding is really better than directly comparing the explicit contact map similarity?\n- The experiments are conducted over only 7 shapes, which is a very small scale.\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I like the SDF part but don't understand the benefits of learning the latent embedding. Also, the method trains different individual embeddings for different shapes, considerably limiting its use case dealing with novel shapes. I have further concerns regarding the experiment scale, the grasp retrieval/transfer, and the claimed dataset contribution.\n\n-- Post rebuttal\nI've read the author's rebuttal and the other reviews. While I still remain concerned about training one embedding per shape, doing experiments over only seven shapes, and only using contact maps as the definition for grasp similarities, I do not object to accepting this paper as the paper's idea is generally interesting and novel, which is worth getting exposed to the community.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Please see the \"weakness\" points above. Major points to address are why learning a latent embedding is beneficial, can you solve for the grasp poses given a new gripper instead of doing retrieval, and can you make the system not overfitting to one shape."}}, {"review_id": "SVqafXrQ4Da", "reviewer": "Reviewer_z2Qy", "summary": "The paper proposes to use signed distance field to represent grasping behavior. The goal is to learn a representation that's transferrable across morphologies. The transfer happens during inference time by optimizing a latent representation. Awesome results are demonstrated by the authors.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The paper is very well written. It's very easy to follow the paper, and the flow of the work develops really well. Many relevant concepts are introduced in a clear manner, which will be very helpful for readers of all kinds of backgrounds.\n* The visualization are presented in a clean and informative ways. They are very helpful and demonstrate ideas clearly. I wish I can see this level of presentation in every paper.\n* The method makes a lot of sense. It captures the essence of \"how something is being grasped\" really well. Using an test time inference / optimization pipeline makes a lot of sense and follows naturally when using an implicit representation.\n\nWeakness:\n* I think it would be great if more details on practicality is provided. this involved a few aspects:\n- how fast can the transfer be performed? Is it possible to get in in real time? I think it might be challenging.\n- how does the pipeline fit into a policy training perspective? currently we are transferring grasp of a single action on a single object. Can such pipeline be extended to a grasping policy?\n* I think a few lines of related works are missing, which is in my opinion is weakest part of the paper. When introducing \"Neural Implicit Representations.\" related works, wouldn't it make sense to introduce efforts that are using INR in robotics? namely:\nline of work from UT Austin:\n- Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations, Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, Yuke Zhu, RSS, 2021  \n- ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J. Guibas, Silvio Savarese, Anima Anandkumar, Yuke Zhu, RSS, 2022\n\nline of work from MIT:\n- Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning, Danny Driess, Jung-Su Ha, Marc Toussaint, Russ Tedrake, CoRL 2021\n- 3D Neural Scene Representations for Visuomotor Control, Yunzhu Li*, Shuang Li*, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba, CoRL 2021\n\nline of work from Google:\n- VIRDO: Visio-tactile Implicit Representations of Deformable Objects., Youngsun Wi, Pete Florence, Andy Zeng, Nima Fazeli. ICRA 2022.\n- NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields. Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Tsung-Yi Lin, Alberto Rodriguez, Phillip Isola. In ICRA 2022.\n\nI really believe a wider coverage of the related work can only benefit the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Awesome paper! Definitely should be an acceptance in my opinion. However, if the authors can do a better job with related work coverage, that would make the paper even better. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See my comment on related works.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Hd3w9zFoFm", "reviewer": "Reviewer_bHvy", "summary": "This paper proposes an algorithm to learn implicit neural grasp representations and presents some applications of these representations. The proposed algorithm builds on DeepSDF [6]. DeepSDF jointly optimizes model parameters and per-shape latent codes over a dataset of 3D shapes. A neural network function parameterized by the learnt model parameters predicts the SDF value from input 3D location and latent code.\n\nHere this idea is extended to grasps by making the function output two SDF values - one for the object and one for the gripper in the grasping pose. Training data is automatically generated using GraspIt!. In addition to optimizing the per-grasp latent codes to minimize SDF prediction error, the latent codes are also encouraged with a triplet loss to cluster grasps with similar object contact patterns, regardless of the grasping gripper geometry.\n\nExperiments evaluate cross-gripper grasp retrieval, object pose estimation for grasping, and grasp transfer from human to robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The basic grasp representation idea presented here has many downstream possibilities because various grasp-level distance metrics can be imposed on this compact representation (the latent code). This paper demonstrates one possibility through an object contact map similarity metric. But one can imagine others e.g. temporal encoding for skills, gripper contact map similarity, etc.\n- The paper is mostly well written and understandable. Figures are helpful.\n- Real robot experiments show the practical application of the proposed algorithm.\n\n**Weaknesses**\n\nConceptual: Inadequate motivation discussion for the proposed training procedure and choice of a single latent code in the paper.\n- The SDF accuracy and cross-gripper contact map similarity objectives conflict with each other in the current formulation, because they share the same latent code. In the limit, training manages to drive the triplet loss to zero, that means two grasps with different grippers but similar object contact maps would have the same latent codes, and hence the same SDF predictions i.e. at least one gripper will be reconstructed incorrectly.\n- This is seen in the significant inaccuracies of the inferred contact maps in Figure 3, which implies large gripper reconstruction inaccuracy. Table 1 chamfer distance numbers corroborate this.\n- This raises the question, why the two objectives need to be 1) solved with the same latent code, and 2) optimized simultaneously. For example, how about training $\\theta$ and latent codes firstly with just the SDF accuracy objective i.e. same as DeepSDF. Then a _separate_ MLP can be trained _separately_ to embed the DeepSDF latent codes into a different latent space with the proposed triplet loss. This would resolve the conflicting objectives mentioned above.\n\nExperimental:\n- Grasp transfer between various grippers is mentioned as a strong motivation (L29), but the real robot experiment shows only one simple top-down grasp.\n\nClarification:\n- Section 5.2, \"grasping with object pose estimation\": Eq. 5 requires the latent code, it is not clear which latent code is used to optimize the object pose.\n- Figure 3: Please show the contact maps from multiple viewpoints.\n- Table 2: how the grasp success rate defined?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending to accept this paper because of the clear research question and good execution. The idea of imposing distance metrics on a neural grasp representation can be developed further for multiple applications.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Please address the \"clarification\" questions in the \"weaknesses\" section. Addressing the conceptual question by counter argument or by implementing and evaluating the the suggested two-stage learning process would be even better, but I don't think is necessary for acceptance."}}]

Author rebuttal:

Decision: Accept (poster)