Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes to use signed distance field to represent grasping behavior. The goal is to learn a representation that's transferrable across morphologies. The transfer happens during inference time by optimizing a latent representation. Awesome results are demonstrated by the authors.
strengths_and_weaknesses: Strength:
* The paper is very well written. It's very easy to follow the paper, and the flow of the work develops really well. Many relevant concepts are introduced in a clear manner, which will be very helpful for readers of all kinds of backgrounds.
* The visualization are presented in a clean and informative ways. They are very helpful and demonstrate ideas clearly. I wish I can see this level of presentation in every paper.
* The method makes a lot of sense. It captures the essence of "how something is being grasped" really well. Using an test time inference / optimization pipeline makes a lot of sense and follows naturally when using an implicit representation.

Weakness:
* I think it would be great if more details on practicality is provided. this involved a few aspects:
- how fast can the transfer be performed? Is it possible to get in in real time? I think it might be challenging.
- how does the pipeline fit into a policy training perspective? currently we are transferring grasp of a single action on a single object. Can such pipeline be extended to a grasping policy?
* I think a few lines of related works are missing, which is in my opinion is weakest part of the paper. When introducing "Neural Implicit Representations." related works, wouldn't it make sense to introduce efforts that are using INR in robotics? namely:
line of work from UT Austin:
- Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations, Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, Yuke Zhu, RSS, 2021  
- ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation Bokui Shen, Zhenyu Jiang, Christopher Choy, Leonidas J. Guibas, Silvio Savarese, Anima Anandkumar, Yuke Zhu, RSS, 2022

line of work from MIT:
- Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning, Danny Driess, Jung-Su Ha, Marc Toussaint, Russ Tedrake, CoRL 2021
- 3D Neural Scene Representations for Visuomotor Control, Yunzhu Li*, Shuang Li*, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba, CoRL 2021

line of work from Google:
- VIRDO: Visio-tactile Implicit Representations of Deformable Objects., Youngsun Wi, Pete Florence, Andy Zeng, Nima Fazeli. ICRA 2022.
- NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields. Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Tsung-Yi Lin, Alberto Rodriguez, Phillip Isola. In ICRA 2022.

I really believe a wider coverage of the related work can only benefit the paper. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Awesome paper! Definitely should be an acceptance in my opinion. However, if the authors can do a better job with related work coverage, that would make the paper even better. 
issues: See my comment on related works.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this work, the authors proposed a new method that leverages 3D SDF-based implicit representation to learn an embedding for object grasps. The learned embedding can index various grasps of the same object but using different grippers, e.g. the two-finger parallel gripper, three or four-finger ones, and even the human hand, so that grasp transfers from one gripper to another or from human demonstration to robot execution are possible. Experiments are done using 7 YCB shapes and 5 different grippers. Simulated grasps are collected for training and real-world/real-robot experiments are presented.
strengths_and_weaknesses: Strengths:
- The problem of grasp transfer across different robot grippers is important, novel, interesting, and challenging.
- The core idea of learning SDF-based 3D implicit representations for the object and gripper geometry jointly is quite new and interesting.
- The results of grasp transfers from human hand demonstrations to robot hand execution are interesting and good.

Weaknesses:
- The method learns one individual embedding for each input shape, and learns different embeddings for different shapes. Although the authors discussed this limitation and mentioned this is a future work, I think this is a quite big limitation, since it prevents the method from being used for a novel shape other than the 7 YCB objects. One needs to get the full 3D model, run the simulated grasps data collection process, and train all networks again. Firstly, it's not already possible to get a full 3D model. Secondly, if you got it, there are plenty of ways to analytically compute good grasps given a gripper, right?
- While I think the joint SDF learning for the object and the gripper is interesting, I don't quite get the motivations of learning a latnet embedding of grasps. The embedding is supervised using a heuristic of grasp similarity using different grippers -- the contacts over the object. Then, why do we need to learn such latent embedding? Why not directly compare the explicit contact heatmaps? The problem exaggerates when I find that the paper only performs grasp retrieval. You can compare the explicit contact heatmaps to do the retrieval, right? Can the authors show that you can solve for the grasp poses instead of retrieving one given a new gripper?
- While I agree that it's hard to define the similarity between two grasps using different grippers and I think the idea of using contact maps is reasonable, I think that many transferred grasps are quite different, for example, the grasp poses can be quite different while the contact regions are similar. I want to see more justifications for only using the contact maps as the criterion, and not using gripper pose similarity. Say, the easiest thing one can do is to compute the gripper geometry similarity? or the gripper skeleton chamfer distance?
- I don't think the authors can claim the data contribution as a main contribution since the grasps are synthetically generated.
- In Table 2, why "near GT" is similar to "near Z"? Also, as mentioned in Line 252, "has similar scores". This exaggerates my concern if learning a latent embedding is really better than directly comparing the explicit contact map similarity?
- The experiments are conducted over only 7 shapes, which is a very small scale.
 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I like the SDF part but don't understand the benefits of learning the latent embedding. Also, the method trains different individual embeddings for different shapes, considerably limiting its use case dealing with novel shapes. I have further concerns regarding the experiment scale, the grasp retrieval/transfer, and the claimed dataset contribution.

-- Post rebuttal
I've read the author's rebuttal and the other reviews. While I still remain concerned about training one embedding per shape, doing experiments over only seven shapes, and only using contact maps as the definition for grasp similarities, I do not object to accepting this paper as the paper's idea is generally interesting and novel, which is worth getting exposed to the community.
issues: Please see the "weakness" points above. Major points to address are why learning a latent embedding is beneficial, can you solve for the grasp poses given a new gripper instead of doing retrieval, and can you make the system not overfitting to one shape.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an algorithm to learn implicit neural grasp representations and presents some applications of these representations. The proposed algorithm builds on DeepSDF [6]. DeepSDF jointly optimizes model parameters and per-shape latent codes over a dataset of 3D shapes. A neural network function parameterized by the learnt model parameters predicts the SDF value from input 3D location and latent code.

Here this idea is extended to grasps by making the function output two SDF values - one for the object and one for the gripper in the grasping pose. Training data is automatically generated using GraspIt!. In addition to optimizing the per-grasp latent codes to minimize SDF prediction error, the latent codes are also encouraged with a triplet loss to cluster grasps with similar object contact patterns, regardless of the grasping gripper geometry.

Experiments evaluate cross-gripper grasp retrieval, object pose estimation for grasping, and grasp transfer from human to robot.
strengths_and_weaknesses: **Strengths**
- The basic grasp representation idea presented here has many downstream possibilities because various grasp-level distance metrics can be imposed on this compact representation (the latent code). This paper demonstrates one possibility through an object contact map similarity metric. But one can imagine others e.g. temporal encoding for skills, gripper contact map similarity, etc.
- The paper is mostly well written and understandable. Figures are helpful.
- Real robot experiments show the practical application of the proposed algorithm.

**Weaknesses**

Conceptual: Inadequate motivation discussion for the proposed training procedure and choice of a single latent code in the paper.
- The SDF accuracy and cross-gripper contact map similarity objectives conflict with each other in the current formulation, because they share the same latent code. In the limit, training manages to drive the triplet loss to zero, that means two grasps with different grippers but similar object contact maps would have the same latent codes, and hence the same SDF predictions i.e. at least one gripper will be reconstructed incorrectly.
- This is seen in the significant inaccuracies of the inferred contact maps in Figure 3, which implies large gripper reconstruction inaccuracy. Table 1 chamfer distance numbers corroborate this.
- This raises the question, why the two objectives need to be 1) solved with the same latent code, and 2) optimized simultaneously. For example, how about training $\theta$ and latent codes firstly with just the SDF accuracy objective i.e. same as DeepSDF. Then a _separate_ MLP can be trained _separately_ to embed the DeepSDF latent codes into a different latent space with the proposed triplet loss. This would resolve the conflicting objectives mentioned above.

Experimental:
- Grasp transfer between various grippers is mentioned as a strong motivation (L29), but the real robot experiment shows only one simple top-down grasp.

Clarification:
- Section 5.2, "grasping with object pose estimation": Eq. 5 requires the latent code, it is not clear which latent code is used to optimize the object pose.
- Figure 3: Please show the contact maps from multiple viewpoints.
- Table 2: how the grasp success rate defined?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I am recommending to accept this paper because of the clear research question and good execution. The idea of imposing distance metrics on a neural grasp representation can be developed further for multiple applications.
issues: Please address the "clarification" questions in the "weaknesses" section. Addressing the conceptual question by counter argument or by implementing and evaluating the the suggested two-stage learning process would be even better, but I don't think is necessary for acceptance.
review_title: null

Meta-review: metareview: All the three reviewers acknowledge somehow the novelty/originality of the paper. During the rebuttal, the authors bring in answers and clarifications to the concerns of R2 and R3. While they are not totally convincing, R2 has turned his recommendation from weak reject to weak accept, while R1 and R3 keep their rating. As a result, I recommend accepting the submission. 

confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)