Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present ISAGrasp, a method for performing dexterous grasps on arbitrary objects using a pre-trained network from a large dataset augmented by a generative model which outputs corresponding grasp points on deformed meshes. This is achieved with a combination of labeled human grasp demonstrations with pose-retargeting, to get corresponding robot (Allegro) grasps. To generate novel objects and grasps, they use DIF-Net, building upon prior work by Deng et al. '21, which returns point-wise deformations of the objects used in the human demonstrations. They compute new grasp locations that minimize the total offset to the original grasp over a local patch of points. Experimentally, their method outperforms RL, heuristic, and grasping baselines. 
strengths_and_weaknesses: Strengths
- Their grasping policy generalizes to novel object instances, provided that an accurate point cloud of the object can be generated from the scene.
- Their method extensively compares to different grasping approaches with a wide range of objects, in addition to ablating the input features used by the grasping policy. These comparisons show strong benefits to shape augmentation.

Weaknesses
- By using an open-loop policy, their method is unable to react to dynamic/cluttered environments or recover from failed grasps
- Their simulation does not account for kinematic infeasibility with the environment, both for the arm and the hand.
- This method does not consider functional/dexterous grasps of objects, and therefore does not leverage the dexterity of the hand.
- Their method uses demonstrations, but does not leverage an RL approach which is designed for use with demonstrations
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, this method is novel in the scope of works that have looked at dexterous grasping, as shape augmentation to scale human demonstrations has not been previously studied. For this, I think the paper does a convincing job of laying out the approach and comparing to other possible approaches. 

My only issues with the paper in its current state are that a) it does not compare to any approaches that also leverage learning from demonstration (which is perhaps the most direct comparison), and b) the RL method used is not well suited for use with demonstrations. As Table 1 does not really show RL being very successful at this task, it would make more sense to instead compare to a method such as DAPG (Rajeswaran et al. '18) to leverage such demonstrations. As for methods that involve learning from demonstrations, both DexVIP (Mandikal and Grauman '21) and DexMV (Qin et al. '21) would be effective comparisons. Another option could also be to compare to GAIL or offline RL approaches such as TD3 or SAC (in simulation). As both of these works also consider pose-retargeting and similar robot grippers, it would be worth trying to see if shape/grasp augmentation can benefit these methods as well (as this is the core contribution of this work).

As for the datasets used, the authors do leverage an extensive set of objects from the YCB, ShapeNet, and GoogleScans datasets, and do show a significant difference in performance on these objects when compared to the baselines. One further comparison to aid in showing what advantage shape augmentation has over domain randomization would be to compare to another grasping policy that is only trained with standard domain randomization (adding perturbations to the initial grasp/object pose/forces on the object).  This would help isolate how much shape augmentation improves the policy compared to each of these randomizations (and when used in conjunction with). 
issues: Summarizing the issues described in the summary of recommendation:

1.  The RL algorithm used (PPO w/ and w/out demonstration) does not make a fair/useful comparison as this method is not designed to work with demonstrations.
2. There is no direct comparison between shape augmentation and other forms of domain randomization (such as initial grasp/object pose/force perturbations), which could be used when learning a grasping policy instead. 
3. In addition to a better RL method to compare to (such as DAPG or simple Dagger/supervised learning), the paper should further highlight the advantages/differences to similar existing methods involving RL + learning from demonstration (such as DexMV and DexVIP). It is unclear whether these methods would perform better or worse than the grasping policy learned with shape augmentation given the same demonstration dataset/retargeting inputs. Mentioning the trade-offs/similarities to these approaches is not clearly discussed in the Related Work section (lines 269-270), and should be expanded further.

I have updated my original review as these issues have been addressed by the authors in their rebuttal.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies the problem of dexterous grasping. It proposes to use correspondence-aware implicit deformation networks to propagate a small number of human grasp demonstration to grasping configurations on a diverse deformed object set. Trained with the generated objects and corresponding grasp configurations, the grasping policy can better generalize to unseen objects in human demonstration.
strengths_and_weaknesses: Strengths:
1. It's a smart idea to augment human grasping demonstrations with correspondence-aware shape deformation networks. It simultaneously creates novel shapes and the corresponding grasp configurations.
2. Simulation experiments demonstrate the efficacy of the proposed shape augmentation in the comparison between last two rows in Table 1.
3. This paper conducted real-world experiments of dexterous grasping, which makes the results much stronger.

Weaknesses:
1. Table 1: The comparison with baselines seems not very fair. Random, Heuristic and GraspIt! should be compared with transformed grasps Gd when performing the rejection sampling (133). Collect data with rejection sampling on augmented shapes using Random, Heuristic and GraspIt! with a same budget and compare the result.
2. 248: what does refinement rate mean?

Minors:
220: baseelines -> baselines
Fig 8 caption: miss one '(' before "Middle Panel)", near the end of the second line
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The idea of this paper is very novel and interesting, but the comparison with baselines is not very fair so it's hard to tell how much gain does the proposed method bring. I believe this can be fixed so I vote for weak accept.
issues: Fix the comparison with the baseline:
Collect grasping data with rejection sampling on (augmented) shapes using Random, Heuristic and GraspIt!. Use a same budget as the proposed method and compare rejection rate and the final performance of same model trained on these different data.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a system, Implicit Shape Augmentated Grasping (ISAGrasp), to augment limited human demonstrations of dexterous grasps.The implicit shape augmentation is built on DIF-Net [11], a correspondence-aware implicit generative model. Novel shapes are generated via deformation and the resulting dense correspondences helps transfer the human demonstration to novel objects. The transferred grasps are refined via simulation and a grasp prediction model is trained on the augmented dataset by supervised learning.
strengths_and_weaknesses: Strengths:
* Human demonstration of dexterous grasping is expensive to collect and always require specialized set up. The paper proposes an interesting way to extrapolate the limited demonstrations to a large dataset of novel objects.
* The correspondence-aware generative model is used in a reasonable way, and the effectiveness of the method is shown by experiment. 
* The system transfers well to real world and achieves a decent result.

Weakness:
* Analysis of the results is insufficient. The authors only show that "method A gets a better score than method B", but doesn't really explain why.
  * Why do the baselines, such as Heuristic and GraspIt perform so poorly? Are there any specific failure mode?
  * Why doesn't data augmentation improve the performance on RescaledYCB? Is it because the distribution of augmented objects are different from the YCB dataset, or it's something else? 
  * I can understrand that augmentation can make the grasping policy generalize better to novel objects, but what are the source of remaining perfomance gap (~30% on 3 datasets)?  
* The limitation part is not very satisfactory. I want to see more limitation on the algorithmic side, for example, what are the failure mode of the method and how can we further improve the success rate.

Questions:
* I like the second term (relative poses) of the regarting optimization. My question is why do you choose the center of the object as anchor point, instead of '''closest point on object surface'. Isn't the latter one more robust to shape variation and better modeling the contact?


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, I think it's an interesting paper the attempts to address the core problem of data-driven dexterous grasping - how scale up the dataset. Personally I'm not fully convinced that augmenting the dataset with generative model is the way-to-go, for example, the performance on YCB-rescaled is not improved after data augmentation. But it could still be a contribution to the community, given that the authors can provide more interpretations of results.
issues: * In Sec 4.3, refinement rate is not explicitly defined.
* Typos: L248, Figure8 -> Figure 8
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposed a data-augmentation framework to learn dexterous grasp pose from point cloud observation. First, the DexYCB dataset which consists of human hand object interaction trajectories is used to provide an initial demonstration. Second, hand motion retargeting is utilized to convert the human hand motion to robot hand joint angles. Third, DIF-Net, an implicit neural network that can keep dense correspondence while deforming an implicit shape representation, is used to deform the original object mesh with sampled latent gaussian vectors. With correspondence after deformation, the grasp pose can also be modified according to the object. Then a rejection-sampling-based grasp refinement step is performed inside PyBullet simulator to eliminate physically-unfeasible grasp candidates. Finally, the author trained a PointNet++ to predict the palm pose and finger joints from point cloud input. 
strengths_and_weaknesses: This paper proposes a data augmentation strategy for dexterous grasping and motivates the problem well in the introduction. The proposed object-centric deformation method is very general and agnostic to tasks and manipulators. There are also several issues that need to be addressed in the paper as follows:

### Strength:

1. The proposed method is simple and elegant. It does not rely on dynamics analysis of the contact wrench space or other force-based measurements. The dynamics correctness checking is achieved solely by a physical simulator while the underlying troublesome computation is hidden below. 
2. In principle, the implicit-based deformation network can generate infinite object mesh and corresponding grasp pose. This method is quite general.
3. The observation that “appending object points with additional information regarding the153
alignment between the robot hand and the local object surface” is beneficial for many robot-orientated regression tasks with point cloud input.

 
### Weakness:

1. The original demonstration dataset (DexYCB) seems not so useful in the whole pipeline, it only provided the initial object-grasp pair. The following data generated is achieved in a sample and reject fashion: sample with DIF-Net and reject with physical simulator. So the original dataset can be replaced by any method that can provide roughly okay grasp proposals, even if the grasp proposal itself is not dynamically-correct or successful. For example, the author can use any grasp proposal network, e.g. Contact Grasp Net, to generate a grasp pose and use the same data augmentation procedure for the following steps. In this case, the training data is not even limited to only YCB objects but any object meshes, which may lead to better generalization performance due to the increasing diversity of the training data. I assume that maybe one benefit of using human demonstration is for functional grasp when you do not want some grasp poses even if they are successful in simulator dynamically but not semantically feasible.

2. The PPO baselines with **sparse reward** seem too weak. For manipulation tasks with a dexterous hand, sparse reward will lead to nearly zero success rate. In this task, writing a distance-based dense reward is also very simple and straightforward, especially the authors can train the algorithm inside a PyBullet simulator.

3. The GraspIt baseline seems not used in a fair way: in the experiments, the author utilizes the results predicted by GraspIt and other heuristic to evaluate the grasp performance. However, the major contribution of this paper is the implicit shape-based augmentation, so a better way to compare GraspIt is as follows: i) replace DeYCB and implicit shape augmentation with GraspIt to generate diverse grasp poses ii) use PyBullet to reject sample the unfeasible grasp poses iii) train the same PointNet++ based on the data generated by GraspIt. Since the contribution is on the training data side, the GraspIt should also be used to generate training data but not evaluation. Otherwise the experiments may show the value of the Grasp Refinement for Dynamics Consistency but not Implicit Shape Augmentation.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall this is a good paper because it 
1. focuses on important techniques, 
2. are very general to tasks and manipulators, and 
3. show large performance improvement in sim and real. In particular, the fact that the DIF-Net will also keep the semantic correspondence can extend the application of this paper to functional grasp, i.e. always hold the handle for a mug while deformation is used.

issues: - More details about the baselines 
- More motivation on using the annotated dataset in the beginning
- Figure to illustrate the symbols in Section 2.2. It is better to give a better visualization on how to estimate a new grasp with symbols, e.g. t^d annotated on the figure.
- More clarification on comparison with the baseline method.
review_title: null

Meta-review: metareview: All the reviewers acknowledge somehow the novelty/originality of the paper but have all questioned the proposed approach in terms of baselines and fair comparisons, e.g.,  PPO, GraspIt. In the phasse of rebuttal, the authors have significantly reinforced the comparison of some well-known baselines suggested by the reviewers. As a result, all the reviewers achieved a consensus for weak accept
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)