Learning Robust Real-World Dexterous Grasping Policies via Implicit Shape Augmentation
Authors: Qiuyu Chen
Karl Van Wyk
Yu-Wei Chao
Wei Yang
Arsalan Mousavian
Abhishek Gupta
Dieter Fox
Venue: CoRL 2022 Poster (2022)
Keywords: Dexterous Manipulation
Learning from Human Demonstration
Data Augmentation

Abstract: Dexterous robotic hands have the capability to interact with a wide variety of household objects. However, learning robust real world grasping policies for arbitrary objects has proven challenging due to the difficulty of generating high quality training data. In this work, we propose a learning system (\emph{ISAGrasp}) for leveraging a small number of human demonstrations to bootstrap the generation of a much larger dataset containing successful grasps on a variety of novel objects.  Our key insight is to use a correspondence-aware implicit generative model to deform object meshes and demonstrated human grasps in order to create a diverse dataset for supervised learning, while maintaining semantic realism. We use this dataset to train a robust grasping policy in simulation which can be deployed in the real world. We demonstrate grasping performance with a four-fingered Allegro hand in both simulation and the real world, and show this method can handle entirely new semantic classes and achieve a 79% success rate on grasping unseen objects in the real world. 

Learning Robust Real-World Dexterous Grasping Policies via Implicit Shape Augmentation
Zoey Qiuyu Chen
Karl Van Wyk? Yu-Wei Chao" Wei Yang*
Arsalan Mousavian
Abhishek Gupta'
Dieter Fox
University of Washington 2NVDDIA
fqluyuc, abhgupta, fox) www.wwwwts.whn..cccoco edu fkvanwyk, ychao, weiy, amousavian/anvidda..
attps://sites..oogrss http:///tee..ooooo..eem///e///// http:///ww//eemmaaaaaaaaaaaaaaaa
Tireto
Retarge
Argrente Datror
Bilker With Erowwe Chypes
Implict Argmenazion
Train
Figure I: illustration of training scheme ISAGrasp. human demonstrations are provided from motion capture. These demonstrations are retargeted to a four- fingerie allegro robot simulation. We use correspondence-awarr generative model to extrapolate the retargeted demonstrations to large dataset of novel objects. We use this dataset train single grasping policy that is able to generalize to variety of unseen objects simulation and real world.
Dexterous robotic hands have the capability interact with wide variety of household objects to perform tasks like grasping. However, learning ro- bust real world grasping policies for arbitrary objects has proven challenging due to the difficulty of generating high quality training data. In this work, we propose a learning system (ISAGrasp) for leveraging a small number of human demonstra- tions to bootstrap the generation of much larger dataset containing successful grasps on a variety of novel objects. Our key insight to use a correspondence- aware implicit generative model to deform object meshes and demonstrated hu- man grasps in order to generate a diverse dataset of novel objects and successful grasps for supervised learning, while maintaining semantic realism. We use this dataset to train a robust grasping policy in simulation which can be deployed the real world. We demonstrate grasping performance with four-fingered Alle- gro hand both simulation and the real world, and show this method can handle entirely semantic classes and achieve 79% success rate grasping unseen objects in the real world
Abstract:
Dexterous Manipulation, Learning from Demonstration, Data Aug- mentation, Grasping
Keywords:
Introduction
Human hands are powerful tools for manipulating a wide range of objects. Our goal is build dex- terous robotic manipulators that can robustly and adeptly interact with human-centric environments. However, robustly controlling a dexterous hand remains challenging due to its high dimensional
Work done while the author was part-time intern at NVIDIA.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

state and action space and its qulti-modat contact dynamics. Recent work has used deep reinforce-
ment learning algorithms learn complex tasks with dexterous manipulators [1, 2, 3. 4, 5, 6]. These methods often require careful reward engineering and environment design, and often lack general- ization, struggling with robust real world deployment. An alternative paradigm involves collecting a large labelled dataset and using supervised learning (imitation learning). This has shown significant success with parallel jaw grippers [7. 8, 9] and suction cups [9, 10]. However, imitation learning methods have proven difficult to scale to multi-fingered robots due to the burden of human data collection. To scale robust policy learning to dexterous grasping, we require techniques that can leverage small amount of human effort to learn robust, general grasping policies. This can be done by extrapolating a small number of human demonstrations to generate an abundance of data that interacts with diverse objects and is successful at grasping objects under real world dynamics. The key question is hoow do we generate this type of diverse training data?
In this work, we propose Implicit Shape Augmentated Grasping (ISAGrass), shown in Figure 1, a learning system that leverages a correspondence-awrrr generative model [11] to extrapolate a small number of human demonstrations to a large dataset of realistic objects and their corresponding grasps. In particular, we directly perform deformations on implicit 3-D shape representations of various objects in a learned latent space. The representation of point-wise shape deformations allows us to generate transformed grasps for novel objects that can be made successful with a small amount of active interaction in simulation. In doing so, implicit shape augmentation allows to grow from a small dataset of human demonstration data for dexterous grasping to a significantly larger and more diverse dataset with novel object shapes and dynamically successful grasps. Given the dataset constructed by implicit shape augmentation, we can now perform large scale supervised learning. In this work, we represent policy as predicting pre-grasp and final pose from pointclouds of the target object, with intermediate motion being performed with standard motion planning libraries. We show empirically that a policy learned via supervised learning on the dataset constructed by ISArras is able to generalize widely across different objects in simulation and the real world. We demonstrate the efficacy of this pipeline on the rescaled YCB instances, ShapeNet and GoogleScans objects in simulation and achieve a 79% success rate on 22 unseen objects in the real world.
In summary, contributions are (1) we propose novel system, ISAGrasp, that is able to generate both a wide variety of objects and the corresponding dexterous grasps from a few human demon- strations. (2) We show that the augmented dataset can be used for learning point-cloud based control policies that are able to robustly grasp a large variety of novel objects in simulation (3) We demon- strate the efficacy of the proposed pipeline by deploying it in the real world. (4) We analyze the proposed pipeline through several ablation studies in simulation.
2 ISAGrasp: Dexterous Grasping Policies via Implicit Shape Augmentation
Human Demo
Retarget
Mesh & Grasp
Dataset
G
Gr
Gd
Odl
Implicit Shape Augmentation
Rejection Sampling
Deformed Mesh & Grasp
Latent Vector
Figure 2: Implicit Shape Augmentation for generating augmented dataset from demonstrations. First a human demonstration is retargeted onto the Allegro hand to generate meshes and grasp labels. This data can then be used to generate variety of new objects via shape augmentation with DIF-Net [11]. Grasps for these deformed objects can then be further refined with rejection sampling to generate dynamically consistent grasps.
To learn robust grasping policies that can operate in the real world for grasping novel objects of various shapes, we propose ISAGrasp -a framework for supervised learning of robust and general-

---

N
dg
ar.
8, 12
(1)
i=0
N
de
Σ
||0;; - Ch: 112
(2)
Human
Allegro
==
d = GMM,, MA),
(3)
Finger Vertor
Ciiject-Finger Victor
Orientation
arg min (Wydg wode + We w.dr); (...)
Figure 3: Illustration of retargeting human hand demonstration to an Allegro hand
Figure 4: Optimization setup describing the retar- geting problem from human to Allegro hand
izable grasping policies from successful grasps on a large variety of objects generated from a small set of human provided expert demonstrations.
Given a pointcloud of object, model grasping policy predicting (1) a pre-grasp pose T that controls robot hand's translation Te and rotation Tq. and (2) a final pose G3 that controls the 16-DoF finger pose that can firmly grab the object.
ISAGrasp assumes access to a set of labelled human grasping demonstrations, can be used to learn grasping policies via supervised learning. However, to learn truly robust and general grasping poli- cies, this dataset must be grown to a much more diverse set of objects and successful grasps. Gen- erating multi-fingered grasping dataset of diverse yet realistie objects is a non-trivial problem [12, 13, 14]. We approach this problem by leveraging a correspondence-aveer aware, deformation based generative model to widely augment the set of human provided demonstrations, which can then be used to learn robust and general policies via supervised learning. An overview of our ISAGrasp system is shown in Figure 2, and detail each component below.
2.1 Human-Robot Retargeting
ISAGrasp first collects N human hand arm demonstrations D= (TD: : 7t. ..... N using motion cap- ture, where each demonstration f; {(hā o)), ( (h o is trajectory of hand pose h and object pose Ot- These demonstrations are then "retargeted" hand [15] in simulation. Note that this is nontrivial problem since these demonstrations are in the morphology of the human hand and are typically not functional when directly mapped to a robot hand using standard Inverse Kinematics [16].
22-DoF floating Allegro
We formulate the retargeting objective as a non-linest optimization problem. First, to allow a robot to grasp in similar pose to human demonstration, we use the same cost function proposed in DexPilot [17], shown Eq. 1, where 8, the ratio between the sizes of the robot and human hands, and each a,, and aλ; is displacement vector between finger tips and the palm of the robot and human hands, shown as blue in Figure 3. While Eq. 1 encourages grasp shape similarity, we additionally minimize the differences between the relative poses to the object, as captured by the vectors between fingers and the object center for the human grasp, Cr,, and the retargeted grasp, cj, (see Eq.2 and orange lines in Fig. 3). Finally, to orient the robot palm similar to the human hand, add Eq. 3, which optimizes the minimum geodesic distance G in SO(3) between the rotation matrix of the human palm M, and the robot palm M,. The final retargeting goal is to find the 22 DoF configuration that minimizes the weighted sum of these three terms, which results in a candidate grasp G, for each object in human provided dataset.
As we describe in Section 2.3, the retargeted grasps are then refined via rejection sampling step and used to generate the dataset with successful, dynamically consistent retargeted grasps. However, since this dataset typically quite limited, we next outline how to augment the dataset with novel objects and grasps via implicit shape augmentation.
2.2 Implicit Shape Augmentation
Given a small set of object point clouds and successful retargeted grasps, G,, we build a large scale augmented dataset of novel objects and their corresponding successful grasps Ga, using correspondence-awaan implicit generative model, DIF-Net [11].

---

To recap at high level, DIF-Net aims to learn an implicit representation of 3D shapes as scalar field. More specifically, DIF-Net represents a 3D shape via a instance agnostic template implicit field (which is common for all shapes of particular category and represent the common features of a category), together with latent conditional 3D deformation field (that perturbs this template field) which allows the shapes to be adapted to every particular object instance while maintaining semantic 3-D structure. Different novel but realistic shapes can be generated by sampling different latent vectors and generating object deformations on known object classes using the learned de- formation and correction fields, while maintaining semantics. This model naturally provides dense correspondences across object instances since different object instances pointwise deformations on the same template.
We use this generative model in two ways: (1) leverage the ability to sample a variety of object instances by sampling latent vectors a to generate various novel objects via deformations and (2) the resulting dense correspondences allow us to estimate dynamically consistent grasps from human demonstrations to novel generated objects. Specifically, we use the latent conditional deformation field from a pretrained DIF-Net to generate novel instances and dynamically consistent grasps. To generate novel objects, we sample latent vectors a from a Gaussian that conditions latent con- ditional deformation field. This deformation field generates point-wise deformations of particular object meshes chosen from the set human demonstrations to generate novel objects. Second, to estimate the new grasps Ga for the deformed objects, we find N reference points the original mesh that are close to the root position t" of the robot hand and compute the position to of the robot relative to a local coordinate system O₁ centered each reference point f = 0, .t. Once the object is deformed, we compute the corresponding coordinate system O on the deformed meshes using the same deformation field, to estimate the new grasp location la using the average of the local offsets: 1d N. MN i 42). In this way, the dense correspondences obtained via the DIF-Net are directly useful in generating grasps for novel, deformed object instances.
Deformation Reference
Implicit Shape Augmentation
Deformation Field
Latent vector
Figure 5: Deformation map and grasping correspondences for objects generated in ISAGrasp. Grasping corre- spondences the original object (reference) and the deformed objects highlighted inside the circle. As be seen, object semantics are maintained. Different object instances are generated by sampling different latents
Figure visualizes how objects and correspondences deform with different sampled latent vectors. We highlight the reference points on the original mesh, and their correspondences on deformed meshes (purple circles). The objects are deformed into a variety of realistic shapes while maintaining the original semantic structures and grasping correspondences.
Using the new grasp location td, together with retargeted orientation q" and finger pose f", we can construct new grasps Ga(t", f")." While Ga are not guaranteed to be successful grasps, they provide good starting points for the local search procedure described next.
2.3 Grasp Refinement for Dynamics Consistency
Given transformed grasps Ga. we use pose perturbation with rejection sampling to generate suc- cessful and dynamically consistent grasping poses. Since our shape augmentation model transforms successful grasps via the 3D deformation field, found that only a small amount of perturbation is typically needed to find successful grasps for the deformed models using rejection sampling. In particular, we sample local perturbations dc. 6, and 8y from a uniform distribution, and add these perturbations to the translation dd rotation q" and finger joints f" of the transformed poses Ga- We evaluate success of each perturbed grasp Gp using a physics simulator (Pybullet [18]) and add domain randomization save robust successful grasps and objects the training dataset.

---

Using above methods, we can generate large and suc- cessful dexterous grasping dataset with variety of novel objects, and then perform supervised policy learning, described in the next section.
PoinaNatt
HIOe
NA Medile
Tramebation
112
R.
2.4 Policy Learning via Supervised Learning
Pain Roteius
Scaae Encedes
As described in the beginning of Section 2, we model the grasping problem predicting a pre-grasp pose TT and final pose Gy given an object pointcloud observation. We perform a standard empirical risk minimization procedure on the network consisting of PointNet++ SA modules [19] as feature extractor. Instead of only feeding the raw point cloud the network, we found significant improvements by appending object points with additional information regarding the alignment between the robot hand and the local object surface. In particular, we use the object point cloud p. the surface normal at each point No. the normal of the table surface No, and the hand facing direction Ny and pointing direction Np define the following feature vectors: fp))=(=.. Py, Pz. (No Ny)). (No Ny), (No -N)), L.y.NN We append these vectors to each point in the point cloud. We found these features provide compact description of alignment between the robot hand and the object, and using the relative vector alignments via the pairwise dot products allows to improve the final grasping performance.
$12
16
P Finger Juints
Figure 6: Policy network architecture. The point-nei++ architecture inputs an object point cloud p. object surface normal No, the table normal N₁, rohot facing direction N, and pointing direction N, to generate palm tramslation, rotation and finger joints for the dexterous grasp.
dataset as an architecture and use
The network outputs a 3-dim translation and 4-dim quaternion, which are used to define the pregrasp pose T. Additionally, the network predicts a 16-dim finger poses, which is used to define final pose Gy. Figure 6 shows our network architecture details. We provide training details in Appendix A.
System Details
Initial Dataset Construction. We extracted human demonstrations from the Dex dataset [20]. which contains mocap sequences Dhuman of humans hand poses or picking up 20 YCB objects [21] with poses of We picked 10 demonstrations per object, resulting in 200 demonstration total.
Appendix B 8.2 further explains our dataset choice.
pretrained DIF-Net [11] augment objects into a variety of novel shapes. First, we sample a 128-dim latent vector from a Gaussian distribution ~N(,, g²), where μ and 0.002. Second, to estimate corresponding new grasps Ga. choose N = 20 closest points on the object surface as reference points and compute corresponding grasp location fu (Section2.2).. To obtain successful deformed grasps, we apply rejection sampling by uniformly sampling &, E [-0.0 m, 0.02 m] for translation tu, and 8, [-0.5, 0.5] radians for rotation Gr, and apply the same perturbation 8y E [-0.1, 0..11.01] radians for finger joints.
ISAGrasp Implementation Details. We
Hardware Setup. We deploy our policy to a robotic platform that has 23 actuators across a KUKA LBR iiwa 7 R800 robot arm and a Wonik Robotics Allegro robotic hand, and two cameras to provide necessary point cloud information. Details are provided in Appendix B.
Experiments
Through experimental evaluation, we aim answer the following questions:
1. Does ISAGrasp generate realistic novel objects based on a small set of scanned objects?
2. Does ISAGrasp allow for easy generation of dynamically consistent grasps on novel objects through grasp transformation and refinement?
3. Do policies learned on the dataset produced by ISAGrasp show improved robustness and generalization on unseen objects?
4. How does the training perform with different input features.
5

---

Table 1: Baselines evaluated in simulation
RescaledYCB
ShapeNet
GoogleScans
Random
0.03
0.05
0.02
Train successful random
0.15
0.42
0.18
Heuristic
0.27
0.40
0.16
Train successful heuristic
0.09
0.15
0.02
Train on successful Grasplt
0.29
0.42
0.20
PPO with dense reward
0.12
0.10
0,06
DAPG +DR
0.46
0.51
0.53
DexYCB DR - ISA
0.34
0.35
0.22
DexYCB + DR ISA
0.74
0.56
0.51
DexYCB +DR +ISA (ours)
0.74
0.74
0.70
We address these questions through a study in PyBullet [18] simulation, followed by real world experimental evaluation using the robotic system described in Section 3. We provide additional details of baselines in appendix B and more analysis on elements of the ISAGrasp in Appendix C.
Evaluation Metrics. We choose three unseen datasets with increasing complexity: RescaledY CBB ShapeNet [22], and GoogleScans [23] (see appendix B). In particular, RescaledYCB contains 65 rescaled YCB objects, ShapeNet contains 200 unseen objects from "Can", "Bottle", "Mug" and "Bowl" categories, and GoogleScans objects contains 200 everyday objects. We place objects randomly on the table, and evaluate performance with success is defined as when the object is above the table by 10cm.
of object mass and friction. The
Baselines. Table 1 additionally compares the performance of our method to other baselines -11 Randomly generate grasps around the object (Random) (2) use random baseline with rejection sam- pling create successful dataset and train policy (3) perform predefined grasp where robot always grasps from the top (Heuristic). (4) train with Heuristic baseline with rejection sampling (5) train with Graspit!, optimization-basee grasp planner from prior work[24]. In addition, train compare RL baselines: (6) one with dense reward function using PPO[25] (7) One using an RL method (DAPG DR) that combines imitation learning objective via behavior cloning with reinforcement learning [26]. (8) train with domain randomization, but no shape augmentation YCB DR ISA) (9) train with shape augmentation domain augmentation (Dex YCB DR ISA) in order to understand the impact of shape augmentation and the impact of domain randomization. Please find further details of these baselines in the Appendix B.
Figure 7: Qualitative results unseen objects in simulation and in the real world. The top 3 rows shows the successful examples of our method GoogleScans objects and the bottom 2 rows show successful examples using our policy deployed on unseen objects real world.

---

Table 2: Real world test of ISAGrasp 22 unseen objects. ISAGrasp is able achieve 79% success
obj
Tray
Crab
Chips
Box1
Box2
Bread
Flower
Pot
Roll
Hat
Honey
#
3/5
4/5
4/5
2/5
4/5
5/5
3/5
4/5
4/5
5/5
5/5
obj
Box5
Scarf
Purse
Tape
Box3
Bottle
Cream
Cap
Drill
Bag
Pringles
#
4/5
5/5
3/5
4/5
5/5
4/5
5/5
5/5
2/5
4/5
3/5
4.1
Simulation Results.
We first evaluate the efficacy of the ISAGrasp system on learning dexterous grasping policies in simulation. Table shows the overall success rate of our method ISAGrasp on three unseen datasets in simulation. ISAGrasp (bottom row) is able to achieve 74% success rate on rescaled unseen YCB objects even the policy is only trained on augmented shapes. In addition, our method achieves 74% and 70% success rate on ShapeNet and GoogleScans objects. We describe analysis on how the various baselines perform in details in Appendix B.
4.2 Real World Experiments.
Next, we evaluated the grasping policy learned in simulation by ISAGrasp, directly in the real world. We perform directly simulation to reality transfer of the learned policy as described in Section 2.4. We evaluate our policy on 22 unseen real world daily objects (see Appendix Figure 17) and evaluate each object 5 times with random poses. We report number of success on each object in Table 2. On average, the policy is able to achieve 79% success rate on real world evaluation on novel objects. We observe several failure cases: (1) if the object is more transparent (Boxl: container box), the pointclouds are incomplete and the network is less robust. (2) objects which require more careful grasping (Power drill) often have lower success rate. These show that can leverage policies trained in simulation directly for real world grasping using ISAGrasp and that the robustness and generalization properties transfer from simulation to the real world.
4.3
Ablations and Analysis
We first provide some insights into the impact of various design decisions in ISAGrasp below:
Impact of using correspondence-avare generative models: To understand how important us-
Correspondence-averr
Random Generated
Refinement Rate
Origizal
Puitswii Guosiia
DEF
SlapeGAN
ShapeGAN(R)
0.09
ShapeGAAN(D)
0.29
DIF-Nei(R) DIF-Nei(D)
0.13
0.51
DIF-Net(Corr)
0.76
Figure& 8: Analysis on shape generation. DIF-Net generates realistic and semantically meaningful objects (Left Panel) while ShapeGAN generates novel objects but often unrealistic and without any correspondences (Middle Panel). We show the refinement rate for obtaining successful grasps on these objects ( R: random grasps, D: retargeted demonstration, Corr: correspondence guidedd grasps (right pancl). The lack of correspondences makes refinement challenging, yielding only 30% success rate compared to DIF-Net at 76%.
ing the dense correspondences provided by DIF-Net are for generating successful grasps on novel objects, we compare DIF-Net [11] with a prrresponddnce-ggnnott model, shapeGAN [14] by show- ing their generated meshes and the refinement rate for obtaining successful grasps. Refinement rate refers to the ratio of grasps that can be successfully refined via rejection sampling to the total number of proposed grasps. We use this metric to compare the efficiency of creating stable grasp datasets generated using different approaches. Shown in Figure (Left), DIF-Net can smoothly deform the original shapes and generate more realistic shapes. We conduct 50 times of perturbation with jection sampling on 100 objects generated by both methods. Figure8 (Right) shows refinement rate (R: random grasps without human demonstrations, D: using original human demonstrations without new grasps Ga estimation, Corr: using new grasps Ga). Initialized with correspondence-guimm grasps, DIF-Net (Corr) achieves 76% refinement rate while ShapeGAN(D) achieves below 30% with the same number of refinements. This indicates the importance of transferring grasps through deformation based transformation for novel objects.
Impact of Input Representation: To understand the choice of input representation for supervised learning, we compare feature choices used as input for training policies, evaluated on objects from
7

---

the ShapeNet and GoogleScans datasets. We observe that using pointcloud surface normal vector No. Robot vectors Nj and Nr performs best.
5 Related Work
Manipulation with Dexterous Hands. To control a dexterous hand, prior work has inves- tigated including planning with analytical mod- els [27] and online trajectory optimization [28]. However, these methods assume accurate dynamics models and robust state estimates, which are difficult to obtain in complex real-world manipulation. Learning-basee
Table 3: Ablations on imput features
P+N+NN +N p P+N.
Success 0.47
0.57
0.72
approaches particularly with deep reinforcement learning (RL) [29, 30, 5, 31, 32. 6, 2] have been investigated. Despite the progress, training deep RL models remains challenging due to high sample complexity and reward engincering. Although this issue has been mitigated by incorporating human demonstra- tions [26, 33, 3, 34, 1J. these methods are still faced with a major challenge in scalability[3, 21. Prior work has collected demonstrations kinesthetically [33]. through VR interfaces [26], or using motion capture (mocap) solutions [35, 36, 37, 3], which are often limited in size. In comparison, the focus of our work allows policies to generalize novel, unseen shapes, which could be easily combined with pipelines for improvement with RL with demonstrations [3, 1, 38, 26].
Data Augmentation and Robustness. Data augmentation has been used traditionally in vision tasks where images cropped, rotated, normalized, etc [39, 40, 41, 42, 41, 42] improve gener- alization and model robustness. In contrast, our work aims to generate novel objects with varying physical dynamics and does not simply learn invariant behavior but learns different grasping be- havior for different objects. In a similar vein, domain randomization is used in robotics, where predefined parameters such lighting, camera are randomized during training(43, 44] in order to learn a policy that is invariant to these parameters. However, this randomization does not aid with generalization novel object shapes. More recently, methods have aimed to generate novel objects shapes programmatically [13]. or use a generative model to create new objects in an adversarial setting [45]. This is. motivated similarly, but differs in being for parallel jaw grasping problems and not generating dynamically consistent grasping behavior via correspondences.
Limitations
Challenging objects: it's more challenging for our policy to succeed when the object is large or too flat. Since our shape augmentation built on dex dataset, it becomes more challenging the test object is too different from the training objects, or requires more specific way of grasping. The shape augmentation also does not cover objects of widely varying scales. See visualizations on chal- lenging objects in Appendix D. Real world experiments: The method assumes access to fairly complete point cloud. It will not succeed in scenarios with heavy amounts of occlusion or noise in the point clouds. Functional grasping: Currently the method does not do dexterous and functional grasps, and is only designed to lift the object. The utility of dexterous hand is perhaps best utilized with functional grasps, but the current system does not optimize for this directly. Accounting for kinematics: The current system does not account for the kinematics of potentially hitting the table when the hand is mounted on full arm setup. This should be accounted for we build on this work in the future.
7 Conclusion
We present ISAGrasp. a novel system for learning dexterous grasping policies in the real world. ISAGrasp leverages data augmentation approach that bootstraps a small number of human demon- strations with large dataset with diverse and novel objects and grasps. By using correspondence- aware generative model, we can deform original object shapes and generate dynamically consistent new grasps. We create large and diverse grasping dataset and train policy via supervised learning that can then be deployed in simulation and the real world on grasping novel objects, achieving over 75% success rate on grasping novel object instances in the real world.
8

---

Acknowledgement
We thank Mohit Shridhar for providing helpful feedback on our initial draft. We thank Aaron Wals- man for providing additional helps on figures and the draft. We are also grateful for Vikash Kumar, Aravind Rajeswaran, Jason Ma and Mandi Zhao for discussions on running RL baseline experi- ments. Part of this work was done while Qiuyu Chen was an Intern at NVIDIA. The work was also funded part by an Intel gift.
References
[1] Mandikal and K. Grauman. DexVIP: Learning dexterous grasping with human hand pose priors from video. In CoRL, 2021.
[2] Chen, J. Xu, and Agrawal. system for general in-hand object re-orientation. In CoRL, 2021.
[3] Y. Qin, Y.-H. Wu, Liu, Jiang, R. Yang, Y. Fu, and X. Wang. DexMV: Imitation learning for dexterous manipulation from human videos. arXiv preprint arXiv:2 058707 2021.
[4] Mandikal and K. Grauman. Learning dexterous grasping with object-centric visual affor- dances. In ICRA, 2021.
[5] A. Nagabandi, K. Konoglie, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In CoRL, 2019.
[6] R. Jeong, J. T. Springenberg, J. Kay, D. Zheng, A. Galashov, N. Heess, and F. Nori. Learning dexterous manipulation from suboptimal experts. In CoRL, 2020.
[7] A. Mousavian, C. Eppner, and D. Fox. 6-DOF GraspNet: Variational grasp generation for object manipulation. In /CCV, 2019.
[8] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1 13438 13444. IEEE, 2021.
[9] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics, arXiv preprint
2017.
[10] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010. 14406, 2020.
[11] Y. Deng, J. Yang, and X. Tong. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10286-10296, 2021.
[12] Brahmbhatt, Handa, J. Hays, and Fox. ContactGrasp: Functional multi-finger grasp synthesis from contact. In IROS, 2019.
[13] Tobin, L. Biewald, R. Duan, M. Andrychowicz, A. Handa, V. Kumar, B. McGrew, A. Ray, J. Schneider, P. Welinder, et al. Domain randomization and generative models for robotic grasp- ing. 20/8 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1489.3489. IEEE, 2018
[14] M. Kleineberg, M. Fey, and Weichert. Adversarial generation of continuous implicit shape representations. arXiv preprint arXiv:2002.003: 00349, 2020.
[15] W. Robotics. Allegro
Hand.
https://www.wwnikkkk
research- ratot--bbt-haaae
[16] M. Gleicher. Retargetting motion to new characters. In SIGGRAPH, 1998.

---

[17] A. Handa, K. Van Wyk, W. Yang. J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. Ratliff, and D. Fox. DexPilot: Vision-based teleoperation of dexterous robotic hand-arm system. In ICRA, 2020.
[18] E. Coumans and Y. Bai. PyBullet: a Python module for physics simulation for games, robotics and machine learning. https:/ htt:://pybulltttee org, 2016-2021.
[19] C. R. Qi, L. Yi, H. Su, and L. Guibas. PointNet++: Deep hierarchical feature learning on point sets in metric space. In NIPS, 2017.
[20] Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. lqbal, S. Birchfield, J. Kautz, and D. Fox. Dex YCB: benchmark for capturing hand grasping of objects. In CVPR, 2021.
[21] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The YCB object and model set: Towards common benchmarks for manipulation research. In 2015 International Conference Advanced Robotics (ICAR), 2015.
[22] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:
2015.
[23] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and V. Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. arXiv preprint arXiv:2204. 202..191111 2022.
[24] A. T. Miller and P. K. Allen. Graspit! versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 144:: 100-122, 2004.
[25] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprini arXiv: D 1707.06347 2017.
[26] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. In RSS, 2018
[27] Y. Bai and C. K. Liu. Dexterous manipulation using both palm and fingers. In ICRA, 2014.
[28] V. Kumar, Y. Tassa, T. Erez, and E. Todorov. Real-time behaviour synthesis for dynamic hand- manipulation. In ICRA, 2014.
[29] V. Kumar, E. Todorov, and S. Levine. Optimal control with learned local models: Application to dexterous manipulation. In ICRA, 2016.
[30] A. Gupta, C. Eppner, S. Levine, and P. Abbeel. Learning dexterous manipulation for a soft robotic hand from human demonstrations, In IROS, 2016.
[31] OpenAI, M. Andrychowicz. B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welin- der, I.. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. arXiv preprint arXiv:
2018.
[32] OpenAl, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, Schneider, N. Tezak, J. Tworek, P. Welinder, I.. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving Rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 2019.
[33] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In ICRA, 2019.
[34] I. Radosavovic, X. Wang, L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. In IROS, 2021.
[35] G. Garciia-ernando, S. Yuan, S. Baek, and T.-K. Kim. First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In CVPR, 2018.
10

---

[36] Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, J. Black, I. Laptev, and C. Schmid. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019.
[37] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas. GRAB: dataset whole-body human grasping of objects. In ECCV, 2020.
[38] M. Vecerik, T. Hester, J. Scholz, F. Wang. O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv: 2017.
[39] G. Benton, M. Finzi, Izmailov, and Wilson. Learning invariances in neural networks from training data. Advances in Neural Information Processing Systems, 33:7666--7761, 2020.
[40] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. Autoaugment: Learning aug- mentation policies from data. arXiv preprint arXiv: 180500951, 2018 8.
[41] C. Shorten and T. M. Khoshgoftaar. A survey image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
[42] L. Perez and J. Wang. The effectiveness of data augmentation in image classification using deep learning, arXiv preprint arXiv:1712.0462 2017.
[43] Tobin, R. Fong, A. Ray, Schneider, W. Zaremba, and Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[44] Tremblay, Prakash, D. Acuna, M. Brophy, V. Jampani, Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 969-977, 2018.
[45] A. Z. Ren and A. Majumdar. Distributionally robust policy learning via adversarial environ- ment generation. IEEE Robotics and Automation Letters, 2022.


Reviews:
[{"review_id": "lurppL0ulDP", "reviewer": "Reviewer_gvHJ", "summary": "The authors present ISAGrasp, a method for performing dexterous grasps on arbitrary objects using a pre-trained network from a large dataset augmented by a generative model which outputs corresponding grasp points on deformed meshes. This is achieved with a combination of labeled human grasp demonstrations with pose-retargeting, to get corresponding robot (Allegro) grasps. To generate novel objects and grasps, they use DIF-Net, building upon prior work by Deng et al. '21, which returns point-wise deformations of the objects used in the human demonstrations. They compute new grasp locations that minimize the total offset to the original grasp over a local patch of points. Experimentally, their method outperforms RL, heuristic, and grasping baselines. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Their grasping policy generalizes to novel object instances, provided that an accurate point cloud of the object can be generated from the scene.\n- Their method extensively compares to different grasping approaches with a wide range of objects, in addition to ablating the input features used by the grasping policy. These comparisons show strong benefits to shape augmentation.\n\nWeaknesses\n- By using an open-loop policy, their method is unable to react to dynamic/cluttered environments or recover from failed grasps\n- Their simulation does not account for kinematic infeasibility with the environment, both for the arm and the hand.\n- This method does not consider functional/dexterous grasps of objects, and therefore does not leverage the dexterity of the hand.\n- Their method uses demonstrations, but does not leverage an RL approach which is designed for use with demonstrations", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this method is novel in the scope of works that have looked at dexterous grasping, as shape augmentation to scale human demonstrations has not been previously studied. For this, I think the paper does a convincing job of laying out the approach and comparing to other possible approaches. \n\nMy only issues with the paper in its current state are that a) it does not compare to any approaches that also leverage learning from demonstration (which is perhaps the most direct comparison), and b) the RL method used is not well suited for use with demonstrations. As Table 1 does not really show RL being very successful at this task, it would make more sense to instead compare to a method such as DAPG (Rajeswaran et al. '18) to leverage such demonstrations. As for methods that involve learning from demonstrations, both DexVIP (Mandikal and Grauman '21) and DexMV (Qin et al. '21) would be effective comparisons. Another option could also be to compare to GAIL or offline RL approaches such as TD3 or SAC (in simulation). As both of these works also consider pose-retargeting and similar robot grippers, it would be worth trying to see if shape/grasp augmentation can benefit these methods as well (as this is the core contribution of this work).\n\nAs for the datasets used, the authors do leverage an extensive set of objects from the YCB, ShapeNet, and GoogleScans datasets, and do show a significant difference in performance on these objects when compared to the baselines. One further comparison to aid in showing what advantage shape augmentation has over domain randomization would be to compare to another grasping policy that is only trained with standard domain randomization (adding perturbations to the initial grasp/object pose/forces on the object).  This would help isolate how much shape augmentation improves the policy compared to each of these randomizations (and when used in conjunction with). ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Summarizing the issues described in the summary of recommendation:\n\n1.  The RL algorithm used (PPO w/ and w/out demonstration) does not make a fair/useful comparison as this method is not designed to work with demonstrations.\n2. There is no direct comparison between shape augmentation and other forms of domain randomization (such as initial grasp/object pose/force perturbations), which could be used when learning a grasping policy instead. \n3. In addition to a better RL method to compare to (such as DAPG or simple Dagger/supervised learning), the paper should further highlight the advantages/differences to similar existing methods involving RL + learning from demonstration (such as DexMV and DexVIP). It is unclear whether these methods would perform better or worse than the grasping policy learned with shape augmentation given the same demonstration dataset/retargeting inputs. Mentioning the trade-offs/similarities to these approaches is not clearly discussed in the Related Work section (lines 269-270), and should be expanded further.\n\nI have updated my original review as these issues have been addressed by the authors in their rebuttal."}}, {"review_id": "kpjeVUZ6qvn", "reviewer": "Reviewer_RWRD", "summary": "This paper proposed a data-augmentation framework to learn dexterous grasp pose from point cloud observation. First, the DexYCB dataset which consists of human hand object interaction trajectories is used to provide an initial demonstration. Second, hand motion retargeting is utilized to convert the human hand motion to robot hand joint angles. Third, DIF-Net, an implicit neural network that can keep dense correspondence while deforming an implicit shape representation, is used to deform the original object mesh with sampled latent gaussian vectors. With correspondence after deformation, the grasp pose can also be modified according to the object. Then a rejection-sampling-based grasp refinement step is performed inside PyBullet simulator to eliminate physically-unfeasible grasp candidates. Finally, the author trained a PointNet++ to predict the palm pose and finger joints from point cloud input. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper proposes a data augmentation strategy for dexterous grasping and motivates the problem well in the introduction. The proposed object-centric deformation method is very general and agnostic to tasks and manipulators. There are also several issues that need to be addressed in the paper as follows:\n\n### Strength:\n\n1. The proposed method is simple and elegant. It does not rely on dynamics analysis of the contact wrench space or other force-based measurements. The dynamics correctness checking is achieved solely by a physical simulator while the underlying troublesome computation is hidden below. \n2. In principle, the implicit-based deformation network can generate infinite object mesh and corresponding grasp pose. This method is quite general.\n3. The observation that “appending object points with additional information regarding the153\nalignment between the robot hand and the local object surface” is beneficial for many robot-orientated regression tasks with point cloud input.\n\n \n### Weakness:\n\n1. The original demonstration dataset (DexYCB) seems not so useful in the whole pipeline, it only provided the initial object-grasp pair. The following data generated is achieved in a sample and reject fashion: sample with DIF-Net and reject with physical simulator. So the original dataset can be replaced by any method that can provide roughly okay grasp proposals, even if the grasp proposal itself is not dynamically-correct or successful. For example, the author can use any grasp proposal network, e.g. Contact Grasp Net, to generate a grasp pose and use the same data augmentation procedure for the following steps. In this case, the training data is not even limited to only YCB objects but any object meshes, which may lead to better generalization performance due to the increasing diversity of the training data. I assume that maybe one benefit of using human demonstration is for functional grasp when you do not want some grasp poses even if they are successful in simulator dynamically but not semantically feasible.\n\n2. The PPO baselines with **sparse reward** seem too weak. For manipulation tasks with a dexterous hand, sparse reward will lead to nearly zero success rate. In this task, writing a distance-based dense reward is also very simple and straightforward, especially the authors can train the algorithm inside a PyBullet simulator.\n\n3. The GraspIt baseline seems not used in a fair way: in the experiments, the author utilizes the results predicted by GraspIt and other heuristic to evaluate the grasp performance. However, the major contribution of this paper is the implicit shape-based augmentation, so a better way to compare GraspIt is as follows: i) replace DeYCB and implicit shape augmentation with GraspIt to generate diverse grasp poses ii) use PyBullet to reject sample the unfeasible grasp poses iii) train the same PointNet++ based on the data generated by GraspIt. Since the contribution is on the training data side, the GraspIt should also be used to generate training data but not evaluation. Otherwise the experiments may show the value of the Grasp Refinement for Dynamics Consistency but not Implicit Shape Augmentation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall this is a good paper because it \n1. focuses on important techniques, \n2. are very general to tasks and manipulators, and \n3. show large performance improvement in sim and real. In particular, the fact that the DIF-Net will also keep the semantic correspondence can extend the application of this paper to functional grasp, i.e. always hold the handle for a mug while deformation is used.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- More details about the baselines \n- More motivation on using the annotated dataset in the beginning\n- Figure to illustrate the symbols in Section 2.2. It is better to give a better visualization on how to estimate a new grasp with symbols, e.g. t^d annotated on the figure.\n- More clarification on comparison with the baseline method.\n"}}, {"review_id": "cWUlKfcAaK6", "reviewer": "Reviewer_KGmi", "summary": "The paper proposes a system, Implicit Shape Augmentated Grasping (ISAGrasp), to augment limited human demonstrations of dexterous grasps.The implicit shape augmentation is built on DIF-Net [11], a correspondence-aware implicit generative model. Novel shapes are generated via deformation and the resulting dense correspondences helps transfer the human demonstration to novel objects. The transferred grasps are refined via simulation and a grasp prediction model is trained on the augmented dataset by supervised learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Human demonstration of dexterous grasping is expensive to collect and always require specialized set up. The paper proposes an interesting way to extrapolate the limited demonstrations to a large dataset of novel objects.\n* The correspondence-aware generative model is used in a reasonable way, and the effectiveness of the method is shown by experiment. \n* The system transfers well to real world and achieves a decent result.\n\nWeakness:\n* Analysis of the results is insufficient. The authors only show that \"method A gets a better score than method B\", but doesn't really explain why.\n  * Why do the baselines, such as Heuristic and GraspIt perform so poorly? Are there any specific failure mode?\n  * Why doesn't data augmentation improve the performance on RescaledYCB? Is it because the distribution of augmented objects are different from the YCB dataset, or it's something else? \n  * I can understrand that augmentation can make the grasping policy generalize better to novel objects, but what are the source of remaining perfomance gap (~30% on 3 datasets)?  \n* The limitation part is not very satisfactory. I want to see more limitation on the algorithmic side, for example, what are the failure mode of the method and how can we further improve the success rate.\n\nQuestions:\n* I like the second term (relative poses) of the regarting optimization. My question is why do you choose the center of the object as anchor point, instead of '''closest point on object surface'. Isn't the latter one more robust to shape variation and better modeling the contact?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I think it's an interesting paper the attempts to address the core problem of data-driven dexterous grasping - how scale up the dataset. Personally I'm not fully convinced that augmenting the dataset with generative model is the way-to-go, for example, the performance on YCB-rescaled is not improved after data augmentation. But it could still be a contribution to the community, given that the authors can provide more interpretations of results.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "* In Sec 4.3, refinement rate is not explicitly defined.\n* Typos: L248, Figure8 -> Figure 8", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "J4bAfVPMvN", "reviewer": "Reviewer_kqPQ", "summary": "This paper studies the problem of dexterous grasping. It proposes to use correspondence-aware implicit deformation networks to propagate a small number of human grasp demonstration to grasping configurations on a diverse deformed object set. Trained with the generated objects and corresponding grasp configurations, the grasping policy can better generalize to unseen objects in human demonstration.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It's a smart idea to augment human grasping demonstrations with correspondence-aware shape deformation networks. It simultaneously creates novel shapes and the corresponding grasp configurations.\n2. Simulation experiments demonstrate the efficacy of the proposed shape augmentation in the comparison between last two rows in Table 1.\n3. This paper conducted real-world experiments of dexterous grasping, which makes the results much stronger.\n\nWeaknesses:\n1. Table 1: The comparison with baselines seems not very fair. Random, Heuristic and GraspIt! should be compared with transformed grasps Gd when performing the rejection sampling (133). Collect data with rejection sampling on augmented shapes using Random, Heuristic and GraspIt! with a same budget and compare the result.\n2. 248: what does refinement rate mean?\n\nMinors:\n220: baseelines -> baselines\nFig 8 caption: miss one '(' before \"Middle Panel)\", near the end of the second line", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea of this paper is very novel and interesting, but the comparison with baselines is not very fair so it's hard to tell how much gain does the proposed method bring. I believe this can be fixed so I vote for weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Fix the comparison with the baseline:\nCollect grasping data with rejection sampling on (augmented) shapes using Random, Heuristic and GraspIt!. Use a same budget as the proposed method and compare rejection rate and the final performance of same model trained on these different data."}}]

Author rebuttal:

Decision: Accept (poster)