Learning Object Affordance with Contact and Grasp Generation
Authors: Haoming Li
Xinzhuo Lin
Yang Zhou
Xiang Li
Jiming Chen
Qi Ye
Venue: Submitted to ICLR 2023 (2023)
Keywords: Object Affordance
Hand Grasps Generation
Conditional Variational Autoencoder
DLearning

Abstract: Understanding object affordance can help in designing better and more robust robotic grasping. Existing work in the computer vision community formulates the object affordance understanding as a grasping pose generation problem, which treats the problem as a black box by learning a mapping between objects and the distributions of possible grasping poses for the objects. On the other hand, in the robotics community, estimating object affordance represented by contact maps is of the most importance as localizing the positions of the possible affordance can help the planning of grasping actions. In this paper, we propose to formulate the object affordance understanding as both contacts and grasp poses generation. we factorize the learning task into two sequential stages, rather than the black-box strategy: (1) we first reason the contact maps by allowing multi-modal contact generation; (2) assuming that grasping poses are fully constrained given contact maps, we learn a one-to-one mapping from the contact maps to the grasping poses. Further, we propose a penetration-aware partial optimization from the intermediate contacts. It combines local and global optimization for the refinement of the partial poses of the generated grasps exhibiting penetration. Extensive validations on two public datasets show our method outperforms state-of-the-art methods regarding grasp generation on various metrics.

Under review as a conference paper at ICLR 2023
LEARNING OBJECT AFFORDANCE WITH CONTACT AND GRASP GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding object affordance can help in designing better and more robust robotic grasping. Existing work in the computer vision community formulates the object affordance understanding as grasping pose generation problem, which treats the problem as black box by learning a mapping between objects and the distributions of possible grasping poses for the objects. On the other hand, in the robotics community, estimating object affordance represented by contact maps is of the most importance localizing the positions of the possible affordance help the planning of grasping actions. In this paper, we propose to formulate the object affordance understanding as both contacts and grasp poses generation. we factorize the learning task into two sequential stages, rather than the black-box strategy: (1) first reason the contact maps by allowing multi-modal contact generation; (2) assuming that grasping poses are fully constrained given contact maps, we learn one-to-one mapping from the contact maps the grasping poses. Further, we propose penetration-awart partial optimization from the intermedi- ate contacts. combines local and global optimization for the refinement of the partial poses of the generated grasps exhibiting penetration. Extensive validations on two public datasets show our method outperforms state-of-the-ar methods re- garding grasp generation various metrics.
INTRODUCTION
Affordance is area studying how object can be used by an agent. Understanding affordance can help design better and more robust robotic systems operating complex and dynamic envi- ronments (Hassanin et al. 2021). For example, cup can be grasped and passed over by hand and a bed can be sat or slept onto by a human. Learning affordance (or affordance understanding) has wide applications like grasping (Bohg et al.) 2013). action recognition and prediction (Jain Koppula et al. 2013 Koppula & Saxena 2013), functionality understanding (Grabner 201T. social scene understanding (Chuang al. 2018) etc. In this paper, we focus object affordance for hands, i.e. hand-object interactions.
et al. 2016
al.
Though of great importance many applications, only several works about 3D grasp synthesis using deep learning (Corona et al., 2020; TTheri et al. 2020) Jiang et al.) [021] Karunratanakul al., 2020 Zhang et al. 2021 Taheri al. [2021) have been proposed the computer vision community. In (Taheri et al., 2020), a dataset for human grasping objects with annotations of full body meshes and objects meshes have been collected, and coarse-to-fine hand pose generation network based on conditional autoencoder (CVAE) is proposed. In (Karunratanakul et al. 2020). new implicit representation is proposed for hand and object interactions. The previous work (Taheri et al. 2021) takes step further to learn dynamic grasping sequences including the motion of the whole body given an object, instead of static grasping poses. Both these work defines affordance as possible grasping poses allowed by the objects. However, instantiations of affordance understanding can include affordance categorization, reasoning, semantic labeling, activity recognition, etc. (Deng
al. 2021
Among all these, semantic labeling of contact areas between agents and objects is found to be of the
most importance (Deng et al. 202 Roy & Todorovic 2016 Zhu et al. 2015) because localizing
the position of possible affordance can greatly help the planning of actions for robotic hands (Mo
et al. 2021 Wu et al. 2021 Mandikal & Grauman 2021
2022 In the robotics community, Mo

---

Under review as a conference paper at ICLR 2023
et al. (2021) and Wu et al. (2021) first estimate the contact points for parallel-jaw grippers and plan paths to grasp the target objects. For dexterous robotic hand grasping, recent works (Mandikal & Grauman 2021: 2022) find that leveraging contact areas from human grasp can improve the grasping success rate significantly in a reinforcement learning framework. However, they assume an object only has one grasp contact area and learn a one-to-one mapping from an object to the contact.
To overcome the limitation of work in both computer vision and robotics community, we propose to formulate the object affordance understanding as both contacts and grasp poses generation. Specifi- cally, we factorize the learning task into two sequential stages, rather than taking a black-box hand pose generative network that directly learns an object to the possible grasping poses in previous work. 1) In the first stage, we generate multiple hypotheses of the grasping contact areas, repre- sented by binary 3D segmentation maps. 2) In the second stage, we learn a one-to-one mapping from the contact to the grasping pose by assuming the grasping pose is fully constrained given contact map. Different from a coarse-to-fine strategy, our decomposition not only provides interme- diate semantic contact maps, but also benefits from the intermediate task learning in the quality of the generated poses. This intermediate task learning has been proven effective in many computer vision tasks [Tang et al. 2019 Wan et al. 2018: Tome al., 2017 Wu et al.. (017). In the robotic grasping, it is shown that optimizing grasping poses directly from contacts is superior to re-targeting observed grasps to the target hands (Brahmbhatt al. 2019b), which also motivates our choice.
Therefore, the other henefit of the intermediate contact representation is enabling the optimization from the contacts. Different from the optimization for the full grasps from scratch in (Brahmbhatt iet al. 2019b),, we propose a penetration-aware partial optimization from the intermediate contacts. combines of local and global optimization for the refinement of the partial poses of the generated grasps exhibiting penetration. The local-global optimization constrains gradients to affect only on the partial poses requiring adjustment, which results faster convergence and better grasp quality than global optimization.
In summary, our key contributions are (I) we formulate object affordance understanding as contact and grasp pose synthesis; (2) we develop novel two-stage affordance learning framework that first generates contact maps and then predicts the grasp pose constrained by the maps; (3) we propose penetration-aware partial optimization from the intermediate contacts for the grasp refinement; (4) benefiting from the first two decomposed learning stages and partial optimization, our method outperforms existing methods both quantitatively and qualitatively.
2 RELATED WORKS
Grasp Generation Human grasp generation is challenging task due to higher degrees of freedom of human hands and the requirement of the generated hands to interact with objects in physically reasonable manner. Most methods use models such as MANO (Romero et al. (017)) to parame- terize hand poses, aiming to directly learn a latent conditional distribution of the hand parameters given objects via large datasets. The distribution is usually learned by generation network models such as Conditional Variational Auto-Encoder (Sohn et al. 2015), or Adversarial Generative Net- works(Arjovsky et al. (2017). To get finer poses, many existing works adopt coarse-to-fine strategy by learning the residuals of the grasping poses in the refinement stage. [Cooona et al. (2020) uses a generative adversarial network to obtain an initial grasp, and then an extra network refine while Taheri et al. (2020) passes hand parameters to CVAE model and output an initial grasp, followed by a further refinement.
In recent work, however, Jiang et al. (022)) proposes to exploit contact maps to refine human grasps by leveraging the consistency of the contact map. Though estimating the hand-object contact maps, it only reasons about the contact consistency to refine the generated pose while our work exploits the contact maps as an intermediate representation for the final grasp generation. On the other hand, in the area of robotic grasping, BBrahmbhatt et al.) (2019)) introduces a loss for optimization using contact maps captured from thermal cameras (Brahmbhatt et al., 2019a 2020) to filter and rank random grasps sampled by Graspit! (Miller & Allen, 2004). It concludes that synthesized grasping poses optimized directly from the contact demonstrate superior quality to other approaches, kinematically re- te-aaggtiing observed human grasps to the target hand model. The contact maps also used in the hand and object reconstruction. Grady et al. (202T) proposes a differentiable contact optimization to refine the hand pose reconstructed from an image.
2

---

Under review as a conference paper at ICLR 2023
Point Clousi
Consact Map C
Hand Parameters
Final Grapp
Penetration Optimization
ContactCVAE
GraspNet
Figure 1: The framework of our method. It consists of three-stages: ContactCVAE, GraspNet and Penetration-awar Partial Optimization. ContactCVAE takes an object point cloud O as input and generates contact map C². GraspNet estimates grasp parameterized by 0 from the contact map C". Finally, penetration-awat partial optimization refines to get the final grasp.
Object Affordance for Robotic Grip Object affordance is not only used in generating human poses and hand poses, but also important for robotic grippers. Many robotic grasping works focus on parallel-jaw grippers. Yan et al. (2018) learns to rate proposed gripper pose to be a success or failure in data-driven way. Mousavian et al. 2019) uses a variational autoencoder(VAE) to gener- ate robotic grasps, and their method can directly work in the real world with only training purely in simulation. In contrast, in (Mo et al. 2021 Wu et al. 2021), locations for grasping points first predicted and the actions are planned to approach these points. Zhou Hauser (2017) increases the number of jaws of the gripper and estimate grasp quality using information collected from over- head depth images of novel objects using a modified convolutional neural network (CNN). In recent work for dexterous robotics hands (Mandikal & Grauman, 2021; 2022). contact maps from human grasping are used to guide the planning of the robotic hands based on reinforcement learning.
METHOD
Figure shows our method pipeline, which generates maps for contact areas by a network naming ContacıCVAA,, maps the contact maps grasping poses by the other network naming GraspNet and refines the generated grasp by a penetration-awan optimization module. In the work, we adopt MANO (Romero et al. 2017) to represent grasps. The MANO model A parameterizes the hand denotes the mesh vertices and faces) by the shape parameters E R and pose parameters / Ro1, i.e. M M(0 B). the work, we use the mean shape and use M = M(#) for brevity.
mesh M (V, F) (V E 7778 F E plsss
In the first stage, ContactCVAE aims learn a contact map distribution represented by latent vector given an input object by conditional variational autoencoder. The network takes an object point cloud O E R x3 and the contact map C E R as the input and learns make the output contact map C' E NN as close to the input contact map as possible. N the number of the points in O. Each point in the point cloud is represented by its normalized 3D positions. Each point in the contact map takes a value in [0, 1] representing the contact score. During inference, given an object, contact map C' can be generated by sampling from z. In the second stage, GraspNet learns a mapping from the contact map C' to the hand mesh M constrained by the map. The pose 6' of the predicted mesh M' from GraspNet refined with a imnetratton-awat partial optimization in the third stage.
3.1 CONTACTCVAE
Architecture Figure demonstrates the architecture of the ContactCVAL network, which a gen- erative model based on CVAE (Sohn et al. 2015). It consists of two blocks: a Condition-Enne Encoder and a Generator. Condition-Encode The Aandition-Encodee 2017]. It takes point cloud as input to extract both local features fi E RNx66 and global features, which are duplicated N times to make a feature map fg R. N for matching the shape of local featurefi. These two features are then concatenated as fig for the conditional inputs for the generator below.
Eggis built on PointNet (Qi et al.)
Generator Network The generator G&, follows an encoder-decoder architecture, As shown on the top of Figure 2 the encoder, Eg.: (C,Ō) - 2, is based on PointNet (Qi et al. 2017) architecture which takes both object point cloud O and contact map C as inputs and outputs the latent code 2 E R04 The encoder is only employed in training and is discarded the inference. The latent code represents sample of the learned distribution Q(=l)t, and is used to generate the contact
3

---

Under review as a conference paper at ICLR 2023
(a) Training Stage
Generator
Contatt Map
PointNer Encoder
Repeat
MI Dacader
Latent 2
MIP
PointNet Encoder
Point Cloud
Condition-
(b) Testing Stage (0.1)
Latent
Sampling
Repeat
Point Cloud
Condition- Encoder
Contact Map
The architecture of ContactCVAE. (a) In the training stage, it takes both object point cloud and contact map as input to reconstruct the contact map; (b) In the testing stage, by sampling from the latent distribution, generates grasp contacts with object point cloud as the conditional input only. @ means concatenation.
Figure 2:
map, where μr, o² denotes the mean and variance of the distribution. We then duplicate the latent code N times to make the latent feature fa for all the points.
The decoder Do, :(s', fig)) - C a classifier for a point which merges three different features (global fü. local ff and latent f!) to classify whether the point belongs to a contact map or not. The decoder Do. uses the MLP architecture and the weights are shared for all points.
Testing Stage During inference, as shown in the bottom of Figure 2 we only employ the Linditional-Eneodee and decoder Dø, A latent code randomly sampled from Gaussian distribution and forms the latent feature fz. At the same time, the Condition-Encee takes an ob- ject point cloud to output the global and local feature. With these features (Jz, fi), Do, outputs the grasp contact C" for the object.
Contact Loss The goal of training the model is optimizing 6.. 9a reconstruct the contact map well. We simplify the goal loss for model over all the points, name as Le1. However, some samples have small contact regions and it is hard for the model to learn those samples well by simply adopting the BCE loss. To address this problem, we additionally introduce the dice loss (Milletari et al., (2166) to train the model. It can assist the model in paying attention small target region. In work, adopt the dice loss for the same purpose and name as L22. the formulation the two loss is defined as:
binary classification task. Thus, we adopt the binary cross-entropy
La Eimaluslogtnn + -1 y:)log(1 - 9:)]
2 2E With
Les
21-0 +E
where d and y; represent the predicted contact and ground truth of point i, respectively.
Following the training of CVAE (Sohn et al., 2013), we use the KL-Divergence loss regularizing the latent distribution be close to standard Gaussian distribution. The loss term is named Lgt- The overall loss function of the ContactCVAE network, Leonfact, is represented as:
Leontact "ooLo1 + Ti Lea 72
(3)
where the %o 0.5, 71 0.5 and 7/- -e-3 are constants for balancing the loss terms.
3.2 GRASPNET
Architecture With the assumption of hands full constrained by a contact map, we adopt one-to-ome mapping function for getting the grasping pose from the generated contact from the first stage. As shown in the Figure 3 the model takes an object point cloud O and its generated (or reconstructed) contact C' as the input to predict the hand mesh for the grasping pose, which represented by the employ PointNet QQ al. 2017) ) to extract the feature, and then use a MLP with four hidden layers to regress the MANO parameters. Given the parameters, the MANO model forms differentiable layer which outputs the hand mesh M.
MANO model (Romero et al. 2017). Specifically,
4

---

Under review as a conference paper at ICLR 2023
Map C
Hand Mesh M
Contact Map C"
PeintNet Encoder
MLP
MANO
Point Cloud
The architecture of GraspNet. It takes the concatenation of the generated (reconstructed) object contact C" and the point cloud as input to predict the grasp mesh parameterized by MANO.
Figure 3:
During the training period, we use both ground truth and reconstructed contact map to train the GraspNet. During inference, we only use the generated contact map predict the grasp mesh. Both reconstructed and generated contact maps are from the ContactCVAF model in the first stage.
Grasp Loss We simply adopt the reconstruction loss (L2 distance) for the predicted vertices, named as Ly. We also use the chamfer distance Led between hands and objects, and penetration loss Lppr from (Taheri al. 2020: Jiang et al. 2021) which punishes penetrations between the hand and object. "The loss on MANO parameters is divided into two parts. We use the L1 loss for the translation parameter (8 E R²) and the geodesic loss (Mahendran et 2017) p for the pose parameter (6' E Rd*),, named as L, and Lp respectively.
Consistency Loss Similar (Jiang al., 2021). we introduce the contact consistency loss Lest ||C" C² Based on the distance between the object and the grasp mesh M, the contact map C" can be inferred. If the grasp mesh M predicted correctly from the GraspNet, the input contact map C" should be consistent with the contact map C
The overall loss of GraspNet, Lygasps the weighted sum all the above loss terms:
Lgrasy b Led + Aptr Le ApIp Lest where =355 Aod=20, 55 A0=0.1, A0= and =0.55 denote the corresponding loss weights.
(4)
3.3
CONETRATION-AWARR PARTIAL. OPTIMIZATION
Loss takes full effect on the target partial poses
Loss affects other partial poses of good quality
0
6¹
Peartrotion-aware
Optimization
Penetration detection
aL
Partially Refined
Globally
aL
a0p
Refined
d6
Figure 4: Left: our penetration-awan partial optimization. Penetration is detected in the index and thumb finger. Partial poses 0p for the index finger are pointed out. Right: the result of global optimization
Though GraspNet gives plausible grasps for most cases, the grasps may exhibit penetration, resulting in low grasp success rate in simulation. Hence, we propose to detect the penetration and refine the partial poses causing it while keep other partial poses of good quality unchanged. The full hand mesh is divided into six parts: five fingers and the palm. If penetration is detected in the palm area, all the poses adjusted. If penetration is detected in a finger part and penetration happens in the palm area, only the partial poses of the finger are adjusted. We use O, to represent the partial poses requiring adjustment and the refined poses 0% are obtained by
* = arg muin (Bp)) C²) wi Wi (M(0p), (M((()))))))) O) + we L1p. 67)).
(5)
dp
Tiast the consistency loss defined above. ("(0,) denotes the process of generating a contact map from the variable 0p. Lptr penalizes the penetration between the hand and object as similar in (Jiang et al. 2021| Karunratanakul et al., 2020). La makes the refined partial poses stay close to the prediction 6% from GraspNet. wp=0.01, W1 and =>.0.02.
Figure (Left) shows an example of our partial optimization for the poses 6, of the finger. In the refinement stage, the loss mainly results from local wrong partial poses, the global optimization arg ming L(0) (Right in Figure 4 has two issues 1) the gradient affects other good poses, 2) the gradient cannot take full effect the refinement for the wrong partial poses.
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENT
4.1 IMPLEMENTATION DETAILS
We sample N 2048 points trained using a batch size of 32 examples, and an Adam optimizer with a constant learning rate of le-4. The training dataset is randomly augmented with [-1, 1Jcm translation and rotation at three (XYZ) dimensions. All the experiments were implemented in Py Torch, in which our models ran 130 epochs in single RTX 3090 GPU with 24GB memory. In the Obman dataset (Hasson al., 2019), all the ground truth contact map is derived by normalizing the distance between the ground truth of hand and object. For the inference refinement (both partial and global optimization), the Adam optimizer with learning rate of 2.0 x 10 is used. In the refinement process, each input is optimized for 200 steps.
object mesh the input object point cloud. Our method is
4.2 DATASETS
Obman We first validate our framework on the Obman dataset (Hasson et al. 2019), which is a large-scale synthetic dataset, including 3D hand interacting with objects. The hands are generated by physical-based optimization engine Graspit! (Miller & Allen 2004), and are parameterized by the MANO model (Romero et al., 2017)- The dataset contain 8 categories of everyday objects selected from ShapeNet (Chang et al., 2015) with total of 2772 meshes. The model trained on this dataset will benefit from the diversified object models. The object contact map is derived as [Theri et al. 2020 by thresholding the normalized distance between the object points and their nearest hand vertices. Points with the distance smaller than a threshold are marked as contact.
The ContactPose dataset (Brahmbhatt et al. 2020) is real dataset for studying hand- object interaction, which captures both ground-truth thermal contact maps and hand-object poses. Though the dataset contains only 25 household objects and 2306 grasp contacts, it captures more real interactions. For example, the contact in ContactPose spreads across large sections of the hand, as opposed that at the fingertips for most cases in Obman. We manually split the dataset into a training and test group according to object type. Specifically, we use 4 objects (cup, toothpaste, stapler, and flashlight) with 336 grasp contacts as test set, and the rest for training the model. ContactPose uses the thermal camera- based method to capture the contact region.
ContactPose
4.3 EVALUATION METRICS
good generated pose should be physically stable and should be in contact with the object without Penetration. In this work, we adopt three metrics to evaluate the quality of generated grasp poses: (1) Penetration The penetration is measured by the depth (Dep, em) and the volume (Vol, cm >) be- tween the objects and generated hand meshes. The depth is the maximum mean of the distances from the hand mesh vertices to the surface of the object if a penetration occurs. Following (Jiang et al.) 2021 Karunratanakul et al. 2020). the volume is measured by voxelizing the hand-object mesh with voxel size 0.5cm. (2) Simulation Displacement (Sim-Disp) The simulation displace- ment is adopted to measure the stability of the generated grasp. We report the average (Mean, em) and variance (Var, cm) of the simulation displacement as measured by a physics-based simulator following the same settings as (Jiang al. [202]] Karunratanakul et al. (020). The displacement is the Euclidean distance between the object centers before and after applying a grasp on the object. Though used in the existing work, the results of previous work (Karunratanakul al. 2020) indicate that a high penetration might correspond to low simulation value and therefore we suggest readers use it for rough reference only. (3) Contact Rate (CR, %) A physically plausible hand-object interaction requires contact between the hand and the object. We define sample as positive if the hand-object contact exists, which means that there exists at least a point on the hand surface is on or inside the surface of the object. The contact rate is the percentage of those positive samples over all the test samples.
In addition to the metrics used in the hand generation work, we introduce two more metrics to evaluate the quality of the grasping pose distributions. (4) Grasp Success Rate (Sim-SR, %) The grasp success rate aims evaluate the rate of grasp success. Specifically, sample as the one with penetration-von 5cm and simulation- mean < 2cm. The success rate is the percentage of those positive samples over all the test samples. (5) Diversity (Dix, em) It is also
define the positive

---

Under review as a conference paper at ICLR 2023
Table I: Quantitative comparison on Obman test set.
Penetration (
Sim-DDsp
Methods
CR (?)
(1)
Sim-SR (7)
Dep
Vol
Grahnel (Taheri et al.
2020
8.41
1.66
98.25
7.93
27.60
GraspPield Kanunaammaka
2020
0.56
6.05
2.07
1.2.81
89,40
Frag af 2021
0.46
5.12
1.52
22.29
99.97
Pash (Baseline) refine Ours global refine Ours
0.70
13.07
1.75
1.2.88
98.56
8.31
21.63
0.50
4.15
2,00
+2.50
97.01
8.89
58.89
0.44
3,94
1.74
#:2.28
100.00
10.18
61.37
GT Ours (GT)
0.01
1.70
1.66
#2.44
100.00
7.86
87.12
(GT)
1.97
0.36
3.75
1.98
#2.46
99.86
8.16
57.19
Table 2: Quantitative comparison on ContactPose test set.
Penetration
Sim- (L)
Methods
(?)
Div (1)
Sim-SR
Dep
Vol
Mean
GrahNeelcuamsenne)) (Taheri
2020
1:5.4
97.51
GrabNet(refinenet)
Tahen el
2120
0.92
16.74
1.04
1.60
97.42
5.92
16.89
Param2Meshi (Gaseine)
1.02
19.18
1.14
1:1.9
81.84
6.37
13.19
Ours w/o refine Ours global refine
0.73
10.06
1.08
11188
98.85
7.40
24.71
0.70
7.49
1.58
1.79
96.23
7.56
32.07
Ours
1.23
1.85
98.85
7.66
38,08
1.09
Ours wlo refine (GT) Ours (GT)
0.60
9.54
0.95
+0.97
100.00
6.79
33,93
0.51
5.91
1.06
1:1.1
99.65
7.03
39.12
significant to evaluate the diversity for the generation task. In this work, we use MAE to measure the diversity of generated results. Specifically, we measure the divergence between each generated sample and all other samples and then average them. Referring to the research (Guo et al. 2020).
the formulation is defined as Eq. 6
N Ng
Diversity
1/
14-uukle
uxl12,
(1 k)
(6)
(N,
i= Ng is the number of generated samples. Ui represent the i-th and k-th generated sample.
k=
4.4 SELF COMPARISON
To verify the contribution of our proposed factorization, we construct three variants of our method. Param2Mesh Baseline: A baseline for grasp generations. It learns the latent distribution of MANO
parameter directly, which similar to the first stage of the previous work (Taheri et al., 2020). Specifically, use the same CVAE of our Contact2Grasp to make fair comparison but replace the encoder Eg_of of our ContactCVAE with fully connected layers to take the MANO parameters as inputs. Given 3D object point cloud and random sample for the distribution, the decoder Do generates MANO parameters directly. Also, the coarsenet (Taheri et al., 2020) can be served as bascline as well but with a different architecture. Ours w/o refine A variant of our method removing the final testing time refinement. Ours global refine: A variant of our method adopting global refinement the third stage. Ours (GT) shows GraspNet in the second stage trained and tested on GT contract maps and Ours w/o refine (GT) without the refinement. By comparing to this variant, we are able to verify the quality of our generated contact maps from ContactCVAI.
Effectiveness of Two Stage Learning Factorization Results both Obman (Table 1 and Con- tactPose datasets (Table 2 show our proposed method can improve Param2Mesh significantly on all metrics, indicating the effectiveness of the two stage factorization. The penetration-we improve by 60% (Ours w/o refine) and 70% (Ours) on Obman. A reasonable grasp pose should embody both low penetration and simulation displacement. Thus the Sim-SR is more comprehensive met- ric considering both of them. It can observed that our method shows the best Sim-SR result on Obman (61.37%), improving Param2Mesh by 118%.
On ContactPose, we also compare our method with other variants in diversity. Our method gener- ates diverse samples than the Param2Mesh baseline and the variant using ground truth contact

---

Under review as a conference paper at ICLR 2023
(Test)
ContactPose (Test)
Failure Cases
Figure 5: The visualization of generated contact and grasp for objects from Obman test set and ContactPose test set. We also present some failure examples. For each example, we present both the predicted grasp pose (left) and the corresponding contact map (right), which is presented in the form of heat map. Note that all the contact map generated from our method.
maps. For the latter, in the second stage only a one-to-one mapping is learned, the diversity actu- ally measures the ground truth poses while our method in the first stage can generate new samples and therefore it is reasonable have more diverse samples than the ground truth poses.
Effectiveness of Penetration- aware Partial Optimization Results on Obman (Table I and Con- tactPose (Table 2 indicate that partial optimization strategy (Ours) achieves better performance than global refinement (Ours glob refine) over all the metrics, which presents the effectiveness of the proposed method.
As shown in Figure 5 we can observe that the generated contact map is reasonable, corresponding the grasp pose. Although there are some failure examples (including unstable grasps and serious penetration), the hand pose is substantially natural as human behavior.
Quality of Generated Contact Maps When compared with Our (GT) and Our w/o refine (GT), the metrics for penetration of our methods are worse but the margin is relative small compared to our improvement from the baseline. For example, on Obman Table T penetration depth of our methods are 0.45mm and 0.44mm while those of Our (GT) and Our w/o refine (GT) are 0.43mm and 0.36 0.36mm. The comparison indicates the generated maps in the first stage are of high quality. The examples in Figure also show that the generated contact convey meaningful information for grasping.
Semantic Analysis of Latent Contact Map Space Using the generated object contacts to formulate the hand grasp is one of contributions and here show whether Contact2Grasp can learn the latent distribution for contact well. In the point generation work (Achlioptas et al.) 2018). it demon- strates the quality of the generation model by showing that the learned representation is amenable intuitive and semantically rich operations. Inspired by the work, we conduct the semantic analysis of the latent space learned from our Contact2CVAE model as detailed below.
First, we use ContactCVAB to generate N contact maps and their corresponding latent z. Second, cluster contact maps using mean shift and adopt the dice metric measure the distance between different contact maps. Then we select a sample from each cluster set randomly. Each sample con- sists of contact map and its corresponding latent z. Then, we interpolate between the latent z of different samples get a new latent and then the new latent is fed into the decoder of ContactC- VAE to get its corresponding contact map. In the paper, we interpolate latent z and therefore get 2 contact maps. Finally, their contact maps are fed into GraspNet to get their corresponding grasp poses.
Figure 6 exemplifies two pairs of the types selected and the contact maps and grasping poses inter- polated between them. Notice that the grasps change gradually in alignment with the contact maps between Type A and Type B. For example, in Figure (b), (b), the yellow arrow and circle denote small differences between the contact maps and the grasp poses. As the contact region gradually appears, the middle finger moves the corresponding position smoothly. Similar interesting observation can
8

---

Under review as a conference paper at ICLR 2023
(a)
BOCK
PACO
TypeA
TуpeB
(b)
TypeA
TypeB
Figure 6: Interpolated contact maps and grasps between different types generated contacts (Type and TypeB). Note that the grasping poses (e.g. finger positions denoted in the yellow circle and arrow) change with transitions between two types of contacts.
be found for manipulating the phone Figure (a) where the hand poses change from holding to pressing gradually.
4.5 COMPARISON WITH EXISTING WORK
When compared with the state-of-the-arts. our method achieves the least penetration, highest grasp success rate, highest contact rate and highest diversity on both Obman and ContactPose. The im- provement can be attributed to the factors as follows. Existing work (Taheri et al. [2200 Jiang et al., 202T) aims learn the mapping relationship between the object and grasp pose distribution directly, e.g. point clouds to joint rotations. Such mapping relationship is highly non-linear. In comparison, our proposal decomposes the hard problem into two tasks by introducing intermediate contact maps. The point clouds of object and the contact areas represented by the segmentation maps the point clouds are aligned in the same 3D space, which is relatively easy to learn. In ContactCVAE, we also make full use of the alignment and design local point feature to point classification mapping in the structure.
Limitations Although method achieves significant performance, there exists limitations. The major limitations of our method can be summarized in two aspects. (1) The two-stage learning: The error from the first stage cannot be corrected. If generated contact map from the first stage of low quality, the resulting grasp may be influenced as well. (2) Ambiguity of Contact Map:
In some cases, the contact maps are ambiguous, which results in more than one plausible grasping pose. Therefore, the assumption of the one-to-one mapping in the second stage does not hold. To address the limitations mentioned above, some solutions can be explored as the future work. (1) Contact Filter: Before inputting the generated contact maps to the GraspNet, a clustering algorithm (e.g. mean shift) can be adopted to filter those deviating too far from the distribution. (2) More detailed contact maps: To address the ambiguity problem, we can introduce more-detailed contact maps, which contains contact areas with different parts the hands providing stronger constraints.
5 CONCLUSION
In this paper, we propose a novel framework for human grasps generation, which holds the potential for different deep architectures. The highlight this work is exploiting the object affordance rep- resented by the contact map, to formulate a functionality-oriente grasp pose and using penetration- poses without hurting good-quality ones. The
aware partial optimization to refine proposed method is extensively validated two public datasets. In terms of diversity and stabil- ity, both quantitative and qualitative evaluations support that, our method has clear advantages over other strong competitors in generating high-quality human grasps.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International Conference on Machine Learn- ing. pp. 40-49, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pp. 214-223, 2017. 2017.
Jeannette Bohg, Antonio Morales, Tamim Asfour, and Danica Kragic. Data-driven grasp synthe- sis-a a survey, IEEE Transactions robotics, pp. 289-309, 2013.
Samarth Brahmbhatt, Cusuh Ham, Charles Kemp, and James Hays. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8709-877 2019a.
Samarth Brahmbhatt, Ankur Handa, James Hays, and Dieter Fox. Contactgrasp: Functional multi- finger grasp synthesis from contact. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2386-2393, 2019b.
Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: dataset of grasps with object contact and hand pose. In European Conference on Computer Vision, pp. 361-378 2020.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rie 3d model repository. arXiv preprint arXiv: 15 1512.03022 2015.
Ching-Yao Chuang, Jiaman Li, Antonio Torralba, and Sanja Fidler. Learning to act properly: Pre- dicting and explaining affordances from images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 975-983, 2018.
Enric Corona, Albert Pumarola, Guillem Alenya, Francese Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5031 5031-5041, 2020.
Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3d affordancenet: A bench- mark for visual object affordance understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1787- 2021.
Helmut Grabner, Juergen Gall, and Lue Van Gool. What makes a chair a chair? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1529-1536, 2011.
Patrick Grady, Chengeheng Tang, Christopher D Twigg, Minh Vo, Samarth Brahmbhatt, and Charles C Kemp. Contactopt: Optimizing contact to improve grasps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1471-1481 2021.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 2021-2023 2020.
Mohammed Hassanin, Salman Khan, and Murat Tahtali. Visual affordance and function understand- ing: A survey. ACM Computing Surveys, pp. 1-35, 2021.
Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11807-11816, 2019.
Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn Deep learning on spatio-temporee graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5308-5317, 2016.
10

---

Under review as a conference paper at ICLR 2023
Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human gräsps generation. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 11107-11116, 2021.
Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In 2020 International Conference on 3D Vision. pp. 333- 333-344,70 2020.
Hema S Koppula and Ashutosh Saxena. Anticipating human activities using object affordances for reactive robotic response. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp- 14-29, 2015.
Hema Swetha Koppula, Rudhir Gupta, and Ashutosh Saxena, Learning human activities and object affordances from rgb-d videos. The International journal of roboties research, pp. 951-970, 2013.
Siddharth Mahendran, Haider Ali, and René Vidal. 3d pose regression using convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2174-2182, 2017.
Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In IEEE International Conference Robotics and Automation, pp. 6169-6176, 2021.
Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference Robot Learning, pp. 651-661, 2022.
Andrew T Miller and Peter K Allen. Graspit! versatile simulator for robotic grasping. IEEE
Robotics & Automation Magazine, pp. 110-122, 2004.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. In International Conference on 3D Vision, pp. 565-571, 2016.
Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6813-6823, 2021.
Arsalan Mousavian, Clemens Eppner, and Dieter Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2901-2911 2019.
Charles Qi. Hao Su. Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652-660, 2017.
Javier Romero, Dimitris Tzionas, and Michael Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions Graphics, 2017.
Anirban Roy and Sinisa Todorovic. A multi-scale enn for affordance segmentation in rgb images. In European conference on computer vision, pp. 186-201, 2016.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Omid Taheri, Nima Ghorbani, Michael Black, and Dimitrios Tzionas. Grab: A dataset of whole- body human grasping of objects. In European conference on computer vision, pp. 581-600) 2020.
Omid Taheri, Vasileios Choutas, Michael J Black, and Dimitrios Tzionas. Goal: Generating 4d whole-body motion for hand-object grasping. arXiv preprint 2rXiv:21 2021.
Danhang Tang, Qi Ye, Shanxin Yuan, Jonathan Taylor, Pushmeet Kohli, Cem Keskin, Kyun Kim, and Jamie Shotton. Opening the black box: Hierarchical sampling optimization for hand pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 2161- 2175, 2019.
11

---

Under review as a conference paper at ICLR 2023
Denis Tome, Chris Russell, and Lourdes Agapito. Lifting from the deep: Convolutional 3d pose estimation from single image. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2500-2509, 2017.
Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. Dense 3d regression for hand pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5147-5156, 2018.
Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, Bill Freeman, and Josh Tenenbaum. Marmet: 3d shape reconstruction via 2.5 d sketches. Advances in neural information processing systems, 30, 2017.
Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang. Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. arXiv preprint arXiv:2106. 14440, 2021.
Xinchen Yan, Jasmined Hsu, Mohammad Khansari, Yunfei Bai, Arkanath Pathak, Abhinav Gupta, James Davidson, and Honglak Lee. Learning 6-dof grasping interaction via deep geometry-awan 3d representations. In IEEE International Conference on Robotics and Automation, pp. 3766- 3773, 2018
He Zhang, Yuting Ye, Takaaki Shiratori, and Taku Komura. Manipnet: neural manipulation synthe- sis with hand-object spatial representation. ACM Transactions Graphics, pp. 1-14, 2021.
Yilun Zhou and Kris Hauser. 6dof grasp planning by optimizing deep learning scoring function. In Robotics: Science and systems workshop on revisiting contact-turning a problem into solution, pp. 2017.
Yixin Zhu, Yibiao Zhao, and Song Chun Zhu, Understanding tools: Task-oriented object modeling. learning and recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2855-2864, 2015.
12


Reviews:
[{"review_id": "pqZLiVbmYt", "reviewer": "Reviewer_8WmN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper touches on an important problem in embodied AI, object affordance understanding. The authors formulate the object affordance in a grasp pose generation paradigm. The grasp pose is predicted by a three-stage pipeline: A contact map is first predicted by a cVAE and then is input to a GraspNet to obtain the hand parameters. The final grasp pose is obtained by Penetration-aware Optimization over the hand parameters.", "summary_of_the_review": "Overall, the proposed method is technically solid, though it lacks technical novelty and does not outperform Jiang et al. significantly.  I would consider increasing my rating if the authors could address my concerns mentioned in the weaknesses section.\n", "strength_and_weaknesses": "\nStrength: \n1.\tThe proposed approach is technically solid. The subcomponents are mainly based on some existing techniques, such as cVAE and PointNet.\n2.\tThe paper is clearly written, and the illustrations are easy to understand.\n\nWeaknesses:\n1.\tThe proposed approach seems to be a combination of the existing methods (e.g., contact map generation, grasp pose prediction) and lacks technical novelty. \n2.\tThe ablation study should include Ours w/o ContactVAE. \n3.\tThe proposed method does not outperform Jiang et al. significantly. In Penetration(Dep), In CR, Ours(100.00%) only outperforms Jiang et al.(99.97%) for 0.03%. In Sim-Disp(Mean), Ours(1.74) is even worse than Jiang et al.(1.52). \n4.\tBesides, the comparison between Jiang et al. and Ours seems unfair since Jiang et al. does not use test time refinement. If we compare Ours wo refinement with Jiang et al., Ours wo refinement only achieves slightly better results in Penetration(Dep) and CR.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZZ8phhNpUce", "reviewer": "Reviewer_7dNk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a three-stage approach for generating grasps for a MANO human hand model given a point cloud of an object of interest. This is achieved by modeling object affordances as the task of predicting contact maps and then finding hand parameters that best achieve the desired contacts.\n\nFirstly, a ContactCVAE is used to sample a contact map given the object point cloud. The ContactCVAE labels each point in the point cloud with a contact score to get the contact map, and is trained to generate the ground truth contact map input during training. Secondly, the predicted contact map along with the object point cloud is fed into a GraspNet to predict the parameters for the MANO hand model. As the hand configuration is very high dimensional (63 dimensional), there may be errors in the prediction which lead to penetration of the fingers or the palm into the object. Thus, the authors propose to use penetration-aware partial optimization in the final stage of the proposed approach to refine the hand parameters predicted by the GraspNet.\n\nThrough experiments on two datasets, the authors demonstrate that their approach is able to outperform baselines on various metrics including penetration, simulation displacement, and contact rate. Additionally, ablations show that the partial optimization is important for improving performance.", "summary_of_the_review": "The paper presents a sensible solution to the problem of determining grasping poses by firstly predicting contact maps and then optimizing a grasp pose for the desired contacts. This is in contrast to existing work which directly predicts a grasp pose and then attempts to refine it. The different components of the proposed framework are well-motivated and easy to follow.\n\nAlthough the results demonstrate improved performance over existing work, the overall architecture of the framework is not particularly novel. In addition, several details of the paper remain unclear which should be addressed including the diversity metric and generalization capabilities.\n", "strength_and_weaknesses": "**Strengths**\n- The proposed framework is intuitive and easy to follow. Breaking down the problem of generating hand grasps to first predicting contact maps is a sensible approach and is representative of an affordance-based representation.\n- The penetration-aware optimization is a straightforward yet effective approach for refining and correcting any errors in the grasp predicted by the GraspNet.\n- The experiments demonstrate that the proposed approach results in higher quality grasps in comparison to current SOTA and ablated baselines.\n- The qualitative analysis of the interpolation between the latents sampled from the ContactCVAE is interesting and demonstrates that the latent space has incorporated useful information.\n\n**Weaknesses**\n- Jiang et al. 2021 also use a CVAE but directly predict the hand parameters. In addition, they also learn a ContactNet to predict the contact map given the hand and object point cloud and then refine the predicted grasp using optimization. The proposed framework feels quite similar to Jiang et al.’s approach, with the main contribution being the reparametrization of the problem as learning to predict intermediate contact maps and a more constrained grasp refinement step. The results demonstrate incremental improvement over this baseline. In addition, why was Jiang et al.’s approach not included in Table 2?\n- The writing quality of the paper could be improved and made more consistent. Some arguments are not particularly well-supported. For example:\n    - It is arguable that existing work treats object affordance understanding as a “black box”. The models and losses are designed to have the right inductive biases to ensure that learning is amenable. Perhaps it is more appropriate to call approaches that learn the direct mapping from object representations to grasping poses as “end-to-end”.\n    - The abstract and introduction makes it seems like there is a distinct difference in affordance prediction between the computer vision and robotics community. This is not particularly true, as there is a large body of work in robotics that directly predicts grasps and forgoes precise contact mapping.\n- The coefficients and terms in the loss functions could be justified better. As it stands, they seem to rely on the fact they were used in previous work.\n    - For example, the reconstruction loss for the predicted vertices, chamfer distance and penetration dominate the loss for GraspNet. Since this is the case, how important of a role do the other loss terms play?\n- In the experimental section, it is unclear how well the framework generalizes to out-of-distribution objects or categories. For the ContactPose dataset, 4 out of the 25 objects are used for testing while the rest for training. It’s not clear to me how that is actually reflected in the metrics or how different those objects are from the training set (e.g. grasping a hammer by the handle is similar to how you might grasp a flashlight by its handle).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DCZuX7_4pQB", "reviewer": "Reviewer_3pXU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Given the complete point cloud of an object, how can a vision model infer the right grasp pose for a human hand such that the object can be grasped stably? The paper tries to address this issue via a staged pipeline. First, it trains a generative model (conditional VAE) to reconstruct the ground-truth contact map of the input object point cloud. Second, given an object point cloud, we can first use the decoder in cVAE to sample a possible contact map, and then train another network to predict the hand grasp pose conditioned on the predicted contact map. The main difference compared to prior works (end-to-end, single stage) on this line is the two-stage training: first predict the contact map, then predict the grasp pose. ", "summary_of_the_review": "The paper presents an intuitive two-stage prediction pipeline for inferring how a hand can grasp objects and shows improvements over prior works on several metrics. The method is quite simple and straightforward. Considering the prior works on this exact problem as well as many works on parallel-jaw grasping, this paper is more like an incremental work. From a robotics perspective, I would also like to see how the defined metrics connect to the performance of an actual downstream task (such as grasping in a physically realistic simulator). And more details on how the metrics are measured would be necessary. It would make the paper much stronger if the paper can include well-designed grasping tests even just in a simulator.", "strength_and_weaknesses": "**Strength**:\n\n* I think the idea of first predicting the contact map before predicting the grasp pose is intuitive and natural. The design choices made in the paper in order to achieve this two-stage prediction are also quite straightforward. For example, to process the point cloud, the paper uses PointNet. To train the contact map prediction network, the paper uses a conditional VAE, as there exist many possible contact maps. \n\n* The paper found that when optimizing the hand pose to avoid penetration, it's better to isolate the optimization on the fingers and on the palm. Local optimization avoids affecting other regions of the hand. This is a potentially useful observation for other robotic hand problems. \n\n\n**Weakness**:\n\n* The paper has few novel technical contributions other than the idea of first predicting the contact map in order to predict the hand pose. Most of the techniques or the loss function designs used in the paper are from prior works and are quite straightforward (in the sense that other people who come to the problem are very likely to come up with a similar solution). On the one hand, it's good that the pipeline is simple and may be easily reproduced. On the other hand, the paper is not technically novel.\n\n* While the paper presents a pipeline to infer the contact map for different objects and some plausible hand grasp poses, there exists a significant gap to apply such models to a real/simulated robotic setting. For example, the paper considers a grasp pose good if it satisfies constraints such as no/small penetration and small object movement. However, to actually grasp the object with a robot hand, even in simulation, contact physics matters, small prediction errors on the pose can lead to drastically different contact forces and then lead to immediate grasp failure, etc. In addition, the assumption of having access to the complete noise-free point cloud can be a problem in real robotics settings as we typically get a partial view of an object from a camera. More works need to be carried out in order to apply such models to more realistic robotic settings.\n\n* Some of the evaluation metrics are not convincing. For example, the `grasp success rate` is defined by the simulation displacement and the amount of penetration. It is not a good definition for grasp success rate. As mentioned above, low-level physics matters. It's unclear if the prediction satisfies penetration-vol< $5cm^3$ and simulation-mean < 2cm, then it can actually enable a successful grasp. Where do these threshold values come from? Why not just place the object on a table in a physics simulator such as pybullet or mujoco and then actually have the simulated hand execute the predicted grasp pose, and see if the object can be stably grasped (one can check it by lifting the hand and shaking it, and see if the object drops). This would be a proper way to measure the grasp success rate. I would like to see the true grasp success rates in the tables. Similarly, for the definition of `contact rate`, is one contact point between the hand and the object sufficient for a good grasp?  If not, this metric would be less useful.\n\n* The paper does not provide enough details on the evaluation details. As metrics are a critical part of the evaluation, it's better to explain the exact procedures to get the quantitative results. For example, what simulator is used to check the `Simulation Displacement`, and how are the objects and the hand are spawned in the simulator, and how is the displacement measured, how many simulated steps are waited for before measuring the displacement, etc.\n\n* Even though the paper reports that the proposed method outperforms the baselines on the defined metrics, it's unclear how much does such improvements contribute to better performance on a downstream task such as grasping. For example, when the maximum penetration depth is reduced from 0.56 to 0.44cm, how much improvement can one expect to get if the hand executes such poses for grasping the objects? One point I am trying to say is that the connection between the reported metrics and the actual physics simulation/motion is too weak. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-OPJQSr4bw", "reviewer": "Reviewer_CDdw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors proposed a new framework for human grasps generation for holding objects in hand. Instead of directly learning mapping from object point cloud to joint angles of hand, authors divide this whole problem into 2 steps. In the first stage, authors have tried to learn ContactCVAE, which given point cloud representation of an object and random latent vector, tries to figure out contact segmentation masks for objects. In the second stage, based on the generated contact segmentation mask, joint angles of hands are being learned using GraspNet. The results of GraspNet are further optimized based on penetration aware partial optimization. Authors have shown that this approach yields better results than single stage approach. To support their findings authors have shown both quantitative and qualitative results on Obman & ContactPose dataset. ", "summary_of_the_review": "I have provided `marginally above the acceptance threshold` rating to the paper. In this paper authors have divided 1 stage approach into 2 stage approach which have shown improvement. However I feel, novelty wise it's an incremental improvement. To show the efficacy, authors have carried out extensive empirical experiments. Also writing wise paper needs good amount improvement to make it more readable.  ", "strength_and_weaknesses": "### Strength\n* Decomposed the problem into 2 stages and have shown its effectiveness over one stage approach. \n* Carried out extensive experiments to show the efficacy of approach. \n* Shown the effectiveness of latent space to interpolate intermediate grasps. \n\n### Weakness\n* Writing can be improved a lot. Sometimes it becomes really hard to follow the paper.\n* One-to-one mapping from affordance segmentation to hand joint angles might not be true in many cases. This is also being mentioned in the limitation of paper.\n* In Table 1 & 2, it looks like `Ours` has better results than `Ours(GT)`, which is very counter intuitive. In the paper it is mentioned that generated maps produced by the proposed approach are of high quality which might be the reason behind it. It would be great to see some visualization of GT vs predicted affordance mask to see the difference.\n* In figure 5, the last row, first column (for hand holding coffee mug), predicted pose and corresponding contact map does not match. It feels that something is off there. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject