Fusing Priori and Posteriori Metrics for Automatic Dataset Annotation of Planar Grasping
Authors: Hao Sha
Lai Qianen
Hongxiang Yu
Rong Xiong
Yue Wang
Venue: CoRL 2022 Poster (2022)
Keywords: Grasp Detection
Grasp Metric
Grasp Dataset
Automatic Annotation

Abstract: Grasp detection based on deep learning has been a research hot spot in recent years.  The performance of grasping detection models relies on high-quality, large-scale grasp datasets. Taking comprehensive consideration of quality, extendability, and annotation cost, metric-based simulation methodology is the most promising way to generate grasp annotation. As experts in grasping, human intuitively tends to make grasp decision based both on priori and posteriori knowledge. Inspired by that, a combination of priori and posteriori grasp metrics is intuitively helpful to improve annotation quality. In this paper, we build a hybrid metric group involving both priori and posteriori metrics and propose a grasp evaluator to merge those metrics to approximate human grasp decision capability. Centered on the evaluator, we have constructed an automatic grasp annotation framework, through which a large-scale, high-quality, low annotation cost planar grasp dataset GMD is automatically generated.

Fusing Priori and Posteriori Metrics for Automatic Dataset Annotation of Planar Grasping
Hao Sha Qianen Lai Hongxiang Yu Rong Xiong Yue Wang"
Control Science and Engineering of Zhejiang University {shahao, laiqe, hongxiangyu, rxiong, jrangg4]@zju.edee is the corresponding author
Grasp detection based on deep learning has been a research hot spot in recent years. The performance of grasping detection models relies on high- quality, large-scale grasp datasets. Taking comprehensive consideration of qual- ity, extendability, and annotation cost, metric-based simulation methodology is the most promising way to generate grasp annotation. As experts in grasping, hu- man intuitively tends to make grasp decision based both on priori and posteriori knowledge. Inspired by that, a combination of priori and posteriori grasp metrics is intuitively helpful improve annotation quality. In this paper, we build hy- brid metric group involving both priori and posteriori metrics and propose a grasp evaluator to merge those metrics to approximate human grasp decision capabil- ity. Centered on the evaluator, we have constructed an automatic grasp annotation framework, through which a large-scale, high-quality, low annotation cost planar grasp dataset GMD is automatically generated.
Abstract:
Keywords: Grasp Detection, Grasp Metric, Grasp Dataset, Automatic Annotation
Introduction
Grasping serves as a basic yet important skill in the robotic manipulation domain since its reliability is always the prerequisite for the following complex tasks. To equip robots with the ability to implement feasible grasps freedom (DoF) planar grasp detection, learning-based algorithms have been the mainstream, which leads to intense demands for large-scale, high-quality, low annotation cost grasp datasets. There mainly three kinds of methodologies for grasp dataset generation: hand-annotated methodology, physical trial methodology, and metrics-base based simulation methodology. The hand hand-annotated method means that annotations are given directly by experts or crowd-sourcing according to their experience, such as the typical Cornell dataset [1]. Grounded on the hypothesis that human intuitively knows dataset is believed with high quality. However,
scene, grasp detection becomes critical problem. For 4 Degree of
what successful grasp look like, the the huge consumption of human labor and time limits the dataset scale. Physical trial methodology means that annotations
collected from grasp attempt results through semi-automatic trials real robots [2, 3]. Although close to ground truth since the training set and testing set share the same domain, real robotie grasping trial: i) relies on solid device support (such as the robot farm [3)) or huge time consumption: ii) is always bounded with specific robot setup leading to difficulties in generalization; iii) cannot be fully automatized which limits its scale, since human interventions are needed for specific cases. Metrics- metries [4] simulation to evaluate the quality of candidate grasp as annotations for synthetic data. Even though the reality gap degrades its accuracy and generalization performance, low cost and high efficiency extension make the metrics-base method popular [5, 6, 7]. Taking comprehensive consideration of quality, extendability and annotation cost, the metrics- based simulation method is the most promising way to generate grasp annotation [8] (Figure shows the comparison).
simulation methodology means calculating various grasp
To improve the quality of the dataset generated through metric-based simulation methodology. more efforts should be put into grasp metrics to mimic human grasp decisions. Human tends to make grasp decision based both on the priori knowledge (physical analysis) and the posteriori knowledge (previous experience). Similarly, grasp metrics are reorganized and split into priori and posteriori
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Hand-annotated method
Hand-annotated
Physical trial
Metric-based simulation
Accuracy
Physical trial method
Metric-based simulation method
Low Cost
Scale
Figure 1: Qualitative comparison between three kinds of data annotation methodologies. The yellow arrow signifies that our newly proposed method aims at improving the accuracy of the existing annotation methods among metric-based methodologies.
metrics in this paper. Priori metrics defined as metrics measured before execution based physical analysis object geometry, and quasi-static properties, such as the e-metric [9]. Although efficient in computing and strongly supported by physical theories, priori metrics only reflect partial information and ignore dynamic effects involved in real grasping. Posteriori metrics are defined as metrics concluded from grasp trial performances in simulation, implicitly offering far ampler infor- mation about the dynamic performance of grasping than priori metrics. Comparison between priori and posteriori metries in [10] shows the latter closer to human judgment since its consideration of reality. However, drawbacks of posteriori metric are as followed: i) High computation cost; ii) Risk of introducing noise and spurious correlations (a specific clarification is in Appendix A. Previous planar grasping annotating method in simulation adopt either pure prior analytical metrics [11, 5] or pure posterior performance-basee
metrics [12, 6] annotate datasets. Since both individual pri- ori and posteriori metric have their drawbacks, the sticking point is how to integrate them to get trade-off between ampleness and comprehensiveness of information, spurious correlations, and calculation cost when using the metrics- method.
In this paper, build up hybrid grasp metric group and propose metric-based grasp evaluator to approximate human grasp decision capability. Centered on the evaluator, we have constructed automatic grasp annotation framework, through which large-scale, high-quality, low annotation cost planar grasp dataset GMD is generated. Through such process, the human grasping experience can be firstly recorded and mapped into grasp metric space and then re-conveyed into the GMD dataset, which ensures that data annotations mimic human decisions. Our GMD dataset consists of 28k objects, 140k depth images, and 1. 1.milion grasp annotations. Due to the extendability of our automatic annotation framework, the scale can be conveniently extended with new objects source. Two levels of experiments demonstrate that benchmark models trained on GMD have higher success rate in both simulation and reality than the contrastive datasets while generating well different domains.
2 Related Work
Pure priori metrics-based datasets generation. [13] use the form closure metric annotate the dataset through Graspit! [14] simulator. Dex-Net2.0 dataset [5] is synthesized according to robust e-metric, which calculate the expectation of traditionale-metric [9] under preset sensor and control noises. Grasp-Net [15] adopts the improved force-closure metric to annotate 1 billion grasp sam- ples on point clouds collected from real-world scenes, which take the minimum friction coefficient needed for keeping grasp in force closure as the quality criterion. Single metric only reflects par-

---

tial aspects of grasp, researchers begin to find a function combining the multi kinds of single priori metric. Qin al. [16] define the antipodal score (similar to force closure metric), occupancy score (the volume of an object within the gripper closing region), collision score (probability of collision with other objects) and try to integrate them by the minimum operation as the final metric. Zhang [17] imposes antipodal score and center score as the metrics to generate the Regard dataset. The former refers to [16], and the latter describes the distance from the grasp to the geometric center of the object.
Pure posteriori metrics-based datasets generation. [18] examine that posteriori metrics-based labels were easier for DNN to learn from since single priori metric is noisier. Thus, posteriori metrics become popular for synthetic dataset generation. Johns et al. [12] used DART physical simulator to generate a grasping dataset, but it's not released publicly. Similarly, Depierre et al. [6] take grasp attempts result in Pybullet simulator as the posteriori metric to generate the Jacquard dataset with 50k pictures and about 1.1M grasp annotations. Eppner et al. [19] mainly take metrics of grasp trial result and the robustness to external shaking disturbance into consideration automatically generate a large-scale dataset for 6-dof grasp detection.
Combination. The previous grasp annotation method either adopts pure prior analytical metrics or posterior metrics, how to get an optimal combination of multi metrics is still an open problem. Rubert et al. have several precious attempts [20] [10] on merging multi priori metrics find a global index and analyze the correlation between metrics to eliminate similar ones. Their experiments show that proper combination can achieve far higher accuracy than individual priori metries. However, they haven't 't tried to involve posterior metrics in their metrics combination. This inspires us to integrate both priori and posteriori metrics for grasp dataset generation.
3 Metries and Evaluator
This section first proposes a hybrid metric group containing both priori and posteriori grasp metrics to extract features of grasp candidates in Section 3.1. And then two-step evaluator proposed to map the metrics space to the grasp feasibility in Section 3.2 (see Figure 2 for a schematic). The evaluator is trained on a small-scale expert set and then used to generate a large-scale dataset automatically in Section 4.
3.1 Hybrid Metric Group
Priori metrics are the metrics measured before execution based on analysis of object geometry and mechanics of priori domain knowledge. Posteriori metrics are the metrics measured during or after grasp execution, relying on posteriori knowledge concluded from trials. Priori metrics explicitly offer object geometry, and quasi-static properties in an analytical way, while posteriori metrics im- plicitly offer dynamical information in an empirical way. To combine their advantages, we collect 36 metrics for grasp evaluation, including 16 priori metrics, 17 posteriori metrics, and 3 contrastive metrics evaluating the difference between priori and posteriori metrics. They cover as wide aspects as possible to approximate human grasp consideration, involving geometric relations, contact force limitations, grasp stability, robustness to disturbance, grasp comfort, etc [4]. Metrics definition, selection, and correlation analysis are shown in Appendix A.2-A.4. These metrics will be used as features for the machine learning models of the subsequent two-stage evaluator.
3.2 Human-Mimicking Automatic Grasp Evaluator
First-stage: Coarse Screening Model. Due to the huge imbalance between positive and negative samples in grasping, positive ones are far more sparse. It's expensive and unnecessary to calculate all 36 metrics for those meaningless negative samples. Therefore, a decision tree is used as a coarse screening filter to accelerate the process of metrics calculation. Since decision results can be given without obtaining all features in advance, lots of meaningless negative samples can be quickly elimi- nated through only few metrics, obviously improving calculation efficiency. To accomplish this task better, we further improve the original decision tree in two aspects: i) introduce metric sampling cost punishment into typical information gain ratio during model training to sort metrics through a trade-off between classification performance and sampling cost (details the equation (3) in the Appendix B.2); ii) conduct the maximum recall pruning during model testing to avoid false-negative
3

---

Input CAD model and sampled grasp cardidates
Coerse screening slage
Fine screaning stage
calculate matrics
Next node Decision Tree
XOBoost
positive label
To acceletaio the compulation
NO
negative label
NO
Figure 2: The proposed grasp evaluator consists coarse and fine screening models. For each grasp candidate, we sample each grasp metric according to the nodes order of the decision tree, until arriving at leaf nodes. The positive grasp determined by the coarse screening model will be sent to the fine screening model to make more precise judgment the final label.
cases which are irremediable in subsequent steps. Details about model improvement and training are shown Appendix B.2.
Second-stage: Fine Screening Model. After the coarse screening process, those ambiguous sam- ples are left, which need more accurate model do the fine screening. Thus, we adopt joint model composed of XGBoost [21 land SVM [22]. The inputs of the models are the feature vec- tors 36 grasp metrics while the outputs are the probabilities that the grasp is feasible with preset thresholds to define the final labels. To utilize the complete 36 metrics as features for training, we conduct supplementary sampling to fill up the metries ignored by the decision tree of those coarsely screened positive samples from the first stage. In the inference process, the prediction result depends on the voting between the two models. Those Grasps accepted by both models are labeled as the final positive samples. Details about model training are shown in Appendix B.2. According to per- formance on the training set, we set relatively high thresholds for the models to ensure the precision and conservativeness of the grasp decision.
Expert Set Collection. To train the proposed two-stage grasp evaluator, we collect a small- hand-anne annotated expert set as the ground truth. Learning from an expert set ensures the high quality of the dataset generated by the evaluator. The final expert set consists of 256 depth images and 2400 grasp annotations. Details are shown in Appendix B.I.
4 GMD Dataset
This section introduces the pipeline our automatic grasp annotation framework centered on the proposed grasp evaluator and metric group (see Figure 3 for schematic).
4.1
Definition
Grasp representation. GMD dataset offers planar grasp for parallel-jaw grasp from the top-down view, thus all of the grasps in this paper are parallel to the desktop. We define the grasp as:
g3d (p, d w) E TR3 S x RR
where p = (x. 2) E R the position of the grasping center point in Cartesian space, Φ E S the angle between the grasping axis and X-axis of the world coordinate system, w is the width when the parallel-jaw is closed on objects. Then grasp annotation conducted in image space can be acquired by the following transformation:
Janno Tie (Tew (gaa)) (p, Φ. w.
where Tew transforms from the world frame to the camera frame, TIe transforms from the camera frame to 2D image coordinates, and is the corresponding binary label of grasp feasibility.

---

Data Preparation
Grasp Quality Evaluator
GMD Dataset
Object CAD models
annotated expert set
Feasible grasps
Two-stage grasp evaluator
Stage
Stage
Proppee Preprocessing and candidate grasp sampling
Coarse screening model
Fine screening
Infeasible grasps
model
Metrics group
Figure 3: The key components our automatic data generation framework are shown. The source CAD models are firstly loaded in the simulation environment and densely sampled for grasp candi- dates. Then, the two-stage evaluator based on the hybrid metric group automatically annotates the grasp candidates to generate the GMD dataset.
Candidate Grasp Sampling Space. As for each candidate grasp Ssamp in the sampling space Gsamps Zsamp is set as a constant far smaller than the height of the object, which means grasping at the near bottom of the objects. Therefore, grasp axis and the object convex hull onto the zy plane, namely Line n Hully * 0. Since the convex hull of the objects is inside the bounding box, the range of G' can be amplified to Line n be obtained. For all geamp in the Gaamp: (Ig. Ng) E B, 2g D, 6g E [- /2,x/22), where the grasping center sampling space B is obtained by extending the length and width of Boxx by the maximum opening width of gripper.
only need to consider the projection of both the
B F V. In this way, the candidate grasp sampling space G Goomp
Data Preparation
Objects Preparation and Simulation Scene Generation. We use 20k object CAD models as object sources to generate the GMD dataset. For each object, we conduct normalization processing. Details are shown in Appendix C.I. The Pybullet physics simulator [23] is used to build up a grasping platform for the subsequent grasping simulation and datasets synthesis.
Candidate Grasp Dense Sampling. To cover the candidate grasping set G as much as possible, dense sampling method used. In the grasping sampling space Isamp defined in Section 4.1, the center point p is uniformly sampled with step length of 5mm, the grasping depth D is given to be lm, and the opening width of the gripper determined by subsequent grasping attempts. Meanwhile, grasp angles are split into 20 portions and uniformly sampled for each grasp center point, and a candidate grasp set G is obtained. On average, 7000 candidate grasps will be obtained each object.
temporary set to be 0. 1m, which will finally be
4.3 Data Annotation and Analysis
Grasp candidates are annotated by the two-stage evaluator with binary feasibility labels. The final positive samples are offered totally by the fine screening model, while the final negative samples are offered half-and-half from two stages after the scale alignment with positive samples. We used the trimesh [24] and pyrender [25] tools to render 5 depth image (600 600) for each grasp scene from multiple random perspectives with inclination angle limited 10". Then, we use the camera matrix to project all grasps from sampling space gad into the image space (defined in Section 4.1). In total, the GMD dataset contains 28k objects, 140k depth images, and 1. 2million grasp annotations. Table shows that we have the largest-scale source objects among typical datasets for planar grasp,

---

which offer great diversity in object geometry. Although the total grasp scale is not the largest because of the annotation sparsity, we don't regard it as a flaw. Instead, annotation sparsity mimics the rigorousness and conservativeness of human grasp behavior on unknown objects, which may improve model performance across different domains (analyzed in Section 5.3). A small part of GMD dataset visualization can be seen in Figure 3, more of them are shown in Appendix F with Figure 8. The analysis annotation cost can be seen in Appendix C.2.
Table 1: Summary of the properties of the famous public planar grasp datasets.
Dataset
Total objects Total images Total grasps Modality
Annotation methodology
Cornell
240
1035
8019
RGB-D
Hand-annotated
Dex-Net2.0
1500
6.7M
6.7M
Depth
Metric-based (priori)
Jacquard
11K
54K
1.1M
RGB-D
Metric-based (posteriori)
GMD (ours)
28K
140K
1.12M
Depth
Metric-based (priori@posteriori)
5
Experimental Results
In this section, we conduct experiments on two levels. The experiments of the first level are aimed at testing the evaluator proposed in Section 3.2 show that evaluator considering both priori and posterior metrics can approximate human grasp decisions better than pure prior or posterior evaluator. The experiments of the second level are aimed evaluating the GMD dataset generated by the proposed automatic annotation framework. We choose Cornell, Dex-Ner2.0, and Jacquard as contrastive datasets and benchmark them on two representative grasp detection models. The benchmark model choices are: for grasping scoring method, GQCNN [5] is chosen; for one-stage grasp detection method, GR-ConyNet [26] is chosen. The model training details described in Appendix D. Three kinds of experiments are conducted to examine the performance of the trained models: i) pross-dataset evaluation; ii) simulation experiment evaluation; iii) physical experiment evaluation (shown in Figure 4).
A Jaccard Index
B
Simulation experiment
Real experiment
SGT
g
AØ
pred
GMD Simulation
Figure 4: Three kinds of experimental setups in the second level experiment. A. The Jaccard index used for Cross-dataset Evaluation; SGT [6] and our GMDSIM simulative grasp criterion for Sim- ulation Experiments; C. The real test object set and experiment platform of Real Robot Experiments.
5.1 First Level Experiment: Evaluator Comparison
Although many previous works take grasp metrics as ground truth annotate datasets, few of them have compared the grasp decision generated by metrics with ground truth labeled by humans. Thus, we choose the same posteriori metric (success rate) with Jacquard [6], and the same priori metric (e-quality) with Dex-Net2.0 [5] to build two evaluators based split the expert set from Section 3.2 into training set and testing set. For every single metric- based grasp evaluator, first calculate the best threshold according ROC the training set and
single metric respectively. We

---

then make them classifier. Contrastive experiments on the testing set are conducted to compare the performance of our coarse screening model, fine screening model, and two single metric evaluators. Results shown in Table 2 prove that our coarse and fine models based on hybrid metrics can better approximate human decisions than pure priori and posteriori metrics used to generate Dex-Net2.0 and Jacquard datasets.
Table 2: Comparison between different metric-based evaluators.
Evaluator
Coarse model
Fine model
c-quality sim-success
Accuracy (%)
94.1
99.5
69.3
67.6
5.2 Second Level Experiment
Cross-dataset Evaluation. To evaluate whether the benchmark grasp detection model trained on GMD can generate well on other datasets, we use the typical rectangle metric [1] to do the cross- dataset evaluation. The Jaccard Index considers a grasp correct if: i) its rotation error with ground-truth is less than 30 and ii) the rectangle IOU with ground-truth is larger than 0.25 (Figure. 4. A). For the testing set, 100 objects not involved in the training are chosen from each dataset. The average results are shown in Table 3, which demonstrate that: i) models trained on GMD can generalize well to other datasets; ii) models trained on other datasets can only perform well on themselves and suffer from a sharp performance drop on other datasets.
Table 3: Cross-dataset evaluation based on Jaccard Index.
GQCNN
GR-ConvNet
test train
Cornell
Jacquard
Dex-Net2.0
GMD (ours)
test train
Cornell
Jacquard
GMD (ours)
Cornell
63.0
45.6
62.3
72.6
Cornell
66.3
$7.2
63.0
Jacquard
81.8
74.8
87.5
86.4
Jacquard
65,5
69.6
61.3
GMD (ours)
51.5
35.4
51.9
78.4
GMD (ours)
63.1
56.0
74.2
Simulation Experiments. For the simulation experiment, the SGT criterion is used, which is adopted in Jacquard [6]. We build up similar testing platform called GMDSIM in the Pybullet simulator (shown in Figure. 4. B), utilizing a different scene set from the data generation scene in Section 5. As for the testing set, 170 unseen object models are selected from the Shapenet dataset. Each object is grasped for five attempts in simulation to calculate the success rate. To examine the generalization to different simulation domains, both SGT with 100 testing objects and GMDSIM test are conducted. The average success rates are shown in Figure 5. A, which demonstrates that: i) models trained on GMD achieve high performance in GMDSIM test and relatively high performance in SGT test, which means that it can adapt well to other domains. ii)models trained on Jacquard can only grasp well in SGT and suffers from sharp performance drop in GMDSIM, which imply that datasets generated by pure posteriori metric may cause ever-fitting. Meanwhile, models trained on Dex-Net2.0 have higher success rates than Jacquard but lower than GMD, which imply that pure priori metric can help overcome over-fitting but the combination of priori and posteriori is the best choice.
Real Robot Experiments. For the real robot experiment, the experiment platform is built up as shown in Figure 4.C. The testing set is composed of regular, adversarial and household objects, totally 25 objects (more details about the setup are in Appendix F. Each object is grasped with attempts. For each attempt, object position and pose will be reset with random in-plane rotation and shift. An attempt will be regarded as successful grasp when the object can be lifted and transferred specific area without dropping. The real-world experiment results in Fig 5. B shows that: models trained on GMD achieve higher performance on all three kinds of objects; ii) models trained on Jacquard have low performance on three kinds of real objects, which may be caused by the reality gap introduced by the use of pure posteriori metric. iii) models trained on Cornell only have good performance on regular objects, but are not good adversarial objects with complex geometry, which may owe to the lack of diversity limited by scale; iv) models trained on Dex-Net2.0 are not good at household and adversarial objects, which may also suffer from reality gap caused by use of pure priori metric but better than the pure posteriori.
7

---

A. Simulation
B. Real world
GQCNN
GR-ConvNet
GQCNN
GR-ConvNet
Croodl
Cerooa
Figure 5: Simulation and real robot experiments results between Cornell, Jacquard, Dex-Net2.0, and GMD benching on GQCNN and GR-ConvNet.
Disscusion. To summarize, models trained on GMD are proven with high success rates in grasping unknown objects and robustness to noises, which is credited to the diversity of objects and com- prehensive consideration of both priori and posterioni metrics. also reveals that synthetic datasets generated in simulation can be used to train models for real-world grasp detection. The ability to overcome the reality gap may originate in mimicking human decision features of grasp metrics. There is an obvious phenomenon that models trained on GMD always tend to choose the most con- servative grasp like human, which is always feasible and robust in both simulation and reality. For instance (shown in Figure 6), models trained on GMD tend to grasp an object in the middle part which is the most conservative behavior like human, and the consistency can be kept regardless of rotation, shift, and noisy disturbance. In contrast, models trained Jacquard and DexNet2.0 probably grasp at two ends of objects. Although feasible, they are more risky choices and weak at disturbance rejection. This may be helpful to generalize different domains, which is also the advantage of GMD over Jacquard and Dex-Net2.0. On the other hand, the diversity supported by the scale is the advantage of GMD over Cornell.
Cornell
Jacquard
Figure 6: Decision results visualization comparison between GQCNN trained on different datasets.
Conclusion
In this paper, build up hybrid grasp metric group involving both priori and posteriori metrics and propose grasp evaluator to merge those metrics to approximate human grasp decision capability. Centered on the evaluator, we have constructed an automatic grasp annotation framework, through which a large-scale, high-quality, low annotation cost planar grasp dataset GMD is generated. Two levels of experiments show that GMD annotated based on hybrid metries is better than the dataset annotated by pure priori (Dex-Net2.0) and pure posteriori metrics (Jacquard), which is credited to the complementation between different kinds of metrics and the human mimicking properties.
Although GMD acts as competitive dataset for planar grasp detection, some limitations remain: i) with a noisy background; ii) it is only suited for top-down grasp rather than 6-dof grasp. As a general metric-based methodology for automatic grasp annotation, our hybrid metric group, grasp evaluator, and even the total annotation framework introduced in Figure 3 can naturally be migrated to generate clustering scene and 6-Dof grasping dataset, which will be conducted in future research.
only considers single object scenes with simple background rather than clustering

---

Acknowledgments
This work is supported and funded by the National Nature Science Foundation of China under Grant 62173293, Zhejiang Provincial Natural Science Foundation China (LD22E050007), and Science and Technology Space Intelligent Control Laboratory (2021-JCIQ-LB-000-13)
References
[1] I. Lenz, H. Lee, and A. Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Research, 34(4-5):705-724, 2015.
[2] L. Pinto and A. Gupta. Supersizing self-supervision: Learning grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages IEEE, 2016.
[3] S. Levine, Pastor, A. Krizhevsky. J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5):421-438 2018.
[4] M. A. Roa and R. Suárez. Grasp quality measures: review and performance. Autonomous robots, 38(1):65--)))) 88, 2015.
[5] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint urXiv: 093.09 2017.
[6] A. Depierre, E. Dellandréa, and L. Chen. Jacquard: A large scale dataset for robotic grasp detection. In 2018 TEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 351 1-3516. IEEE, 2018
[7] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Craspnet-lbillion A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 11444-1 1453, 2020.
[8] Mahler, M. Matl, V. Satish, Danielczuk, B. DeRose, S. McKinley, and K. Goldberg. Learning ambidextrous robot grasping policies. Science Robotics, 4(26))caa44944, 2019.
[9] C. Ferrari and J. F. Canny. Planning optimal grasps. In ICRA, volume pages 2290-2295. 1992.
[10] C. Rubert, D. Kappler, J. Bohg, and A. Morales. Predicting grasp success in the real world- a study of quality metrics and human assessment. Robolics and Autonomous Systems, 121: 103274, 2019.
[11] J. Mahler, F. Pokorny, B. Hou, M. Roderick, M. Laskey, M. Aubry, K. Kohlhoff, T. Kroger, Kuffner, and K. Goldberg. Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards. pages 1957-1964, 05 2016. doi: 10. 201.1////AA.2211....99....
[12] E. Johns, S. Leutenegger, and A. J. Davison. Deep learning a grasp function for grasping under gripper pose uncertainty. In 2016 IFFF/RS International Conference Intelligent Robots and Systems (IROS), pages 4461 -4668 IEEE, 2016.
[13] C. Goldfeder, M. Ciocarlie, H. Dang, and P. K. Allen. The columbia grasp database. In 2009 IEEE international conference on robotics and automation, pages 1710-1716. IEEE, 2009.
[14] A. T. Miller and P. K. Allen. Graspit! versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 1(4):110-122, 2004.
[15] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Graspnet-lbiilion: A large-scale benchmark for general object grasping. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.

---

[16] Y. Qin, R. Chen, H. Zhu, M. Song, J. Xu, and H. Su. S4g: Amodal single-view single-shot se (3) grasp detection cluttered scenes. In Conference on robot learning. pages 53-65, PMLR, 2020.
[17] H. Zhang, D. Yang, H. Wang, B. Zhao, X. Lan, J. Ding, and N. Zheng. Regrad: A large-scale relational grasp dataset for safe and object-specic robotic grasping in clutter. IEEE Robotics and Automation Letters, 2022.
[18] D. Kappler, J. Bohg, and S. Schaal. Leveraging big data for grasp planning. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 4304 4311. IEEE, 2015.
[19) C. Eppner, A. Mousavian, and D. Fox. Acronym: A large-scale grasp dataset based on sim- ulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1222-6227 IEEE, 2021.
[20] C. Rubert, B. León, A. Morales, and J. Sancho- Bru. Characterisation of grasp quality metrics. Journal of Intelligent & Robotic Systems, 89(3):319-342, 2018
[21] T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. pages 785-794, 08 2016. doi: 10. 1521144429999997229997337
[22] P.-H. Chen, C.-J. Lin, and B. Schölkopf. A tutorial on v-support vector machines. Applied Stochastic Models in Business and Industry, 212):111-136, 2005.
[23] Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016-2019. URI, http::/tp:/yy org.
[24]
Dawgon-Haggerty et al. trimesh.
[25] Matthew Madl et al. pyrender. URL htttp::/ppprender.reetteeeeeeeee
[26] S. Kumra, S. Joshi, and F. Sahin. Antipodal robotic grasping using generative residual con- volutional neural network. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2020.
[27] W. Wohlkinger, A. Aldoma, R. Rusu, and M. Vineze. 3dnet: Large-scale object class recogni- tion from cad models. Proceedings- IEEE International Conference on Robotics and Automa- tion, pages 5384-5391, 05 2012. 10::10.1 11111//CCARAA.22222215111
[28] D. Morrison, P. Corke, and J. Leitner. Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. 03 2020.
[29] A. Kasper, Z. Xue, and R. Dillmann. The kit object models database: An object model database for object recognition, localization and manipulation in service robotics. The International Journal of Robotics Research, 31, 07 2012. 10. 0177702788349114455557
[30] A. Chang. T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song. H. Su, J. Xiao, L. Yi. and F. Yu. Shapenet: An information-rich 3d model repository. 12 2015.
[31] R. Quinlan. C4.5: Programs for Machine Learning, volume 1. 01 1993. ISBN 1-55860-238-0.
[32] B. Li, Y. Lu, C. Li, A. Godil, T. Schreck, M. Aono, M. Burtscher, Q. Chen, N. Chowdhury, B. Fang, H. Fu, T. Furuya, H. Li, J. Liu, Johan, R. Kosaka, H. Koyanagi, R. Ohbuchi, A. Tatsuma, and C. Zou. A comparison of 3d shape retrieval methods based on a large-scale benchmark supporting multimodal queries. Computer Vision and Image Understanding, 131, 11 2014. doi: 10. c0166f.cviu.211..100
[33] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. Dollar. Benchmarking in manipulation research: The yeb object and model set and benchmarking protocols. 02 2015.
[34] J. Leitner, A. W. Tow, N. Sünderhauf, J. E. Dean, J. W. Durham, M. Cooper, M. Eich, C. Lehn- ert, R. Mangels, C. McCool, et al. The acrv picking benchmark: A robotic shelf picking bench- mark to foster reproducible research. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 4705-4712, IEFF, 2017
10


Reviews:
[{"review_id": "tHhnBlxafxn", "reviewer": "Reviewer_Apob", "summary": "The authors present a dataset containing a grouping of 36 grasp metrics for planar, two-finger grasps. These metrics include 16 priori, 17 posteriori, and 3 contrastive metrics, annotated automatically on 1.12 million grasps. They describe both metrics and an evaluator that can sample and collect grasps on a real platform in an automated fashion. To prepare the data, they first densely sample candidate grasps which results in 7000 grasps on each of the 20k objects in simulation. The paper also presents an extensive survey of different planar grasping metrics and datasets, in addition to a new dataset for planar grasping on a table collected in an automated fashion. Their work analyzes these metrics and contains empirical experiments comparing how existing benchmark models (GQCNN and GR-ConvNet) perform when trained with their dataset (GMD) as opposed to the Cornell, Jacquard, and Dex-Net 2.0 datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Their dataset features a comprehensive set of prior and posterior metrics that can be used to fit models to screen candidate grasps and automatically generate a high-quality grasping dataset.\n- Compiles an extensive evaluation of their dataset to 3 other datasets on two different models\n- Develop a metric-based grasp quality evaluator that can screen candidate grasps and yield high-quality grasps in the planar grasping setting\n- Develop a testing platform, GMDSIM, used to evaluate multiple grasping datasets\n\nWeaknesses\n- Their work considers simpler parallel jaw-gripper planar grasps, which limits the uses of this dataset and set of metrics to other hand/object pairings\n- Some typographical and organizational improvements are needed\n- Only consider top-down grasps, which may have limited depth information for grasps for irregularly shaped objects", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper both presents an extensive survey of different planar grasping metrics and datasets and a new dataset for planar grasping on a table collected in an automated fashion. Their work analyzes these metrics, and contains empirical experiments comparing how existing benchmark models (GQCNN and GR-ConvNet) perform when trained with their dataset (GMD) as opposed to the Cornell, Jacquard, and Dex-Net 2.0 datasets. Aside from minor improvements and polish, this paper is a great reference for grasp metrics, as well as offers several contributions such as a dataset, grasping simulator, and candidate filtering grasping evaluator.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Typos/improvements\n- Figure 3 caption (simulaiton)\n- lines 151-152: (sentence beginning as \"Since annotation sparsity...\" is a fragment)\n- line 125: xoy plane -> xy plane\n- line 154: visualization spelling\n- line 150: sentence unclear, \"...great diversity in geometric\"\n- Figure 4 caption should be more descriptive\n- Standard error bars on figure 5 bar plot would help summarize success rates from wide range of objects \n- Figure 6 differences between plots are imperceptible, and can use further context in the writing. Perhaps cropping images, and drawing circles on the contact regions can help \n\nAdditional related work to compare to:\nYang, Daniel, et al. \"Robotic grasping through combined image-based grasp proposal and 3d reconstruction.\" 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.\ntheir work consider shape reconstruction from images to construct a point cloud used to generate grasp proposals "}}, {"review_id": "f2GnQMZs5-n", "reviewer": "Reviewer_5FZ9", "summary": "The paper introduces an approach to combine both priori and posteriori metrics relevant for the grasping process for automated evaluation and annotation of grasp candidates. Based on the metrics and a small-scale manually annotated dataset, the authors define a grasp quality evaluator and use it to create a large-scale dataset by automatically annotating grasp feasibility based on the evaluator. The approach is considering feasibility of planar grasps from depth images of single objects. In the experiments the authors compare the performance of the proposed evaluator for w.r.t. a pure priori or posteriori metric for automated annotation and find that the proposed evaluator annotation performs significantly better on their expert dataset. They also compare the performance of two grasp detection models when trained with their dataset w.r.t. other datasets and find that training with their dataset leads to better results across datasets, in simulation and on a real robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper introduces an interesting annotation procedure which incorporates human knowledge in the process of automatic annotation of feasible grasps based on both priori and posteriori metrics. \nEven though there are similar datasets available, the proposed dataset extends the number of covered objects and the annotation methodology, which might be useful to the research community\n\nWeaknesses:\nThere are several major issues (listed below) that significantly influence the paper quality, clarity and the interpretation of the results.\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors have addressed important questions in grasping, namely automated data annotation and generation for training data-driven grasping models, and their dataset might be useful to the community. However, in the reviewer's opinion the paper itself needs major revisions in the writing and organization and there are several open issues to be addressed/clarified related to the experiments.\n\n(Edit: The authors addressed the technical questions and issues sufficiently, but the text quality should be further improved, the general recommendation is updated to weak accept)", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Title: In reviewer's opinion the title of the paper is not well aligned with the content. Namely \"On Label Annotation for Grasping: Metrics and Datasets\" suggests a review paper on the topic of grasping annotations and datasets. On the other hand, the main contribution of the paper is its own proposed dataset and the annotation procedure. \n(Edit: The authors defined more precisely the title and it is in line with requests from other reviewers, so this issue is resolved).\n\nRelated work section placement: The paper builds on a lot of related work both in annotation metrics and in similar datasets, however, the authors place the \"Related work\" section at the end. Related work section would be more appropriate to come after the introduction, after which the authors can clarify their contributions better w.r.t. existing work. \n(Edit: The authors restructured the paper and related work is moved earlier, so this issue is resolved).\n\nFirst level experiments (Section 4.1): You split the the expert dataset to training and test portion and train 2 single-stage (priori and posteriori) classifiers. Then, you use the test portion of the expert set to evaluate them and the coarse and fine models of your evaluator. But you also used the expert set to train the coarse and fine models. Was the test portion from this experiment not used already to train the coarse and fine models? If it was, then the comparison is not fair. Please clarify this. \n(Edit: The authors clarified that the test set is separate and unseen during training, so this issue is resolved).\n\nSecond level experiments (Section 4.2): In the original paper of GR-ConvNet the authors report much higher accuracy when evaluating GR-ConvNet on the Corenell and the Jacquard datasets using depth data. Comparing that with the results of your experiment in the case when you use GR-ConvNet to train and test on the same datasets you get much worse results (>20%) which makes the reviewer wonder whether the training hyperparameters are properly set for the experiments. Can you explain why the results are much worse when you train and test GR-ConvNet on Corenell and Jacquard respectively in your experiment? \n(Edit: The authors clarified that due to their experiment setup they used different subset of the test set w.r.t. the original paper, which lead to the lower results, as well as that the original GR-ConvNet models achieved similarly lower results on the test subset, so this issue is resolved).\n\nIn general, the language in the paper is not at publication quality - there are many typos (easy to fix) and poorly connected sentences. The paper needs detailed proofreading and significant improvement of the flow within and between sentences at many places. Some examples:  \n- Typos and errors easy to correct: \"trial\" instead of \"trail\", \"geometry\" instead of \"geometric\", ... singular/plural mistakes, missing \"a\" and \"the\" articles, third person mistakes, use of \"famous\", \"huge\", \"precious\" is not most adequate\n- Some example sentences: \nTo equip robots with the ability of detecting feasible grasps in a scene, grasp detection has become a hot area. (Edit: There is grammar mistake in the revised sentence)\nSingle metric only reflects partial aspects of grasp, researchers begin to find a function combining multi priori metrics. \n(Edit: The authors tried to address this issue, but there are still many typos - which tools like the free version of Grammarly can easily spot. The text quality should be further improved by the authors, a proofreading service and/or software tools. This issue is still open). \n\n"}}, {"review_id": "WwqURnbHhSV", "reviewer": "Reviewer_MbMM", "summary": "This paper proposes a method to evaluate grasps based on both the physical properties of the target object and grasping experience. The grasp evaluation procedure is then used to generate a dataset to train a parallel-jaw, top-down grasp planner. The method and the dataset are evaluated in simulation and on a physical system.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea proposed in this paper, to combine a physical prior and an experimental posterior, is interesting and works well when objects are isolated.\n\n2. The paper includes a vast amount of experiments in simulation and on a physical platform with multiple datasets and grasp planners.\n\nWeaknesses:\n1. The evaluator is trained on carefully selected grasps, and as a result may be biased. The authors also mention in the discussion that GMD tends to be conservative and chooses grasps around the center of the object (an argument that may support this hypothesis). This might work with isolated objects, however it is not clear that it will work in common real-world situations in which not all grasps are available due to occlusions or collisions.\n\n2. The grasp representation includes w, the width when the PJ gripper is closed. Since this paper only deals with grasping isolated objects, I’m assuming it’s not needed to avoid collisions with nearby obstacles. Is it needed for grasping objects with a specific geometry? If so, are these objects in the objects set presented in Fig 4? I’d appreciate a discussion/explanation as to why this parameter is needed to represent a grasp.\n\n3. A significant part of the paper is described in the appendix. I’d recommend to cut down some of the text and maybe shrinking down the figures to make space for more information on the features, the evaluator and the experimental setup \n\n4. There are spelling mistakes and grammar errors throughout the paper. For example, the combination “physical trail” (used several times in the paper) is not clear, maybe should be changed to “physical trial” like in Fig. 1?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It’s an interesting idea but given that the paper only demonstrated that it works for isolated objects I am recommending a weak accept.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See weaknesses above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "5njTDqJx725", "reviewer": "Reviewer_Vr9d", "summary": "The paper introduces a novel method for generating large-scale synthetic training and evaluation datasets for planar grasping. The key idea is to combine multiple metrics, a-priori (derived from first principles) and a-posteriori (generated from grasp trials in simulation) metrics and join their output scores through two learned models -- a coarse-grained decision tree and a fine-grained XGBoost/SVM model. The paper uses the process to generate a large-scale grasping dataset termed \"GMD\", and demonstrates superior performance when training GQCNN [5] and GR-ConvNet [18] on GMD, and confirms these results both through simulated as real-world experiments.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is technically sound and makes an interesting contribution to planar grasping. In general, the paper is easy to understand. The experiments are well-explained, -conceived and -analyzed, and support the general claim of the paper, that a dataset with a combination of metrics results in more accurate grasp detection models. I also liked very much the discussion of different types of annotation schemes and metrics.\n\nThe main weaknesses I see are whether the contributions remain significant when taking into account more recent state-of-the-art methods. Furthermore, the manuscript should be improved wrt. clarity, orthography and typesetting.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper is technically sound and makes an interesting contribution to planar grasping. I would like to confirm that the shown results remain relevant for 6 DoF grasping, and also on more recent grasping methods.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "#### Title\n\nI would suggest to include the term \"planar\" explicitly in the title to clarify the scope of the paper, i.e. \"On Label Annotation for *Planar* Grasping: Metrics and Datasets\".\n\n#### SoTA models\n\nThe chosen models are not very recent and not necessarily SoTA for planar grasping (see e.g. arXiv: 2207.02556, Deep Learning Approaches to Grasp Synthesis: A Review, Section III.A, page 6; https://arxiv.org/pdf/2105.00329.pdf). Would the proposed method also show substantial improvements in those cases? I would like to see this discussed -- or even validated experimentally, if necessary -- in the revision of the paper.\n\nSimilarly, while the results may be convincing for the planar grasping domain, I'm wondering whether they generalize to more complex grasping scenarios, as claimed in the conclusion section. For example, planar grasping has the key advantage that the grasp candidate space can be sampled almost exhaustively, which is not the case for high-DOF scenarios. Papers such as that grasp sampling starts playing a more and more important role. It would be great to see this discussed in the paper.\n\n#### Clarity\n\nAt several points, I would have liked to see some more details covered in the main paper and not only the appendix. For example:\n- What is the high-level idea behind training the coarse-grained decision tree? At least 1-2 sentences should explain how it is build.\n- The main categories of metrics used should be briefly described in the paper.\n\nFurthermore, some concepts and terms are not explained precisely, or not properly introduced, examples:\n- The first sentence of the introduction (Grasping ...) confuses me, in particular \"always the guarantee for the following complex tasks\". I think this is wrong and confuses that a good grasp is a sufficient, not a necessary condition. So I believe \"guarantee\" should be replaced by \"prerequisite\"\n- Line 92: \"(i) Introduce\" I found this sentence very difficult to understand - only later I understood that you are referring to decision tree training here. Explain that you are using a decision tree, why and -- on the high-level -- how it is trained (e.g. IG is standard for decision tree, might not even be necessary mentioning it)\n- Line 103. Sentence \"Grasps determined ...\" is ungrammatical. Are you saying that a grasp is rejected if rejected by both models?\n- I found the sentence in line 123-124 difficult to parse. What is g_samp in G_samp?\n- What is \"SGT\"? (line 189)\n- Line 105: set relatively \"high thresholds\" - be precise, wrt to what?\n\nAdditionally, the paper has some recurring typesetting issues (no space in front of left parenthesis \"(\") and quite a few orthographic and grammatical errors.\n\n\n#### Misc comments/questions\n\n- I like Figure 1 but I would suggest to explicitly say that this is a sketch and not based on data.\n- Given that you are saying that you set high thresholds to ensure conservative grasps; how do you deal with a \"zero-recall\" problem, i.e. that for an item no grasp can be found? Did such a problem occur?\n"}}, {"review_id": "-ZzxYjXEdz", "reviewer": "Reviewer_Bsku", "summary": "The paper proposes a learnt grasp evaluation model that can be used for automatic top-down grasp annotation. The grasp evaluator is trained using features such as physics-based metrics as well as 'posteriori' metrics i.e. metrics from actual grasp trials. The advantages of the dataset created (GMD) using the automatic top-down grasp label generation have been highlighted in the experiments. Specifically, it is shown that a grasping network trained on GMD can outperform networks trained on other top-down grasp datasets such as Dex-Net 2.0, Jacquard and Cornell.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of using automatic grasp label generation is sound and can greatly help in terms of scalability with a relatively low cost.\n- The combination of both physics-based metrics as well as actual grasp trials is interesting and potentially quite useful.\n- The grasp features used (both priori and posteriori) are extensive and their co-relations have been sufficiently analyzed (Appendix).\n- Beyond a few typos, the paper is reasonably well-written and the visualizations are useful.\n\nWeaknesses:\n- A major thing that is unclear from the manuscript is the usage of posteriori metrics during the grasp annotation phase. Since the grasp evaluator (fine screening) model needs all 35 features as input, how are these inputs provided for automatic annotation of a novel grasp? Priori metrics can of course be calculated but how can posteriori metrics be computed in this case? Would this require a simulation trial for each grasp? Would this not be expensive?\n- The GMD dataset consists of many more objects as compared to the other datasets. It is thus possible that the performance improvements that GMD helps realize are simply due to this increase in the scale of the dataset.\n- The authors need to test and clarify if the grasp-evaluator on its own gives a better grasp prediction as compared to metrics such as force-closure. To do this, two datasets of similar size and same objects could be compared, with one dataset annotated using force-closure and one dataset annotated in a similar fashion as GMD.\n- It should be highlighted how the learnt grasp evaluator differs from prior work such as [*] and [10].\n- It is not clear how computationally expensive it is to use the grasp annotation as compared to computing priori or posteriori metrics only.\n- While top-down grasping is still important, it would be interesting to see how the grasp evaluator could be extended to work with 6D grasps.\n\nMinor:\n- It is unclear what the yellow arrow in Figure 1. signifies.\n- Typos: 'trail' instead of trial, 'generating' instead of generalizing. Unclear line 102-103: \"XGBoost and SVM are parallel to predict the test data respectively\". Unclear line 92: \"introduce metric sampling cost punishment into typical information gain ratio\".\n\n[*] C. Rubert, D. Kappler, A. Morales, S. Schaal and J. Bohg, \"On the relevance of grasp metrics for predicting grasp success,\" IROS 2017", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As mentioned, the automatic grasp annotation scheme is potentially very useful and the combination of priori and posteriori metrics is interesting. The results are promising in terms of outperforming Dex-Net and other datasets. The analysis of different grasping metrics could also be useful for the community.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Clarifications are needed regarding:\n- The use of posteriori metrics in the grasp evaluator as test time (annotation time).\n- The effectiveness of the learnt grasp evaluator as compared to only priori metrics in a dataset of the same size. Also, the computational efficiency of the grasp annotator needs to be compared to the case where only priori or posteriori metrics are used.\n- Author's claims related to \"human-like\" grasp decisions need to be clarified. How can it be shown that the proposed grasp evaluator labels grasps in a 'human-like' fashion?"}}]

Author rebuttal:

Decision: Accept (poster)