Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a learnt grasp evaluation model that can be used for automatic top-down grasp annotation. The grasp evaluator is trained using features such as physics-based metrics as well as 'posteriori' metrics i.e. metrics from actual grasp trials. The advantages of the dataset created (GMD) using the automatic top-down grasp label generation have been highlighted in the experiments. Specifically, it is shown that a grasping network trained on GMD can outperform networks trained on other top-down grasp datasets such as Dex-Net 2.0, Jacquard and Cornell.
strengths_and_weaknesses: Strengths:
- The idea of using automatic grasp label generation is sound and can greatly help in terms of scalability with a relatively low cost.
- The combination of both physics-based metrics as well as actual grasp trials is interesting and potentially quite useful.
- The grasp features used (both priori and posteriori) are extensive and their co-relations have been sufficiently analyzed (Appendix).
- Beyond a few typos, the paper is reasonably well-written and the visualizations are useful.

Weaknesses:
- A major thing that is unclear from the manuscript is the usage of posteriori metrics during the grasp annotation phase. Since the grasp evaluator (fine screening) model needs all 35 features as input, how are these inputs provided for automatic annotation of a novel grasp? Priori metrics can of course be calculated but how can posteriori metrics be computed in this case? Would this require a simulation trial for each grasp? Would this not be expensive?
- The GMD dataset consists of many more objects as compared to the other datasets. It is thus possible that the performance improvements that GMD helps realize are simply due to this increase in the scale of the dataset.
- The authors need to test and clarify if the grasp-evaluator on its own gives a better grasp prediction as compared to metrics such as force-closure. To do this, two datasets of similar size and same objects could be compared, with one dataset annotated using force-closure and one dataset annotated in a similar fashion as GMD.
- It should be highlighted how the learnt grasp evaluator differs from prior work such as [*] and [10].
- It is not clear how computationally expensive it is to use the grasp annotation as compared to computing priori or posteriori metrics only.
- While top-down grasping is still important, it would be interesting to see how the grasp evaluator could be extended to work with 6D grasps.

Minor:
- It is unclear what the yellow arrow in Figure 1. signifies.
- Typos: 'trail' instead of trial, 'generating' instead of generalizing. Unclear line 102-103: "XGBoost and SVM are parallel to predict the test data respectively". Unclear line 92: "introduce metric sampling cost punishment into typical information gain ratio".

[*] C. Rubert, D. Kappler, A. Morales, S. Schaal and J. Bohg, "On the relevance of grasp metrics for predicting grasp success," IROS 2017
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: As mentioned, the automatic grasp annotation scheme is potentially very useful and the combination of priori and posteriori metrics is interesting. The results are promising in terms of outperforming Dex-Net and other datasets. The analysis of different grasping metrics could also be useful for the community.
issues: Clarifications are needed regarding:
- The use of posteriori metrics in the grasp evaluator as test time (annotation time).
- The effectiveness of the learnt grasp evaluator as compared to only priori metrics in a dataset of the same size. Also, the computational efficiency of the grasp annotator needs to be compared to the case where only priori or posteriori metrics are used.
- Author's claims related to "human-like" grasp decisions need to be clarified. How can it be shown that the proposed grasp evaluator labels grasps in a 'human-like' fashion?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors present a dataset containing a grouping of 36 grasp metrics for planar, two-finger grasps. These metrics include 16 priori, 17 posteriori, and 3 contrastive metrics, annotated automatically on 1.12 million grasps. They describe both metrics and an evaluator that can sample and collect grasps on a real platform in an automated fashion. To prepare the data, they first densely sample candidate grasps which results in 7000 grasps on each of the 20k objects in simulation. The paper also presents an extensive survey of different planar grasping metrics and datasets, in addition to a new dataset for planar grasping on a table collected in an automated fashion. Their work analyzes these metrics and contains empirical experiments comparing how existing benchmark models (GQCNN and GR-ConvNet) perform when trained with their dataset (GMD) as opposed to the Cornell, Jacquard, and Dex-Net 2.0 datasets.
strengths_and_weaknesses: Strengths
- Their dataset features a comprehensive set of prior and posterior metrics that can be used to fit models to screen candidate grasps and automatically generate a high-quality grasping dataset.
- Compiles an extensive evaluation of their dataset to 3 other datasets on two different models
- Develop a metric-based grasp quality evaluator that can screen candidate grasps and yield high-quality grasps in the planar grasping setting
- Develop a testing platform, GMDSIM, used to evaluate multiple grasping datasets

Weaknesses
- Their work considers simpler parallel jaw-gripper planar grasps, which limits the uses of this dataset and set of metrics to other hand/object pairings
- Some typographical and organizational improvements are needed
- Only consider top-down grasps, which may have limited depth information for grasps for irregularly shaped objects
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper both presents an extensive survey of different planar grasping metrics and datasets and a new dataset for planar grasping on a table collected in an automated fashion. Their work analyzes these metrics, and contains empirical experiments comparing how existing benchmark models (GQCNN and GR-ConvNet) perform when trained with their dataset (GMD) as opposed to the Cornell, Jacquard, and Dex-Net 2.0 datasets. Aside from minor improvements and polish, this paper is a great reference for grasp metrics, as well as offers several contributions such as a dataset, grasping simulator, and candidate filtering grasping evaluator.
issues: Typos/improvements
- Figure 3 caption (simulaiton)
- lines 151-152: (sentence beginning as "Since annotation sparsity..." is a fragment)
- line 125: xoy plane -> xy plane
- line 154: visualization spelling
- line 150: sentence unclear, "...great diversity in geometric"
- Figure 4 caption should be more descriptive
- Standard error bars on figure 5 bar plot would help summarize success rates from wide range of objects 
- Figure 6 differences between plots are imperceptible, and can use further context in the writing. Perhaps cropping images, and drawing circles on the contact regions can help 

Additional related work to compare to:
Yang, Daniel, et al. "Robotic grasping through combined image-based grasp proposal and 3d reconstruction." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
their work consider shape reconstruction from images to construct a point cloud used to generate grasp proposals
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper introduces an approach to combine both priori and posteriori metrics relevant for the grasping process for automated evaluation and annotation of grasp candidates. Based on the metrics and a small-scale manually annotated dataset, the authors define a grasp quality evaluator and use it to create a large-scale dataset by automatically annotating grasp feasibility based on the evaluator. The approach is considering feasibility of planar grasps from depth images of single objects. In the experiments the authors compare the performance of the proposed evaluator for w.r.t. a pure priori or posteriori metric for automated annotation and find that the proposed evaluator annotation performs significantly better on their expert dataset. They also compare the performance of two grasp detection models when trained with their dataset w.r.t. other datasets and find that training with their dataset leads to better results across datasets, in simulation and on a real robot.
strengths_and_weaknesses: Strengths:
The paper introduces an interesting annotation procedure which incorporates human knowledge in the process of automatic annotation of feasible grasps based on both priori and posteriori metrics. 
Even though there are similar datasets available, the proposed dataset extends the number of covered objects and the annotation methodology, which might be useful to the research community

Weaknesses:
There are several major issues (listed below) that significantly influence the paper quality, clarity and the interpretation of the results.
 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The authors have addressed important questions in grasping, namely automated data annotation and generation for training data-driven grasping models, and their dataset might be useful to the community. However, in the reviewer's opinion the paper itself needs major revisions in the writing and organization and there are several open issues to be addressed/clarified related to the experiments.

(Edit: The authors addressed the technical questions and issues sufficiently, but the text quality should be further improved, the general recommendation is updated to weak accept)
issues: Title: In reviewer's opinion the title of the paper is not well aligned with the content. Namely "On Label Annotation for Grasping: Metrics and Datasets" suggests a review paper on the topic of grasping annotations and datasets. On the other hand, the main contribution of the paper is its own proposed dataset and the annotation procedure. 
(Edit: The authors defined more precisely the title and it is in line with requests from other reviewers, so this issue is resolved).

Related work section placement: The paper builds on a lot of related work both in annotation metrics and in similar datasets, however, the authors place the "Related work" section at the end. Related work section would be more appropriate to come after the introduction, after which the authors can clarify their contributions better w.r.t. existing work. 
(Edit: The authors restructured the paper and related work is moved earlier, so this issue is resolved).

First level experiments (Section 4.1): You split the the expert dataset to training and test portion and train 2 single-stage (priori and posteriori) classifiers. Then, you use the test portion of the expert set to evaluate them and the coarse and fine models of your evaluator. But you also used the expert set to train the coarse and fine models. Was the test portion from this experiment not used already to train the coarse and fine models? If it was, then the comparison is not fair. Please clarify this. 
(Edit: The authors clarified that the test set is separate and unseen during training, so this issue is resolved).

Second level experiments (Section 4.2): In the original paper of GR-ConvNet the authors report much higher accuracy when evaluating GR-ConvNet on the Corenell and the Jacquard datasets using depth data. Comparing that with the results of your experiment in the case when you use GR-ConvNet to train and test on the same datasets you get much worse results (>20%) which makes the reviewer wonder whether the training hyperparameters are properly set for the experiments. Can you explain why the results are much worse when you train and test GR-ConvNet on Corenell and Jacquard respectively in your experiment? 
(Edit: The authors clarified that due to their experiment setup they used different subset of the test set w.r.t. the original paper, which lead to the lower results, as well as that the original GR-ConvNet models achieved similarly lower results on the test subset, so this issue is resolved).

In general, the language in the paper is not at publication quality - there are many typos (easy to fix) and poorly connected sentences. The paper needs detailed proofreading and significant improvement of the flow within and between sentences at many places. Some examples:  
- Typos and errors easy to correct: "trial" instead of "trail", "geometry" instead of "geometric", ... singular/plural mistakes, missing "a" and "the" articles, third person mistakes, use of "famous", "huge", "precious" is not most adequate
- Some example sentences: 
To equip robots with the ability of detecting feasible grasps in a scene, grasp detection has become a hot area. (Edit: There is grammar mistake in the revised sentence)
Single metric only reflects partial aspects of grasp, researchers begin to find a function combining multi priori metrics. 
(Edit: The authors tried to address this issue, but there are still many typos - which tools like the free version of Grammarly can easily spot. The text quality should be further improved by the authors, a proofreading service and/or software tools. This issue is still open).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper introduces a novel method for generating large-scale synthetic training and evaluation datasets for planar grasping. The key idea is to combine multiple metrics, a-priori (derived from first principles) and a-posteriori (generated from grasp trials in simulation) metrics and join their output scores through two learned models -- a coarse-grained decision tree and a fine-grained XGBoost/SVM model. The paper uses the process to generate a large-scale grasping dataset termed "GMD", and demonstrates superior performance when training GQCNN [5] and GR-ConvNet [18] on GMD, and confirms these results both through simulated as real-world experiments.

strengths_and_weaknesses: Overall, the paper is technically sound and makes an interesting contribution to planar grasping. In general, the paper is easy to understand. The experiments are well-explained, -conceived and -analyzed, and support the general claim of the paper, that a dataset with a combination of metrics results in more accurate grasp detection models. I also liked very much the discussion of different types of annotation schemes and metrics.

The main weaknesses I see are whether the contributions remain significant when taking into account more recent state-of-the-art methods. Furthermore, the manuscript should be improved wrt. clarity, orthography and typesetting.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, the paper is technically sound and makes an interesting contribution to planar grasping. I would like to confirm that the shown results remain relevant for 6 DoF grasping, and also on more recent grasping methods.
issues: #### Title

I would suggest to include the term "planar" explicitly in the title to clarify the scope of the paper, i.e. "On Label Annotation for *Planar* Grasping: Metrics and Datasets".

#### SoTA models

The chosen models are not very recent and not necessarily SoTA for planar grasping (see e.g. arXiv: 2207.02556, Deep Learning Approaches to Grasp Synthesis: A Review, Section III.A, page 6; https://arxiv.org/pdf/2105.00329.pdf). Would the proposed method also show substantial improvements in those cases? I would like to see this discussed -- or even validated experimentally, if necessary -- in the revision of the paper.

Similarly, while the results may be convincing for the planar grasping domain, I'm wondering whether they generalize to more complex grasping scenarios, as claimed in the conclusion section. For example, planar grasping has the key advantage that the grasp candidate space can be sampled almost exhaustively, which is not the case for high-DOF scenarios. Papers such as that grasp sampling starts playing a more and more important role. It would be great to see this discussed in the paper.

#### Clarity

At several points, I would have liked to see some more details covered in the main paper and not only the appendix. For example:
- What is the high-level idea behind training the coarse-grained decision tree? At least 1-2 sentences should explain how it is build.
- The main categories of metrics used should be briefly described in the paper.

Furthermore, some concepts and terms are not explained precisely, or not properly introduced, examples:
- The first sentence of the introduction (Grasping ...) confuses me, in particular "always the guarantee for the following complex tasks". I think this is wrong and confuses that a good grasp is a sufficient, not a necessary condition. So I believe "guarantee" should be replaced by "prerequisite"
- Line 92: "(i) Introduce" I found this sentence very difficult to understand - only later I understood that you are referring to decision tree training here. Explain that you are using a decision tree, why and -- on the high-level -- how it is trained (e.g. IG is standard for decision tree, might not even be necessary mentioning it)
- Line 103. Sentence "Grasps determined ..." is ungrammatical. Are you saying that a grasp is rejected if rejected by both models?
- I found the sentence in line 123-124 difficult to parse. What is g_samp in G_samp?
- What is "SGT"? (line 189)
- Line 105: set relatively "high thresholds" - be precise, wrt to what?

Additionally, the paper has some recurring typesetting issues (no space in front of left parenthesis "(") and quite a few orthographic and grammatical errors.


#### Misc comments/questions

- I like Figure 1 but I would suggest to explicitly say that this is a sketch and not based on data.
- Given that you are saying that you set high thresholds to ensure conservative grasps; how do you deal with a "zero-recall" problem, i.e. that for an item no grasp can be found? Did such a problem occur?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method to evaluate grasps based on both the physical properties of the target object and grasping experience. The grasp evaluation procedure is then used to generate a dataset to train a parallel-jaw, top-down grasp planner. The method and the dataset are evaluated in simulation and on a physical system.
strengths_and_weaknesses: Strengths:
1. The idea proposed in this paper, to combine a physical prior and an experimental posterior, is interesting and works well when objects are isolated.

2. The paper includes a vast amount of experiments in simulation and on a physical platform with multiple datasets and grasp planners.

Weaknesses:
1. The evaluator is trained on carefully selected grasps, and as a result may be biased. The authors also mention in the discussion that GMD tends to be conservative and chooses grasps around the center of the object (an argument that may support this hypothesis). This might work with isolated objects, however it is not clear that it will work in common real-world situations in which not all grasps are available due to occlusions or collisions.

2. The grasp representation includes w, the width when the PJ gripper is closed. Since this paper only deals with grasping isolated objects, I’m assuming it’s not needed to avoid collisions with nearby obstacles. Is it needed for grasping objects with a specific geometry? If so, are these objects in the objects set presented in Fig 4? I’d appreciate a discussion/explanation as to why this parameter is needed to represent a grasp.

3. A significant part of the paper is described in the appendix. I’d recommend to cut down some of the text and maybe shrinking down the figures to make space for more information on the features, the evaluator and the experimental setup 

4. There are spelling mistakes and grammar errors throughout the paper. For example, the combination “physical trail” (used several times in the paper) is not clear, maybe should be changed to “physical trial” like in Fig. 1?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: It’s an interesting idea but given that the paper only demonstrated that it works for isolated objects I am recommending a weak accept.
issues: See weaknesses above.
review_title: null

Meta-review: metareview: The paper proposes a set of grasp evaluation metrics, and a learned grasp evaluator to automatically assess grasp quality, leading to also an automatic way of annotating two-fingered grasps. 

The **strengths** of the paper are:
- the combination of physics-based prior and experimental-based posterior metrics for grasp evaluation are effective;
- the automatic grasp annotation can provide datasets with high-quality grasps;
- the extensive evaluation of the provided dataset against other datasets and with two different models;
- a simulation environment for grasp annotation and evaluation;
- real world experiments with different objects, and encouraging results on isolated top-down grasps.

The major **weaknesses** of the paper can be summarized as:
- the consideration of only top-down, parallel jaw grasps, that is a prominent topic in the grasping literature, and discussion regarding 6D grasps is not included.
- the effect of posterior metrics on grasp annotation is not clear;
- statistical correlation of the size of the dataset and the reported performance compared to the other datasets;
- details regarding the experimental design and evaluation are not well-reported;
- the limitations of the proposed method are not well explained.

The strengths of the paper are plenty, however there are pertaining issues with the paper, and the authors should address all raised points raised by the reviewers to convince about their contribution.

**Post-rebuttal assessment:** The authors provided sufficient additional information, they updated their paper, and concretely described the limitations of their method. The AC considers this paper an interesting contribution in the topic of robot grasp learning and quality assessment.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)