Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper studies the problem of grasp synthesis of transparent objects using RGBD images. The authors propose a pipeline based on Neural Radiance Fields (NeRFs), which they modify in several ways to make the training process amenable to real-time robotics use. These include interleaving dataset updates with NeRF training, additional loss terms and hyperparameter adjustments, and early stopping. They further propose to train a grasp planing network (based on prior work) directly on the NeRF depth images, rather than the images generated by the depth sensor. The authors study their method's performance onboard a physical robot, and demonstrate much faster performance than DexNet, fast re-training of the NeRF for decluttering, and ablate the proposed NeRF modifications to clearly demonstrate their contribution.
strengths_and_weaknesses: + The proposed method can quickly train a NeRF representation, and works for transparent objects, a current open problem in grasp synthesis from visual data.

+ The hardware experiments are thorough and demonstrate a clear improvement over (approximately) the state of the art, DexNeRF. The modifications to the NeRF training, in particular, are novel and appear quite useful.

- The method requires a somewhat prescribed "capture trajectory" to build the NeRF model, although this is usually stopped early once the grasp generation is confident.

- The grasp planning network has a Sim2Real issue (trained in sim, deployed on real objects) although the authors claim training on NeRF depth ameliorates this somewhat.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Strong accept. The proposed modifications to the NeRF training make it applicable for real-time use in robotics, and are ablated to demonstrate their benefits in real hardware. Grasp synthesis for transparent/reflective objects is still quite challenging, and the authors present a method that seems to do this quite effectively. These are exciting, compelling, and novel results.
issues: - I'm not sure the Sim2Real gap is discussed in enough detail in the text. Section 5.5 seems to address this, but mostly reports that training grasp networks on RGB-D images from reflective objects (this is the motivation of the paper?). I would argue a better experiment would be to test this on opaque, matte objects -- if the simulation-trained grasp planner performs similarly to DexNet in this case, you have made a strong case that the Sim2Real gap is quite small.

- In general, the text in the experiments section suffers from a lack of clarity (Section 5.5 was particularly hard to parse). I think this part of the paper would benefit from some editing for clarity/grammar. 

- This is more a direction for future work, but it seems there would be better ways to generate the capture trajectory based on error metrics rather than following a path on a sphere. 

- Similarly, I found the "deletion" mechanism a bit less-than-compelling. This isn't an issue that should prevent publication, but it seems a more elegant way to handle the problem would be to segment the NeRF into several smaller objects that are composited during rendering.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This manuscript introduces a method to progressively update and quickly train NeRF models to grasp transparent objects. The authors propose to reuse weights for sequential grasping tasks, use an early stopping strategy to speed up the completion, and add regularization loss to mitigate the floaters issue. To reduce the sim2real gap, the authors propose to train the network to predict grasps directly on NeRF-rendered depths maps. The proposed pipeline is applied in real-world experiments, achieving an 89% success rate for single object grasping and reducing the capture time significantly for both single object grasping and sequential grasping tasks.
strengths_and_weaknesses: Strengths:
1. The paper is well written and easy to follow. 
2. It’s exciting to see that the proposed grasping pipeline only takes seconds to grasp a transparent object with the NeRF model.

Weaknesses:
1  It’s not clear whether the proposed strategy of regularization is generally applicable or not (See Issue 2).
2. More failure case analyses and real-world videos of these failure cases actually strengthen the paper.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The manuscript introduces a pipeline for robots to grasp transparent objects taking only seconds per object, which is useful and important for many real-world robotic applications. But it’s not clear whether the proposed strategies are generally helpful or not (Issue 2) or only work with specific cases (Issue 1). Therefore, I will recommend weak acceptance.

issues: 1. In lines 136-138, “In sequential grasping scenarios, scenes often change by only the removal of the last object grasped. To take advantage the information already trained, we use the NeRF  network weights from the previous grasp in the subsequent grasp.” In the sequential grasping scenarios, some objects might move (such as translate or even rotate) due to collision, is the strategy of reusing NeRF weights still useful?

2. In lines 144-145, “we adapt the total-variation regularization loss (TV-loss) from Plenoxels [34] to discourage floaters and encourage smooth scene geometry” Does the regularization loss result in larger depth estimation error, thus decreasing the grasp performance if the object surfaces are not smooth?
					 				
3. In line 237, “ If a grasp planner generates a wrong  grasp leading to a joint over-torque error, we terminate the experiment as a failure.” It will strengthen the paper if the failure cases are also contained in the real-world video. 							

4. In line 259, “GT-Net achieves 0 %, Rad-Net achieves 42 %, and Dex-Net achieves 0.1%, suggesting that there is a  large distribution shift from training on ground-truth depth to testing on NeRF-depth in simulation.” What is the error between predicted depth maps and ground-truth depth maps in these grasping regions? It will strengthen the paper if failure case analyses are provided.”
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a method for robotic grasping of transparent objects combining a NeRF-based perception and rendering module with a CNN-based grasp network. To reduce the long training time required by previous publications on grasping transparent objects with NeRF-based perception, the authors propose several improvements:
- They leverage a recent NeRF-variant, which reduces training time drastically
- They train on images captured during robot motion and use a predicted confidence score to decide when to stop the capturing phase
- They re-use weights of previously trained NeRFs for faster training in evolving scenes
Furthermore, to cope with the inaccuracies resulting from capturing during motion, the authors propose a geometric regularization.
strengths_and_weaknesses: Strength:
- The method achieves a significant reduction in training time compared to previous methods

Weaknesses:
- Even though the grasp inference time has been reduced drastically compare to previous methods, the proposed approach is still too slow for practical applications. This should be discussed in the limitations.
- There are some unclarities listed below in the issues. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The suggested method drastically improves inference time for robotic grasping of transparent objects. The improvements the authors propose compared to previous work seem interesting and effective. Though the overall system is probably still too slow for real world use cases, the proposed method is a good step towards that goal.
issues: Section 5.4:
- Did the authors test, whether the graspability heuristic used here is actually reliable, i.e. whether the predicted grasps actually lead to successfully grasped objects?

Section 5.5: 
- Why are different numbers of views used in simulation and in the real setup? Adding additional views in simulation should be cheap. 

Experiments: 
It would be interesting to see, how a method capturing less images from fixed view points performs as compared to the proposed method capturing images while moving. 

Minor: 
Do the authors have any intuition of why their method cannot cope with adding objects to the scene?
review_title: null

Meta-review: metareview: The paper introduces a new method for sequential robotic grasping by learning NeRF models of a scene during grasping. The novelty of the method is on how to train NeRF models faster during grasping as well as a grasping network adapted to images generated from NeRF models. Overall, the proposed method is novel and solves an important grasping problem in robotics. The experimental results demonstrate the effectiveness of the proposed method. Concerns from the reviewers are about unclear parts in the paper.

The concerns from the reviewers have been successfully addressed during the rebuttal. The authors are encouraged to revise the final paper accordingly.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)