Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent Objects
Authors: Justin Kerr
Letian Fu
Huang Huang
Yahav Avigal
Matthew Tancik
Jeffrey Ichnowski
Angjoo Kanazawa
Ken Goldberg
Venue: CoRL 2022 Oral (2022)
Keywords: NeRF
grasping
transparent objects
speed

Abstract: Sequential robot grasping of transparent objects, where a robot removes objects one by one from a workspace, is important in many industrial and household scenarios. We propose Evolving NeRF (Evo-NeRF), leveraging recent speedups in NeRF training and further extending it to rapidly train the NeRF representation concurrently to image capturing. Evo-NeRF terminates training early when a sufficient task confidence is achieved, evolves the NeRF weights from grasp to grasp to rapidly adapt to object removal, and applies additional geometry regularizations to make the reconstruction smoother and faster. General purpose grasp planners such as Dex-Net may underperform with NeRF outputs because there can be unreliable geometry from rapidly trained NeRFs. To mitigate this distribution shift, we propose a Radiance-Adjusted Grasp Network (RAG-Net), a grasping network adapted to NeRF's characteristics through training on depth rendered from NeRFs of synthetic scenes. In experiments, a physical YuMi robot using Evo-NeRF and RAG-Net achieves an 89% grasp success rate over 27 trials on single objects, with early capture termination providing a 41% speed improvement with no loss in reliability. In a sequential grasping task on 6 scenes, Evo-NeRF reusing network weights clears 72% of the objects, retaining similar performance as reconstructing the NeRF from scratch (76%) but using 61% less sensing time. See https://sites.google.com/view/evo-nerf for more materials.


Reviews:
[{"review_id": "wmSq9E69AEN", "reviewer": "Reviewer_UJkS", "summary": "The authors propose a method for robotic grasping of transparent objects combining a NeRF-based perception and rendering module with a CNN-based grasp network. To reduce the long training time required by previous publications on grasping transparent objects with NeRF-based perception, the authors propose several improvements:\n- They leverage a recent NeRF-variant, which reduces training time drastically\n- They train on images captured during robot motion and use a predicted confidence score to decide when to stop the capturing phase\n- They re-use weights of previously trained NeRFs for faster training in evolving scenes\nFurthermore, to cope with the inaccuracies resulting from capturing during motion, the authors propose a geometric regularization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The method achieves a significant reduction in training time compared to previous methods\n\nWeaknesses:\n- Even though the grasp inference time has been reduced drastically compare to previous methods, the proposed approach is still too slow for practical applications. This should be discussed in the limitations.\n- There are some unclarities listed below in the issues. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The suggested method drastically improves inference time for robotic grasping of transparent objects. The improvements the authors propose compared to previous work seem interesting and effective. Though the overall system is probably still too slow for real world use cases, the proposed method is a good step towards that goal.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Section 5.4:\n- Did the authors test, whether the graspability heuristic used here is actually reliable, i.e. whether the predicted grasps actually lead to successfully grasped objects?\n\nSection 5.5: \n- Why are different numbers of views used in simulation and in the real setup? Adding additional views in simulation should be cheap. \n\nExperiments: \nIt would be interesting to see, how a method capturing less images from fixed view points performs as compared to the proposed method capturing images while moving. \n\nMinor: \nDo the authors have any intuition of why their method cannot cope with adding objects to the scene?"}}, {"review_id": "RvuLnAq6YKm", "reviewer": "Reviewer_NJjN", "summary": "This manuscript introduces a method to progressively update and quickly train NeRF models to grasp transparent objects. The authors propose to reuse weights for sequential grasping tasks, use an early stopping strategy to speed up the completion, and add regularization loss to mitigate the floaters issue. To reduce the sim2real gap, the authors propose to train the network to predict grasps directly on NeRF-rendered depths maps. The proposed pipeline is applied in real-world experiments, achieving an 89% success rate for single object grasping and reducing the capture time significantly for both single object grasping and sequential grasping tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy to follow. \n2. It’s exciting to see that the proposed grasping pipeline only takes seconds to grasp a transparent object with the NeRF model.\n\nWeaknesses:\n1  It’s not clear whether the proposed strategy of regularization is generally applicable or not (See Issue 2).\n2. More failure case analyses and real-world videos of these failure cases actually strengthen the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The manuscript introduces a pipeline for robots to grasp transparent objects taking only seconds per object, which is useful and important for many real-world robotic applications. But it’s not clear whether the proposed strategies are generally helpful or not (Issue 2) or only work with specific cases (Issue 1). Therefore, I will recommend weak acceptance.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. In lines 136-138, “In sequential grasping scenarios, scenes often change by only the removal of the last object grasped. To take advantage the information already trained, we use the NeRF  network weights from the previous grasp in the subsequent grasp.” In the sequential grasping scenarios, some objects might move (such as translate or even rotate) due to collision, is the strategy of reusing NeRF weights still useful?\n\n2. In lines 144-145, “we adapt the total-variation regularization loss (TV-loss) from Plenoxels [34] to discourage floaters and encourage smooth scene geometry” Does the regularization loss result in larger depth estimation error, thus decreasing the grasp performance if the object surfaces are not smooth?\n\t\t\t\t\t \t\t\t\t\n3. In line 237, “ If a grasp planner generates a wrong  grasp leading to a joint over-torque error, we terminate the experiment as a failure.” It will strengthen the paper if the failure cases are also contained in the real-world video. \t\t\t\t\t\t\t\n\n4. In line 259, “GT-Net achieves 0 %, Rad-Net achieves 42 %, and Dex-Net achieves 0.1%, suggesting that there is a  large distribution shift from training on ground-truth depth to testing on NeRF-depth in simulation.” What is the error between predicted depth maps and ground-truth depth maps in these grasping regions? It will strengthen the paper if failure case analyses are provided.” \t"}}, {"review_id": "MrdCcFvroC", "reviewer": "Reviewer_iTZS", "summary": "This paper studies the problem of grasp synthesis of transparent objects using RGBD images. The authors propose a pipeline based on Neural Radiance Fields (NeRFs), which they modify in several ways to make the training process amenable to real-time robotics use. These include interleaving dataset updates with NeRF training, additional loss terms and hyperparameter adjustments, and early stopping. They further propose to train a grasp planing network (based on prior work) directly on the NeRF depth images, rather than the images generated by the depth sensor. The authors study their method's performance onboard a physical robot, and demonstrate much faster performance than DexNet, fast re-training of the NeRF for decluttering, and ablate the proposed NeRF modifications to clearly demonstrate their contribution.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The proposed method can quickly train a NeRF representation, and works for transparent objects, a current open problem in grasp synthesis from visual data.\n\n+ The hardware experiments are thorough and demonstrate a clear improvement over (approximately) the state of the art, DexNeRF. The modifications to the NeRF training, in particular, are novel and appear quite useful.\n\n- The method requires a somewhat prescribed \"capture trajectory\" to build the NeRF model, although this is usually stopped early once the grasp generation is confident.\n\n- The grasp planning network has a Sim2Real issue (trained in sim, deployed on real objects) although the authors claim training on NeRF depth ameliorates this somewhat.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Strong accept. The proposed modifications to the NeRF training make it applicable for real-time use in robotics, and are ablated to demonstrate their benefits in real hardware. Grasp synthesis for transparent/reflective objects is still quite challenging, and the authors present a method that seems to do this quite effectively. These are exciting, compelling, and novel results.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- I'm not sure the Sim2Real gap is discussed in enough detail in the text. Section 5.5 seems to address this, but mostly reports that training grasp networks on RGB-D images from reflective objects (this is the motivation of the paper?). I would argue a better experiment would be to test this on opaque, matte objects -- if the simulation-trained grasp planner performs similarly to DexNet in this case, you have made a strong case that the Sim2Real gap is quite small.\n\n- In general, the text in the experiments section suffers from a lack of clarity (Section 5.5 was particularly hard to parse). I think this part of the paper would benefit from some editing for clarity/grammar. \n\n- This is more a direction for future work, but it seems there would be better ways to generate the capture trajectory based on error metrics rather than following a path on a sphere. \n\n- Similarly, I found the \"deletion\" mechanism a bit less-than-compelling. This isn't an issue that should prevent publication, but it seems a more elegant way to handle the problem would be to segment the NeRF into several smaller objects that are composited during rendering."}}]

Author rebuttal:

Decision: Accept (oral)