Active Perception for Grasp Detection via Neural Graspness Field

Authors: haoxiang  ma
modi  shi
boyang  gao
di  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 03:54:20

Abstract: This paper tackles the challenge of active perception for robotic grasp detection in cluttered environments. Incomplete 3D geometry information can negatively affect the performance of learning-based grasp detection methods, and scanning the scene from multiple views introduces significant time costs. To achieve reliable grasping performance with efficient camera movement, we propose an active grasp detection framework based on the Neural Graspness Field (NGF), which models the scene incrementally and facilitates next-best-view planning. Constructed in real-time as the camera moves, the NGF effectively models the grasp distribution in 3D space by rendering graspness predictions from each view. For next-best-view planning, we aim to reduce the uncertainty of the NGF through a graspness inconsistency-guided policy, selecting views based on discrepancies between NGF outputs and a pre-trained graspness network. Additionally, we present a neural graspness sampling method that decodes graspness values from the NGF to improve grasp pose detection results. Extensive experiments on the GraspNet-1Billion benchmark demonstrate significant performance improvements compared to previous works. Real-world experiments show that our method achieves a superior trade-off between grasping performance and time costs.

Main paper: 
 introduction: 
 Learning-based robotic grasp synthesis [24] has been explored to enable the manipulation of various objects across different grippers, sensors, and scenarios. The completeness and accuracy of the scene representation significantly influence the performance of these methods, as the geometric ambiguity can confuse the synthesis of grasp poses. To address this issue, previous works [4, 8, 20] have employed multi-view information to reconstruct different 3D representations. However, as the robot needs to move when observing from different views, scanning the entire scene to achieve complete coverage incurs a substantial time cost and is challenging to apply in real-world environments.To enhance the efficiency of multi-view perception for robotic grasping, previous works have introduced active perception methods to select the Next-Best-View (NBV). Part of the methods [13, 7, 5] apply active 3D reconstruction techniques, treating grasp detection as a secondary task. However, active reconstruction tends to select viewpoints that cover more unobserved space, which may not be optimal for grasp detection, as different regions have varying relevance to grasping. Consequently, active perception strategies based on 3D reconstruction have limited performance in grasp detection tasks. Recently, ACE-NBV [32] combines grasp detection and NBV planning by specially designing Figure 1 : Overview of the active grasp detection system. The RGB, depth and predicted graspness from a new view are mapped to the Neural Graspness Field (NGF) by rendering loss. After each mapping step, the scene geometry is exported from the neural representation using the marchingcubes algorithm [19] and the candidate positions for grasp synthesis are sampled by neural graspness sampling. If the maximum perception step is reached or a specific result condition is satisfied, e.g., a sufficient number of high-quality grasps are detected, the robot arm is employed to execute the detected grasps. Otherwise, the Next-Best-View (NBV) planner is employed to sample candidate views and select the view with the largest information gain for robot movement. a grasp detection network to predict the grasp affordance of candidate views. The viewpoint with the highest predicted grasp quality is then selected for reconstruction to efficiently obtain feasible grasps. However, this approach has two main limitations. First, it suppresses the perception of views that have low grasp quality but provide grasp-related geometric information, which can lead the method to fall into local optima. Second, directly predicting information gain through a specially designed grasp detection network can be limited by the network's generalization ability, potentially resulting in performance degradation when applied to novel environments.To address the aforementioned issues, inspired by the development of the neural representation and radiance field [21, 14] , we propose to employ a Neural Graspness Field (NGF) to model the scene grasp distribution and build the active grasp detection system based on it. The graspness of a position measures the sum of feasible grasps in its pose space. The NGF is optimized online during the camera movement by back-propagating the rendering loss of the view graspness. The online-training scheme employed for NGF modeling makes it easier to transfer to unseen scenarios. Moreover, distilling multi-view information into 3D space using the neural representation enforces the multi-view consistency, making the NGF less susceptible to the depth noise and view sparsity compared to directly predicting the grasp distribution on a reconstructed 3D representation [33] . With the neural representation of the scene grasp distribution, the active perception problem can be defined as minimizing the error of the scene grasp distribution modeled by NGF. This is achieved by strategically selecting views that can bring the largest information gain for the NGF after mapping, thereby incrementally refining the modeled grasp distribution towards an optimal state.In this paper, we propose an active perception method for the robotic grasp detection consisting of two components: neural graspness field mapping and graspness inconsistency-guided next-bestview planning. For neural graspness field mapping, we extend a NeRF-based real-time mapping system [12] to render view graspness by adding a separate network branch. The view graspness is predicted by a pre-trained graspness network from the corresponding depth image. For NBV planning, we provide a graspness inconsistency-guided strategy targeting minimizing the inconsistency between the current NGF and the ground-truth scene grasp distribution. For a given view, its information gain is described as the inconsistency between the view graspness rendered from the NGF and the pseudo label predicted from the rendered depth image. Furthermore, we propose an inference strategy based on our active grasp detection framework, which decodes the graspness score from the NGF to generate grasp samples instead of predicting from explicit 3D geometries. The contributions of this paper can be summarized as follows:• We propose an active grasp detection framework via neural graspness field to model the scene grasp distribution online during camera movement.• We adopt a graspness inconsistency-guided strategy for next-best-view planning, which targets on reducing the uncertainty of the neural graspness field.• A neural graspness sampling inference strategy is proposed to enhance the performance of the grasp detection framework.2 Related Work 
grasp detection: 
 Grasp detection aims to generate feasible and diverse gripper poses for given objects. Early methods mainly studied the theoretical framework for robotic grasping by analyzing the contacts between the gripper and object models. Recently, to achieve grasping of unseen objects, data-driven grasp detection methods have been extensively studied. Some works [9, 34, 1] investigate grasp detection in planar space, simplifying the problem by directly locating rotated grasp rectangles on images. To generate more diverse grasps that support complex downstream manipulation tasks, 6-DoF grasp detection has been proposed to predict grasping in SE(3) space. [30] samples gripper poses on point clouds using geometric priors and scores these samples using a Convolutional Neural Network (CNN). [18] follows the sample-and-score framework but introduces PointNet [27] to achieve better scoring performance. Some recent works adopt end-to-end networks for 6-DoF grasp detection to achieve real-time inference. [23] employs a variational autoencoder for grasp generation given the object point clouds, and [28] proposes a single-shot grasp proposal network to enhance the efficiency of grasp detection. [10] densely annotates grasp labels in clutter to construct a large benchmark and provides an end-to-end 6-DoF grasp detection baseline trained with the large amount of annotations. Following this, [31] defines graspness to represent the grasp distribution in a scene and proposes a graspness discovery method. Although the aforementioned methods have achieved good results, using single-view point clouds as input leads to a lack of geometric information due to occlusions and limited field of view, which affects the grasping performance on some objects. To solve this problem, [4, 20] utilize Truncated Signed Distance Functions (TSDFs) to map multi-view frames before generating grasps, and [8] employs a generalizable neural radiance field to reconstruct the scene from cameras set around the scene, which supports the grasp detection of transparent and specular objects. However, these methods rely on scanning the scene from all possible views or several views surrounding the scene, resulting in excessive robot execution time for moving the camera. The large time cost reduces the usability of grasp detection methods in real-world scenarios.Active perception [3] aims to develop an agent that knows why it wishes to sense, and then chooses what to perceive, and determines how, when and where to achieve that perception. Planning the next-best-view is a commonly used method to realize an active perception system. In terms of grasp detection, active perception is introduced to determine the camera views that can achieve a trade-off between the grasp performance and time cost. There are two main lines of work in this area: those that treat grasp detection as a secondary task of 3D reconstruction and those that directly incorporate grasp detection into the view planning process. Some works [13, 2, 7, 5] plan the view sequence based on the 3D reconstruction metric of the grasp-relevant region, where the grasp detection is treated as a secondary task. [13] models the uncertainty of occluded voxels as a mixture of Gaussians and utilizes trajectory optimization to generate the view sequence. [2] provides an active vision approach to maximize surface reconstruction quality near the contact point region, and [7] adopts reinforcement learning for view planning with an object mask-guided reward function. To achieve close-loop control, [5] incorporates a grasp detection network to continuously predict grasps after each view mapping. However, employing grasp detection as a secondary task of 3D reconstruction overlooks the internal relation between grasp synthesis and scene reconstruction, which can lead to sub-optimal results. Another line of works [11, 22, 32] directly incorporates grasp detection into the view planning process. [11] maps grasp detection performance as a function of viewpoint for each object but struggles with novel objects. [22] uses the entropy of the network prediction to determine the next-best-view in a top-down grasping setting and recently, [32] proposes an affordance-driven policy based on an implicit grasp detection network to generate grasp affordance for unseen views. However, these approaches that utilize the output of grasp detection networks for view planning rely on the specific design of the network and are easily influenced by the generalization ability of the grasp detection network, making it challenging to apply these methods to diverse real-world scenarios.In contrast to these approaches, our method utilizes view grasp distribution, which is independent of specific network designs, and constructs the NGF online. This enables our method to easily adapt to diverse real-world scenarios at test time.3 Method 
system overview: 
 The objective of grasp detection in cluttered scenes is to generate diverse feasible grasps for each object in the scene. Given a robotic arm with a mounted depth camera, active grasp detection aims to find a camera movement policy that can achieve high-quality results within a maximum of T time steps. An overview of our active grasp detection system is provided in Fig. 1 . Inspired by neural feature fields [15] and semantic NeRF [33] , we incorporate a graspness field to represent the scene grasp distribution. The NGF employs a separate branch besides the appearance and geometry to render grasp distribution information from multiple views. The graspness score proposed in [31] measures the graspable landscape in cluttered scenes given a position p. With L grasp candidates G p = {g p k |k = 1, ..., L} sampled in its configuration space, i.e., approach direction, gripper depth and in-plane rotation, the ground-truth graspness score g p is defined as:EQUATIONwhere q k is the grasp quality score computed from force closure analysis, c k indicates the collision state of the gripper in clutter and 1 is the indicator function. For an observed view, the NGF aims to map the view appearance, depth and graspness, as shown in Fig. 2 (a). The NGF is composed of two parts: axis-aligned feature planes and SDF-based volume rendering. The axis-aligned feature planes store learned features at different resolutions, which are queried and interpolated based on the 3D positions of the sampled points. For points p sampled in a ray, the corresponding features are queried from the feature planes and decoded by MLPs to get raw color ϕ c (p), raw Truncated Signed Distance Field (TSDF) ϕ t (p) and raw graspness ϕ g (p). To convert raw TSDF values to volume densities, the SDF-based volume rendering from StyleSDF [25] :EQUATIONwhere β is a learnable parameter. With the volume density, the color, depth and graspness of each ray r can be computed as:EQUATIONwhere z n is the ray depth of p n and weight w n is formulated as:EQUATIONFor a given view, it is challenging to directly obtain the corresponding ground-truth graspness during online mapping. Therefore, we use a pre-trained graspness network that takes the depth image as input and predicts the view graspness to render the NGF. By rendering from different views, the graspness field can reduce the graspness noise caused by single-view depth and render the graspness distribution from any given view, which can be used to measure the grasp-correlated information gain.Given a sequence of viewpoints s = (v 1 , v 2 , ..., v N ) and sequence set S, the NBV planning problem can be defined as:EQUATION)where N is the maximum step of robot movement and I(v n ) represents the information gain of viewpoint v n . The selection of an informative view is influenced by the scene representation and the definition of information gain. We consider the NGF obtained from observing the entire scene as the ground-truth scene grasp distribution and define the information gain of a view as the improvement in the NGF prediction by mapping this view. Given an unseen candidate view, the NGF can estimate the view graspness based on the existing observations, and the improvement is expressed as the inconsistency between the ground-truth graspness g(d) predicted from the real depth and the graspness ĝ rendered by the current graspness field.However, for the NBV problem, obtaining the real depth image of a candidate view is not possible. To address this, our inconsistency-guided NBV policy adopts the pseudo-label paradigm by substituting the ground-truth grasp distribution g(d) with the pseudo-graspness g( d), which is widely employed in other semi-supervised and active learning vision tasks [17, 29] . As shown in Fig. 2 (b), we leverage the NGF's ability to render a pseudo-depth image d of the candidate view by volume rendering the TSDF values and then pass this pseudo-depth image through the graspness prediction network to obtain the pseudo-graspness g( d). The pseudo-graspness is used to calculate the information gain of the candidate view, which is defined as:EQUATIONwhere g(r) represents the rendered graspness of sampled ray and the summation symbol represents the rendered graspness of the whole view.It should be noted that the effectiveness of our NBV policy relies on the premise that the g( d)predicted by the graspness network is closer to the ground-truth g(d) compared to ĝ. The smaller error in g( d) can be attributed to two reasons: First, the robot-mounted camera moves continuously in small steps, resulting in minimal differences in the observed geometric information between views, leading to insignificant errors in the rendered depth d. Second, the graspness prediction network, trained on a large dataset of real point clouds, inherently provides robustness to depth noise. We visualize the rendered graspness error E ĝ = |g(d) -ĝ| and the pseudo-graspness errorE g( d) = |g(d) -g( d)| in Fig. 3 (a), where E ĝ is significantly larger than E g( d) and the difference decreases with more steps.The proposed pseudo-graspness information gain can incorporate the view grasp distribution prior into the planning process, which is encoded by the pre-trained graspness network. Thus the NBV system can select the view containing the most graspness information that has not been distilled from the pre-trained network to the NGF. We visualize the ground-truth graspness g(d), pseudo-graspness g( d), rendered graspness ĝ and the information gain I in different views of the neural graspness field in Fig. 3 (b ). For different views, the pseudo-graspness predicted from the rendered depth image can approximately represent the ground-truth g(d) but the accuracy of the rendered graspness ĝ varies, which introduces different information gains. For the grasp detection method, predicting the grasp distribution in clutter and sampling positions for grasp synthesis is an important part. Previous methods usually employ different encoders to predict grasp distribution from explicit 3D representations but the incomplete and noisy geometry information can lead to inaccurate grasp distribution. To achieve more precise grasp sampling, in addition to using the NGF for active perception, we propose an inference strategy based on sampling from the NGF. Given the positions p sampled from the reconstructed surface, the graspness of the position can be decoded from the NGF. During inference, we replace the graspness sampled from the neural representation with the graspness predicted by the grasp detection network and utilize Furthest Point Sampling (FPS) on the positions larger than a threshold T to get positions for grasp synthesis, which is formulated as:EQUATIONwhere ϕ g is the graspness branch of the NGF.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: Our paper doesn't contain theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental setup: 
 Simulation Setup We construct a simulation active grasp benchmark based on the GraspNet-1Billion benchmark [10] , which consists of 100 scenes for training and 90 scenes for testing. The test set is divided into seen, similar, and novel sets based on the included objects. Each scene is captured from 256 views using Intel RealSense and Kinect cameras. We conduct all the experiments with the data captured by the Realsense camera. We set the pre-collected 256 views as the perception space for NBV planning. Since moving the camera is a continuous process, moving it over long distances would waste the information captured during the movement. Therefore, we sample the candidate views from the current view with a relatively small step size. In our experiments, we set the step size to 10cm and set maximum step to 10. For evaluation, we follow the metric used in the GraspNet-1Billion benchmark, which simulate grasps with friction µ ranging from 0.2 to 1.2 with interval δµ = 0.2. Following [20] , we sample 5 grasps for each object in the scene to calculate the average precision AP. The training and evaluation of the simulation experiments are conducted on a single NVIDIA V100 GPU.Baselines We compared the following baseline methods to validate the effectiveness of our proposed method. The baselines can be divided into two categories: NBV for robotic grasping [5, 32] and NBV planning based on NeRF [26, 16] . Close-loop NBV [5] utilizes ray casting to calculate the number of unobserved voxels of objects, which drives exploration targeting occluded object parts. ACE-NBV [32] incorporates grasp affordance prediction into NBV planning and selects the view with the largest grasp affordance as the next-best-view. ActiveNeRF [26] proposes an plug-in uncertainty estimation method for NeRF based on Bayesian estimation and Uncertainty-policy [16] computes the entropy of the weight distribution of each ray as the uncertainty.Implementation Details For the mapping of NGF, the first view is trained for 100 iterations and following views are trained for 50 iterations. For each ray, 32 points are sampled for stratified sampling and 8 points for importance sampling. Only the coarse planes in [12] are employed for mapping. For NBV planning, we downsample the original image to 1/8 to sample rays for graspness rendering to speed up the computation of view information gain. We utilize the first-stage network of GSNet [31] which predicts the graspness score for point-clouds as the graspness network in this paper. For the grasp detection network used for inference, we adopt the baseline method from [20] which uses the reconstructed scene geometry as input. For each scene, 1024 points are sampled from the NGF for grasp pose synthesis. To validate the effectiveness of the proposed NGF and the graspness inconsistency-guided NBV policy based on it, we re-implement other view planning policies on the same ESLAM mapping framework [12] which the NGF is built on. As shown in Fig. 4 , our pseudo-graspness guided policy achieves superior performance after the first several views on seen, similar, and novel sets. Compared to ActiveNeRF [26] and Uncertainty-policy [16] , our method selects views with more grasp distribution discrepancy instead of geometry or appearance ambiguity in the neural representation, which improves the results. Compared to policies targeting grasp detection [5, 32] , our method is specially designed to reduce the uncertainty of the NGF by distilling the prior knowledge of a pre-trained network, thus achieving superior performance with more views. It should be noted that ACE-NBV [32] can achieve comparable results to ours in the initial steps while showing little improvement as more views are added. This is because the affordance-based policy only selects views with more feasible grasps but does not consider optimizing the scene grasp representation throughout the entire planning process. We apply the neural graspness sampling during the inference of the grasp detection network to validate its effectiveness. The results are shown in Fig. 5 . The sampling strategy improves the grasp detection results on seen, similar and novel objects at each step. Constructing a NGF through online multi-view rendering, compared to directly predicting the grasp distribution from 3D geometric information using the network, can reduce the errors in the scene grasp distribution caused by incomplete geometric information. Furthermore, since the optimization is performed online for each scene, it demonstrates better robustness compared to direct network prediction and thus the results on the novel set improve significantly. Overall Performance We compare the overall performance after 10 views of our method with previous active grasp detection methods [5, 32] on the GraspNet-1Billion benchmark, as shown in Table 1 . We employ the same grasp detection network trained on the GraspNet-1Billion benchmark for all these methods. Our active grasp detection method improves the performance by 8.38%, 6.71%, 2.98% on the seen, similar and novel sets compared to ACE-NBV [32] for grasp detection in clutter, demonstrating the effectiveness of the proposed method. All views represents a complete reconstruction using all 256 views, serving as an upper-bound reference for active perception methods. We conduct real-world experiments of the proposed active grasp detection method on a 6-DoF UR-10 robot arm with a mounted RealSense D435i depth camera. The robot setup and objects used for experiments are shown in Fig. 7 . We select 25 objects from the YCB dataset [6] with various sizes and shapes for grasp detection. In the experiments, each cluttered scene is composed of 5 objects and we place these objects in different poses to evaluate each scene for 3 times. We employ the grasp success rate as the metric. As shown in Table 2 , our method achieves 12.00% and 4.00% improvement on success rate compared to ACE-NBV [32] and Close-loop NBV [5] , respectively.Overall NBV Planning Mapping Grasp Detection Robot Execution 3.44s 1.00s (29.07%) 0.45s (13.08%) 0.23s (6.69%) 1.76s (51.16%) Table 3 : Runtime analysis of the proposed method.Runtime Analysis We provide a runtime analysis of the proposed active perception system, as shown in Table 3 . The analysis is performed on a workstation with a single NVIDIA 3090 GPU and an AMD Ryzen 5 2600 six-core processor. The average execution time for each step is 3.44 seconds, with the robot execution accounting for approximately 50% of the total time. In the active grasp detection system, the majority of the time is consumed on NBV planning, while updating the NGF (mapping) and grasp detection take a relatively small proportion of the time. By investing some time in NBV planning, we achieve a trade-off between the performance and time cost compared to scanning the entire scene. Fig. 8 visualizes the NGF with different perception views, where the yellow region represents a higher grasp probability. It can be observed that the NGF can not only reconstruct the 3D geometry of the scene but also jointly model the graspness. With approximately 5 views planned using active grasp detection, the NGF can effectively model the grasp distribution of objects. As more steps are taken, the details of the geometry and grasp distribution of the scene can be incrementally refined. Fig. 9 illustrates the view trajectories obtained by different view planning strategies. The close-loop NBV [5] approach, which employs unobserved space as the metric, guides the camera view path to scan regions with minimal overlap with the currently observed areas, aiming to maximize scene coverage. However, this method does not prioritize the graspable regions of the objects. In contrast, ACE-NBV [32] incorporates a grasp detection network to guide view planning by selecting views with the highest grasp affordance. Nevertheless, this approach tends to repeatedly scan a limited region, potentially leading to sub-optimal local results. Compared to these methods, our proposed approach efficiently scans the grasp-correlated regions of the scene while ensuring comprehensive scene reconstruction.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide necessary information to reproduce the experimental results, including hyper-parameters and the related papers. 
limitations: 
 Our approach has two limitations. First, the time cost of NBV planning is positively correlated with the number of candidate views. When more refined view sampling is required, the planning time increases. Since our information gain computation is differentiable, this issue may be alleviated by sparse view sampling combined with pose optimization for the selected views. Second, our method cannot handle dynamic scene changes. Although efficient for static scenes, when grasping fails, e.g., objects fall out of the gripper or change pose without being grasped, the robot must re-execute the perception process. Incorporating techniques used in dynamic radiance fields could potentially address this problem.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We used a public dataset in the simulation experiments, and the related code will be open sourced after the paper is accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we propose an active perception method for grasp detection by introducing the neural graspness field, which models the grasp distribution of a scene. By rendering the graspness predicted from a pre-trained network for each view, the NGF can be optimized online and reduce the noise of graspness in each view. Based on which, we introduce a graspness inconsistency-guided NBV policy to select the view with the largest inconsistency between the rendered graspness and pseudo-graspness label. Furthermore, we introduce neural graspness sampling to decode the grasp distribution from the neural representation, which benefits the position sampling of grasp pose synthesis. The experiments conducted on the simulation and real-world settings demonstrate the effectiveness of the proposed active grasp detection method. Overall performance on the kinect camera We compare the overall performance of kinect camera on the GraspNet-1Billion benchmark, as shown in Table 4 . Our active grasp detection method improves the performance by 7.61%, 6.78%, 0.38% on the seen, similar and novel sets compared to ACE-NBV [32] for grasp detection in clutter. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide the necessary training and test details of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We provide the standard deviation in the simulation experiments by inference 5 times on each scene.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the information about the computer resources for simulation and real-world experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This paper conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper is a purely academic work and has no societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: MethodsThe scene setting for real-world experiment is shown in Fig. 10 . In total, we constructed 5 scenes, each containing 5 objects. For each scene, we repeated the experiment 3 times by changing the poses of the objects within the scene. The number of the success attempts for each scene is provided in Table 5 .Justification: We discuss the limitations of our paper in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: The paper presents an active perception framework for robotic grasp detection, demonstrating improved performance on a large-scale benchmark and real-world validation. Reviewers appreciated the novelty but raised questions about calculating information gain, the clarity of neural grasp sampling, and the details of experimental setups. The authors responded to these concerns with additional explanations and data, satisfactorily addressing the reviewers' issues.
Considering the overall positive reviews, the authors' effective rebuttal, and the potential contributions to the field, the paper is recommended for acceptance.

Reviews: review: paper_summary: paper_summary: This work proposes an active perception method for grasp detection composed of two parts: neural graspness field mapping and next-best-view planning with a graspness inconsistency-guided strategy. And a corresponding inference strategy is also proposed by decoding the graspness score from NGF to generate grasp samples. The evaluation benchmark is constructed based on the GraspNet-1B benchmark and the experiment results indicate consistent improvements on the seen, similar and novel sets.
strength_weakness: strength_weakness: 1. Relatively small performance improvement on the novel set compared to ACE-NBV.
2. Generalization and distraction experiments are not provided to verify model's robustness.
questions: questions: 1. Could you account for why with plan step increases, for the novel set, your method encounters a significant drop in performance?
2. Could you provide further generalization and distraction experiments to verify your model's robustness?
limitations: limitations: 1. The NGF-based method may be constrained to static environments and cannot adapt to dynamic scenarios.
paper_summary: paper_summary: The paper introduces a novel framework utilizing a Neural Graspness Field (NGF) in conjunction with a pre-trained graspness prediction network to enhance active grasp detection. It applies online training to the NGF upon encountering a new scene view, producing RGB, depth, and graspness maps. The method involves computing information gains from potential views by assessing discrepancies between the NGF-generated and pre-trained network graspness maps to select the most informative views. During grasp inference, the framework samples grasp poses guided by NGF-derived graspness scores and samples them using the Farthest Point Sampling (FPS) method.
strength_weakness: strength_weakness: 1. A primary concern is the incremental training approach of the NGF. Since the NGF is trained incrementally, the rendered graspness maps \hat{g} and depth maps might be inaccurate for novel views at the initial stage, and thus are inappropriate to be used to compute information gain. 
2. The NGF's grasp knowledge, being distilled from the pretrained graspness network, suggests that its maximum achievable performance may be inherently limited to that of the teacher network. How can you demonstrate that the proposed framework is better than the single-shot method? 
3. According to Eqn(1), the graspness score of a position is defined as the mean grasp quality scores of different orientations on that position. When sampling grasp poses at the inference stage, it seems that they only sample positions for the grippers. How to determine the orientations?
4. Visual results (Figure 3) indicate that initial high information gains may be attributed more to background inconsistencies rather than the target object regions, potentially skewing the focus of planning efforts. The inconsistency of the background region should be neglected when planning.
questions: questions: 1. Despite the reported rapid mapping operation time of 0.45 seconds in Table 3, the paper notes extensive training iterations for both initial and subsequent views (100 and 50 iterations). Further details on achieving such fast training times would be beneficial for understanding the feasibility of the NGF training process.

2. Equation (6) introduces confusion regarding the role of the summation symbol and the meaning of the variable $r$. Clarification of these components would aid in a better comprehension of the equation's intent and application.
limitations: limitations: See weaknesses and questions.
paper_summary: paper_summary: This paper studies active perception for robotic grasp detection. It proposes an active grasp detection framework based on the Neural Graspness Field (NGF), which incrementally models the scene and facilitates next-best-view planning. For next-best-view planning, it aims to reduce the uncertainty of the NGF through a graspness inconsistency-guided policy, selecting views based on discrepancies between NGF outputs and a pre-trained graspness network. Additionally, it presents a neural graspness sampling method that decodes graspness values from the NGF to improve grasp pose detection results.
strength_weakness: strength_weakness: Main concern：
1. First of all, I regret that the authors do not submit video materials in the supplementary materials. Although the authors analyze the real-world performance in the paper, the lack of videos makes it difficult to further verify the authenticity and effectiveness of the real-world experiments
2. The GraspNet-1Billion dataset includes two cameras, as mentioned in line L196 of the paper. However, in Table 1, the authors only provide experimental results for one camera, which I find insufficient. Testing on different cameras can more comprehensively demonstrate the effectiveness of the method.
3. Although the authors conduct sufficient comparison experiments to demonstrate the effectiveness of their method, I think they lack analysis experiments on some hyperparameters. For example, the authors set the maximum step to 10. I would like to ask the authors what the basis for setting it to 10 is, and what would happen if I used 7 steps or even fewer?
4. I think the recording of real-world experiments is insufficient. The authors should at least document the number of scenarios designed, the objects used in each scenario, and the average accuracy for each scenario. Simply recording an average success rate in Table 2 without detailing the experimental setup is, in my opinion, not rigorous. Additionally, there are no example images of cluttered scenes, making it difficult to assess the complexity of these scenarios. Typically, five objects do not constitute a cluttered scene; cluttered scenes usually consist of 10-15 or even more objects.
5. In section 4.5, I cannot intuitively see the clear performance advantages or disadvantages between Close-loop NBV and "Ours". In my opinion, the differences are due to the distinct training methods of the two planners, resulting in varied decision-making. I suggest the authors provide more distinguishable trajectory visualizations (e.g., whether the trajectory planning of Close-loop NBV causes collisions with objects, thereby disrupting the scene).

Others：
1. In equation (3), the formula for Pn​ is missing a closing parenthesis. Additionally, equation (3) does not clearly indicate the elements of ray tracing that are being focused on. For example, in NeRF, estimating density only requires the 3D location, while estimating color involves both the 3D location and the 2D viewing direction. The authors do not clearly explain in equation (3) which information c(r), d(r), and g(r) utilize.
2. In Table 1, the term "All views" is unclear, and there is no corresponding explanation provided in lines L245-L250.
3. Although the authors conduct an ablation study on GraspnessSample, I find the ablation experiments insufficient. As cited in the paper, the effectiveness of Graspness is already thoroughly validated in the GSNet paper. Conversely, the authors do not provide sufficient ablation studies on other parts of the planner. I believe these other parts are the more critical aspects of this paper.
questions: questions: See Weaknesses.
limitations: limitations: The author mention the limitations of the paper and suggest possible solutions.
paper_summary: paper_summary: This paper proposes an next-best-view planning method for grasp perception. The authors use neural field to model the grasp distribution of a scene, which is learned from the graspness detection result of different views. Then the NBV is designed as the view with the largest inconsistency between the rendered graspness from the neural field and pseudo-graspness label predicted by the network on the rendered depth. Simulation and real-world experiements show that the proposed method is slightly bettern than exising methods.
strength_weakness: strength_weakness: First of all, the meaning of NBV for grasp percetpion is not very clear to me. The goal of grasp percetpion is to find ONE successful and executable grasp in the scene, rather than detecing all the feasible grasps. 

Secondly, the gain of the proposed method is not well validated by the experiments. In both simulation and real-world experiments, the proposed method is only slightly better than baselines. The margin is not significant.

Thirdly, the grasp network that is used to predict the graspness for neural field may produce inconsistent results across views. How do the authors address this problem during field optimization. The depth of the field for unseen views can also be errorneous.
questions: questions: 1. Sec 3.4 is not clear. Why do the authors need FPS? There is no ablation study on this design. Is the graspness score generated by the network or neural field?
2. Colorbar. should be added in Fig3, 8

3. How is the deviation between NGF and predicted graspness computed?
limitations: limitations: Yes, limitations are addressed in the paper: the time cost and the method cannot be used for dynamic scenes.

Comments: title: Official Comment by Reviewer 4AgP
comment: Thanks for your feedback and we are pleased that our response has successfully addressed your concerns.
Thank you for your response. I think it answered my question. Therefore, I am willing to raise my score to borderline accept.
Dear Reviewer peVt,

We appreciate your previous feedback and have provided our response to your questions. As the discussion phase is nearing its end on August 13, we want to check if you have had the opportunity to review our latest response.

If you have any further concern or require additional clarification, we are willing to address them within the remaining discussion period.

Thank you for your time and consideration.
We appreciate the reviewer for considering our response and are pleased that it has successfully addressed the concerns.
Thanks for your feedback and suggestion. We will add the ablation studies mentioned in the answer of Q2 in the next version.
Thanks for your feedback and we are pleased that our response addresses most of your concerns. As for the follow-up questions, we address them below:

**Q1**: We use $g(d)$ to guide the training of $\phi_g(p)$ because we observed that after mapping a view, the view-rendered depth $\hat{d}$ has better quality than the original depth map observed by the camera, with fewer missing points and smoother surfaces. Using the rendered depth as input to calculate view graspness achieves better results than directly using the original depth. Based on this observation, we employ a two-stage training process. First, we optimize the RGB and depth. Next, we use the rendered depth to calculate $g(\hat{d})$ for the guidance of graspness.

**Q2**: We understand your concern about the final performance, where our results reported in the paper are lower than GSNet, which only utilizes single view point-cloud as input. 

Firstly, we want to clarify that the target of GSNet and our method is quite different. Our method aims to build a complete 3D representation for grasp detection rather than design a grasp detection network, which means our method can be applied to any grasp detection method with 3D information as input. With actively reconstructed 3D information, grasp detection methods can achieve better performance especially for the unseen objects whose shape deviate from those in the training set. Unseen objects are not incorporated in the training process so the grasp detection netowrk struggles with the single-view input, where the geometry information is incomplete and noisy. Besides, utilizing multi-view information for 3D reconstruction can make the grasp detection method work on transparent and specular objects [1], while the single view depth map is in very low quality.

For the final result, in our original paper, we use the baseline grasp detection network from [2] rather than GSNet for evaluation. For a fair comparison, we provide the results inferred by pre-trained GSNet here. CD stands for collision detection. With the input reconstructed by 10 steps of active perception, GSNet performs better on similar and novel sets while showing a performance drop on seen objects. Our reconstructed data is not involved during training, so we tend to attribute the performance drop on seen objects to the distribution gap between reconstructed and single-view point clouds. We will attempt to include the reconstructed point clouds for training to achieve better results in the future.

| Method                | Seen-AP   | Similar-AP | Novel-AP  |
| --------------------- | --------- | ---------- | --------- |
| GSNet-singleview      | **65.70** | 53.75      | 23.98     |
| GSNet-10 steps        | 57.62     | **55.42**  | **24.53** |
| GSNet-singleview + CD | **67.12** | 54.81      | 24.31     |
| GSNet-10 steps + CD   | 61.78     | **61.60**  | **26.55** |



**References**

[1] Graspnerf: Multiview-based 6-dof grasp detection for transparent and specular objects using generalizable nerf, Dai et al., In *IEEE International Conference on Robotics and Automation*, 2023.

[2] Generalizing 6-dof grasp detection via domain prior knowledge, Ma et al., In *IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024.
Thank you for the response. Most of my concerns are addressed, and I raise my rate to `borderline accept'
Thank you for your response to my concerns. I will keep my score and hopefully, you can add generalization and distraction experiments as you mentioned in your answer to Q2.
I appreciate the authors' response, which addresses most of my concerns and helps me understand this work better. Here are some of my follow-up questions:

1. Since you already have the observed depth map, why do you use $g(\hat{d})$ to guide the training of $\phi_g(p)$ rather than $g(d)$?

2. For Q2, 'single-shot' refers to the first case mentioned above by the authors. It seems the single-shot method [1] already achieves better APs compared to the results reported in Table 1. 
Could the authors elaborate on the advantages of the proposed method compared to [1]?

[1] Wang, Chenxi, et al. "Graspness discovery in clutters for fast and accurate grasp detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
We appreciate the reviewer's feedback and are pleased that our response has addressed the concerns, resulting in a raised score. We will refine the ablation study and finish the Kinect camera experiment to refine our paper.
Thanks to the author's reply, your experiments address most of my concerns, so I'm willing to raise my score to "borderline accept". Hopefully, you will be able to refine your ablation studies in the next version as mentioned in Q8's reply.