Active Perception for Grasp Detection via Neural Graspness Field
Authors: Haoxiang Ma
Modi Shi
Boyang Gao
Di Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Robotic Grasping
Active Perception
Primary area: robotics

Abstract: This paper tackles the challenge of active perception for robotic grasp detection in cluttered environments. Incomplete 3D geometry information can negatively affect the performance of learning-based grasp detection methods, and scanning the scene from multiple views introduces significant time costs. To achieve reliable grasping performance with efficient camera movement, we propose an active grasp detection framework based on the Neural Graspness Field (NGF), which models the scene incrementally and facilitates next-best-view planning. Constructed in real-time as the camera moves, the NGF effectively models the grasp distribution in 3D space by rendering graspness predictions from each view. For next-best-view planning, we aim to reduce the uncertainty of the NGF through a graspness inconsistency-guided policy, selecting views based on discrepancies between NGF outputs and a pre-trained graspness network. Additionally, we present a neural graspness sampling method that decodes graspness values from the NGF to improve grasp pose detection results. Extensive experiments on the GraspNet-1Billion benchmark demonstrate significant performance improvements compared to previous works. Real-world experiments show that our method achieves a superior trade-off between grasping performance and time costs.

Active Perception for Grasp Detection via Neural Graspness Field
Haoxiang Ma Modi Shi Boyang Gao2 Di Huang!
State Key Laboratory of Complex and Critical Software Environment, School of Computer Science and Engineering, Beihang University, Beijing, China Geometry Robotics
wttp:/iiiannn22mmmm...mm dhuang))buaa. edu. cn, boyang wooagggg@g@geeeerorooooooooo
Abstract
This paper tackles the challenge of active perception for robotic grasp detection in cluttered environments. Incomplete 3D geometry information can negatively affect the performance of learning-based grasp detection methods, and scanning the scene from multiple views introduces significant time costs. To achieve reliable grasping performance with efficient camera movement, propose active grasp detection framework based on the Neural Graspness Field (NGF), which models the scene incrementally and facilitates next-best-We view planning. Constructed in real-time the camera moves, the NGF effectively models the grasp distribution in 3D space by rendering graspness predictions from each view. For next-best-vic sett-best-iiwwwww planning, we aim to reduce the uncertainty of the NGF through a graspness inconsistency- guided policy, selecting views based on discrepancies between NGF outputs and a pre-trained graspness network. Additionally, we present a neural graspness sampling method that decodes graspness values from the NGF to improve grasp pose detection results. Extensive experiments on the GraspNet- Billion benchmark demonstrate significant performance improvements compared previous works. Real-world experiments show that our method achieves superior trade-off between grasping performance and time costs. Code is available at wihhtps:: pattps://githbb. com/ Dahhaoxiagg622Actiiive
Introduction
Learning- based robotic grasp synthesis (24) has been explored to enable the manipulation of various objects across different grippers, sensors, and scenarios. The completeness and accuracy of the scene representation significantly influence the performance of these methods, as the geometric ambiguity can confuse the synthesis of grasp poses. To address this issue, previous works g 20 have employed multi-view information to reconstruct different 3D representations. However, as the robot needs to move when observing from different views, scanning the entire scene to achieve complete coverage incurs a substantial time cost and is challenging to apply in real-world environments.
To enhance the efficiency of multi-view perception for robotic grasping, previous works have intro- duced active perception methods to select the Next-Best View (NBV). Part of the methods [3 apply active 3D reconstruction techniques, treating grasp detection as secondary task. However, active reconstruction tends to select viewpoints that cover more unobserved space, which may not be optimal for grasp detection, as different regions have varying relevance to grasping. Consequently, active perception strategies based on 3D reconstruction have limited performance in grasp detection tasks. Recently, ACE-NBV 32 combines grasp detection and NBV planning by specially designing
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Marching Cubes
Neural Graspness Fields
Next-Best- Bes---eet--wwww Planner
View Mapping
ML.s
Neural Graspness Sumpling
Grasping Excution
No
Grasp Pose Detection
Stop Criteria
Yes
Satisfied?
Figure 1: Overview of the active grasp detection system. The RGB, depth and predicted graspness from new view are mapped to the Neural Graspness Field (NGF) by rendering loss. After each mapping step, the scene geometry is exported from the neural representation using the marching- cubes algorithm [19] and the candidate positions for grasp synthesis are sampled by neural graspness sampling. If the maximum perception step reached or a specific result condition is satisfied, e.g., sufficient number of high-quality grasps are detected, the robot arm is employed to execute the detected grasps. Otherwise, the Next- Bext-Best-WWe View (NBV) planner employed sample candidate views and select the view with the largest information gain for robot movement.
a grasp detection network to predict the grasp affordancc of candidate views. The viewpoint with the highest predicted grasp quality is then selected for reconstruction to efficiently obtain feasible grasps. However, this approach has two main limitations. First, it suppresses the perception of views that have low grasp quality but provide grasp-related geometric information, which can lead the method to fall into local optima. Second, directly predicting information gain through a specially designed grasp detection network can be limited by the network's 's generalization ability, potentially resulting in performance degradation when applied to novel environments.
To address the aforementioned issues, inspired by the development of the neural representation and radiance field 22(14), we propose to employ a Neural Graspness Field (NGF) to model the scene grasp distribution and build the active grasp detection system based on it. The graspness of a position measures the sum of feasible grasps in its pose space. The NGF is optimized online during the camera movement by back propagpppggg the rendering loss of the view graspness. The poline-training scheme employed for NGF modeling makes it easier to transfer to unseen scenarios. Moreover, distilling multi-view information into 3D space using the neural representation enforces the multi-view consistency, making the NGF less susceptible to the depth noise and view sparsity compared to directly predicting the grasp distribution on a reconstructed 3D representation [33]. With the neural representation of the scene grasp distribution, the active perception problem can be defined as minimizing the error of the scene grasp distribution modeled by NGF. This achieved by strategically selecting views that can bring the largest information gain for the NGF after mapping, thereby incrementally refining the modeled grasp distribution towards an optimal state.
In this paper, we propose an active perception method for the robotic grasp detection consisting of two components: neural graspness field mapping and graspness inconsistency-guie -ugided next-best- view plamning. For neural graspness field mapping, we extend a WeRF-based real-time mapping system [2] to render view graspness by adding separate network branch. The view graspness is predicted by pre-trained graspness network from the corresponding depth image. For NBV planning. we provide a graspness eeonsistencyyguiiee strategy targeting minimizing the inconsistency between the current NGF and the ground-truth scene grasp distribution. For a given view, its information gain is described as the inconsistency between the view graspness rendered from the NGF and the pseudo label predicted from the rendered depth image. Furthermore, we propose an inference strategy based on our active grasp detection framework, which decodes the graspness score from the NGF to generate grasp samples instead of predicting from explicit 3D geometries. The contributions of this paper can be summarized as follows:
We propose an active grasp detection framework via neural graspness field to model the scene grasp distribution online during camera movement.

---

We adopt graspness inconsisteney- guided strategy for next-best-vien planning, which targets on reducing the uncertainty of the neural graspness field.
A neural graspness sampling inference strategy is proposed to enhance the performance of the grasp detection framework.
Related Work
2.1 Grasp Detection
Grasp detection aims to generate feasible and diverse gripper poses for given objects. Early methods mainly studied the theoretical framework for robotic grasping by analyzing the contacts between the gripper and object models. Recently, to achieve grasping of unseen objects, data-driven grasp detection methods have been extensively studied. Some works g 34 investigate grasp detection in planar space, simplifying the problem by directly locating rotated grasp rectangles on images. To generate more diverse grasps that support complex downstream manipulation tasks, 6-DoF grasp detection has been proposed predict grasping in SF((3 space. (30] samples gripper poses on point clouds using geometric priors and scores these samples using a Convolutional Neural Network
(CNN). 18] follows the sample-and-score framework but introduces PointNet (27] to achieve better scoring performance. Some recent works adopt end-to-end networks for 6-DoF grasp detection to achieve real-time inference. (23 employs variational autoencoder for grasp generation given the object point clouds, and [28] proposes a single-shot grasp proposal network to enhance the efficiency of grasp detection. [T0] densely annotates grasp labels in clutter construct a large benchmark and provides an end-to and-to-end& 6-DoF grasp detection baseline trained with the large amount of annotations. Following this, [31] defines graspness to represent the grasp distribution in a scene and proposes graspness discovery method. Although the aforementioned methods have achieved good results, using single-view point clouds as input leads to a lack of geometric information due to occlusions and limited field of view, which affects the grasping performance on some objects. To solve this problem, 4 20 utilize Truncated Signed Distance Funetions (TSDFs) to map multi-view frames before generating grasps, and [8] employs generalizable neural radiance field to reconstruct the scene from cameras set around the scene, which supports the grasp detection of transparent and specular objects. However, these methods rely on scanning the scene from all possible views or several views surrounding the scene, resulting in excessive robot execution time for moving the camera. The large time cost reduces the usability of grasp detection methods in real-world scenarios.
2.2 Active perception for Grasp Detection
Active perception 3 aims to develop an agent that knows why it wishes to sense, and then chooses what to perceive, and determines how, when and where achieve that perception. Planning the next-best-view is commonly used method to realize an active perception system. In terms of grasp detection, active perception introduced to determine the camera views that can achieve a trade-off between the grasp performance and time cost. There are two main lines of work in this area: those that treat grasp detection as a secondary task of 3D reconstruction and those that directly incorporate grasp detection into the view planning process. Some works [13] 2 7 plan the view sequence based on the 3D reconstruction metric of the grasp-relevant region, where the grasp detection is treated as a secondary task. [13 models the uncertainty of occluded voxels as a mixture of Gaussians and utilizes trajectory optimization to generate the view sequence. (2 provides an active vision approach maximize surface reconstruction quality near the contact point region, and (7) adopts reinforcement learning for view planning with an object mask-guided reward function. To achieve close-loop control, [5] incorporates grasp detection network continuously predict grasps after each view mapping. However, employing grasp detection as a secondary task of 3D reconstruction overlooks the internal relation between grasp synthesis and scene reconstruction, which can lead to sub-optimal results. Another line of works [11 22 [2] directly incorporates grasp detection into the view planning process. [I]] maps grasp detection performance as a function of viewpoint for each object but struggles with novel objects. (22) uses the entropy of the network prediction to determine the text-best-view
a top-down grasping setting and recently, 32 proposes an affordance-driven policy based on an implicit grasp detection network to generate grasp affordance for unseen views. However, these approaches that utilize the output of grasp detection networks for view planning rely on the specific design of the network and are casily influenced by the generalization ability of the
3

---

grasp detection network, making it challenging to apply these methods to diverse real-world scenarios. In contrast to these approaches, our method utilizes view grasp distribution, which independent of specific network designs, and constructs the NGF online. This enables our method to easily adapt to diverse real-world scenarios test time.
Method
3.1
System Overview
The objective of grasp detection in cluttered scenes is to generate diverse feasible grasps for each object in the scene. Given rohotic with mounted depth camera, active grasp detection aims find a camera movement policy that can achieve high-quality results within a maximum of I' time steps. An overview of our active grasp detection system is provided Fig.
Candidate View
SDF based Volume Rembering
Graspness Netwwork
SOK basoo Volitee Kondeng
Color Feature Plane
MLPs
(x(2)
TSDF Feature Plane
(
g
Depth d
TSDF Featur: Pimne
Graspuess Inconsisteney
Pi
Grsspees Netwoik
MLPs
Graspmess Feature Plane
ME.Ps
Gropress f
Graqpees
(a) Neural graspness Field Mapping
(b) Graspness auconiitencyyguiide NBV Planning
Figure 2: The pipeline of the proposed mapping and NBV planning methods.
3.2 Neural Graspness Field Mapping
Inspired by neural feature fields [5] and semantic NeRF we incorporate a graspness field to represent the scene grasp distribution. The NGF employs a separate branch besides the appearance and geometry render grasp distribution information from multiple views. The graspness score proposed in [31 measures the graspable landscape in cluttered scenes given position With L grasp candidates Gp { lk = 1,... L}} sampled its configuration space, ie., approach direction, gripper depth and in-plane rotation, the ground-truth graspness score Fp is defined as:
Eke 1(gh > 1) ((x))
gp
|Gpl
where is the grasp quality score computed from force closure analysis, c indicates the collision state the gripper clutter and 1 is the indicator function. For an observed view, the NGF aims to map the view appearance, depth and graspness, shown in Fig. 2 (a). The NGF is composed of two parts: axis-aligned feature planes and SDF-based volume rendering. The axis-aligned feature planes store learned features different resolutions, which are queried and interpolated based on the 3D positions of the sampled points. For points p sampled in a ray, the corresponding features are queried from the feature planes and decoded by MLPs to raw color (p),), raw Truncated Signed Distance Field (TSDF) (p) and raw graspness d, (p). To convert raw TSDF values to volume densities, the SDF-based volume rendering from StyleSDF [25]:
o (p) = 3 Sigmoid( - (p)))
where 3 is learnable parameter. With the volume density, the color, depth and graspness of each ray r' can be computed as:
N 8(r) = Σ
wn (p.) and d(r)
on and g(r) = Σ wn wn (pn)
(3)
n=1

---

where 2n is the ray depth of pn and weight wn is formulated as:
Wn = earp( Σ o1 (xpp-()p)(((())))))
(4)
For given view, it is challenging directly obtain the corresponding ground-truth graspness during online mapping. Therefore, we use a pre-trained graspness network that takes the depth image as input and predicts the view graspness graspness field can reduce the graspness noise caused by single-view depth and render the graspness distribution from any given view, which be used to measure the grasp-correlated information gain.
render the NGF. By rendering from different views, the
3.3 Graspness Inconsistency-guides Next-Best- View Planning
Given sequence of viewpoints = (V1, 02 ... UN) and sequence set S, the NBV planning problem can be defined as:
N
argmar .ES
I(U.)
(5)
where N is the maximum step of robot movement and (vx)) represents the information gain of viewpoint Up- The selection of an informative view is influenced by the scene representation and the definition of information gain. We consider the NGF obtained from observing the entire scene the ground-truth scene grasp distribution and define the information gain of a view as the improvement the NGF prediction by mapping this view. Given an unseen candidate view, the NGF can estimate the view graspness based on the existing observations, and the improvement is expressed as the inconsistency between the ground-truth graspness gd)) predicted from the real depth and the graspness 9 rendered by the current graspness field.
However, for the NBV problem, obtaining the real depth image of candidate view is not possible. To address this, our inconsistency-guin
NBV policy adopts the pseudo-label paradigm by substituting the ground-truth grasp distribution g(d) with the pseudo-graspness g(d). which is widely employed in other semi-supervised and active learning vision tasks [17 29] As shown in Fig. 2 (b), we leverage the NGF's ability to render a pseudo-depth image d of the candidate view by volume rendering the TSDF values and then pass this pseudo-depth image through the graspness prediction network to
obtain the beeudo-graspnes: g(d). The pseudo-graspness is used to calculate the information gain of the candidate view, which is defined as:
((v)
Σ E((r) (()r)))()) r(م)
rEv
where g(r) represents the rendered graspness of sampled ray and the summation symbol represents the rendered graspness of the whole view.
should be noted that the effectiveness of our NBV policy relies on the premise that the g(d) predicted by the graspness network is closer to the ground-truth g(d) compared to g. The smaller error in g(d) can be attributed to reasons: First, the robot-mounted camera moves continuously in small steps, resulting in minimal differences in the observed geometric information between views, leading to insignificant errors in the rendered depth d. Second, the graspness prediction network, trained on large dataset of real point clouds, inherently provides robustness to depth noise. We visualize the rendered graspness error Eg = (g ÿl and the -aauddd-ssspeesss error E Eoch - (d( s(d) in Fig. 3 (a), where Eg significantly larger than E sا and the difference decreases with more steps.
The proposed
information gain can incorporate the view grasp distribution prior into the planning process, which is encoded by the pre-trained graspness network. Thus the NBV system can select the view containing the most graspness information that has not been distilled from the pre-trained network to the NGF. We visualize the ground-truth graspness g(d), graado-graspness g(d),, rendered graspness ý and the information gain in different views of the neural graspness field in Fig- 3 (b). For different views, the pseudo-graspness predicted from the rendered depth image can approximately represent the ground-truth g(d) but the accuracy of the rendered graspness ğ varies, which introduces different information gains.
5

---

View
a(d)
lafd
616
0,1003
0:1069
610-
0.0316
(a)
(b)
Figure 3: (a) The pseudo-graspness error E s(d) and rendered graspness error Eg of initial steps. (b) Visualization of the pseudo -ranoo-sssspess,, rendered graspness and the corresponding information gain of different views.
3.4 Neural Graspness Sampling
For the grasp detection method, predicting the grasp distribution in clutter and sampling positions for grasp synthesis is important part. Previous methods usually employ different encoders to predict grasp distribution from explicit 3D representations but the incomplete and noisy geometry information can lead to inaccurate grasp distribution. To achieve more precise grasp sampling, in addition to using the NGF for active perception, we propose an inference strategy based on sampling from the NGF. Given the positions p sampled from the reconstructed surface, the graspness of the position can be decoded from the NGF. During inference, we replace the graspness sampled from the neural representation with the graspness predicted by the grasp detection network and utilize Furthest Point Sampling (FPS) on the positions larger than a threshold T to get positions for grasp synthesis, which is formulated as:
Samples FF >= 73)
where by is the graspness branch of the NGE
Experiments
4.1 Experimental Setup
Simulation Setup We construct a simulation active grasp benchmark based on the G Billion benchmark [IO], which consists of 100 scenes for training and 90 scenes for testing. The set is divided into seen, similar, and novel sets based on the included objects. Each scene is captured from 256 views using Intel RealSense and Kinect cameras. We conduct all the experiments with the data captured by the Realsense camera. We set the pre-collected 256 views as the perception space for NBV planning. Since moving the camera distances would waste the information captured during the movement. Therefore, we sample the candidate views from the current view with relatively small step size. In our experiments, we set the step size to 10cm and set maximum step to 10. For evaluation, we follow the metric used in the GraspNet- Billion benchmark, which simulate grasps with friction μ ranging from 0.2 to 1.2 with interval 811 0.2. Following (20). we sample 5 grasps for each object in the scene to calculate the average precision AP. The training and evaluation of the simulation experiments are conducted on a single NVIDIA V100 GPU.
continuous process, moving it over long
Baselines We compared the following baseline methods validate the effectiveness of our proposed method. The baselines can be divided into two categories: NBV for robotic grasping 5 32 and NBV planning based on NeRF 226 16). Close-loop NBV [5] utilizes ray casting calculate the number of unobserved voxels of objects, which drives exploration targeting occluded object parts. ACE-NBV 32 incorporates grasp affordance prediction into NBV planning and selects the view with the largest grasp affordance as the next-best-view: ActiveNeRF (2 proposes an plug-in uncertainty estimation method for NeRF based on Bayesian estimation and Uncertainty-polt entropy of the weight distribution of each ray as the uncertainty.
16 computes the

---

Implementation Details For the mapping of NGF, the first view is trained for 100 iterations and following views are trained for 50 iterations. For each ray, 32 points are sampled for stratified sampling and 8 points for importance sampling. Only the coarse planes in [12 are employed for mapping. For NBV planning. we downsample the original image to 1 /8 to sample rays for graspness rendering to speed up the computation of view information gain. We utilize the first-stage network of GSNet [3] which predicts the graspness score for point-clouds as the graspness network in this paper. For the grasp detection network used for inference, we adopt the baseline method from [20] which uses the reconstructed scene geometry as input. For each scene, 1024 points are sampled from the NGF for grasp pose synthesis.
4.2 Simulation Experiments
Sean feseets
Sam Results
0.0
Nevel Results
6.5
032
0.00
642
4.0
8:00
041
Dur
0:4
Dets
6:31
000
(a) Seen
(b) Similar
(c) Novel
Figure 4: Comparison on different NBV policies based on the proposed NGF.
Comparison on different NBV policies To validate the effectiveness of the proposed NGF and the graspness inconsistency-guien NBV policy based on it, we re-implement other view planning policies on the same ESLAM mapping framework [12 which the NGF is built on. As shown in Fig. 4 our gseudo-graspness guided policy achieves superior performance after the first several views on seen, similar, and novel sets. Compared to ActiveNeRF (26) and incertainty-polic
[16], our method selects views with more grasp distribution discrepancy instead of geometry appearance ambiguity in the neural representation, which improves the results. Compared to policies targeting grasp detection [5] 32), our method is specially designed to reduce the uncertainty of the NGF by distilling the prior knowledge of pre-trained network, thus achieving superior performance with more views. It should be noted that ACE-NBV B22 can achieve comparable results to ours in the initial steps while showing little improvement as more views are added. This is because the affordance- basce policy only selects views with more feasible grasps but does not consider optimizing the scene grasp representation throughout the entire planning process.
Sean Fasuks
Sim Raauta
Hovil Rasalt
63c
031
0.34
032
0.4
0.0
0.4
0:00
040
0.4
(a) Seen
(b) Similar
(c)
Novel
Figure 5: The comparison of grasp detection result generated with the graspness predicted from 3D) geometry and sampled from NGF.
Effectiveness of Neural Graspness Sampling We apply the neural graspness sampling during the
inference of the grasp detection network to validate its effectiveness. The results are shown in Fig. 5) The sampling strategy improves the grasp detection results on seen, similar and novel objects at each step. Constructing a NGF through online multi-viey rendering, compared to directly predicting
7

---

the grasp distribution from 3D geometric information using the network, can reduce the errors in the scene grasp distribution caused by incomplete geometric information. Furthermore, since the optimization is performed online for each scene, it demonstrates better robustness compared to direct network prediction and thus the results on the novel set improve significantly.
Seen
Similar
Novel
Methods
AP
APo.s
APp.4
AP
APo.s
APo.4
AP
APo.s
APo.4
Close-loop
43.84
53.95
34.18
42.17
51.51
34.02
19.54
23.96
9.49
ACE-NBV
46.74
56.17
38.13
46.14
55.42
38.86
21.76
26.89
12.16
Ours
55.12
65.07
48.88
52.85
62.63
46.49
24.74
30.21
12.00
All views
63.75
73.30
58.38
61.54
71.17
55.94
24.89
30.18
13.95
Table I: Overall results compared to the state-of-the- afrtr--rth--tttte active grasp detection methods.
Overall Performance We compare the overall performance after 10 views of our method with previous active grasp detection methods 5 on the GraspNet- Billion benchmark, as shown in Table We employ the same grasp detection network trained on the GraspNet- Billion benchmark for all these methods. Our active grasp detection method improves the performance by 8.38%, 6.71%, 2.98% the seen, similar and novel sets compared to ACE-NBV [32) for grasp detection in clutter, demonstrating the effectiveness of the proposed method. All views represents a complete reconstruction using all 256 views, serving as upper-bound reference for active perception methods.
4.3 Real-world Experiments
Model
Success Rate (%)
Close-loop S
70.67 (53/75)
ACE-NBV
32
62.67 (47/75)
Ours
74.67 (56/75)
Figure 7: The robot setup of real-world experiments and the objects used for grasping.
Table 2: Results of the real-world grasping experi- ments.
We conduct real-world experiments the proposed active grasp detection method on 6-DoF UR- 10 robot arm with mounted RealSense D435i depth camera. The robot setup and objects used for experiments are shown in Fig. We seleet 25 objects from the YCB datasct [6] with various sizes and shapes for grasp detection. In the experiments, each cluttered scene is composed of 5 objects and we place these objects in different poses to evaluate each scene for 3 times. We employ the grasp success rate the metric. As shown in Table 2 our method achieves 12.00% and 4.00% improvement on success rate compared to ACE-NBV 32 and Close-loop NBV [5), respectively.
Overall 3.44s
NBV Planning
Mapping
Grasp Detection
Robot Execution
1.00s (29.07%)
0.45s (13.08%)
0.23s (6.69%
1.76s (51.16%)
Table Runtime analysis of the proposed method.
Runtime Analysis We provide a runtime analysis of the proposed active perception system, as shown in Table 3 The analysis is performed on a workstation with single NVIDIA 3090 GPU and an AMD Ryzen 5 2600 six-core processor. The average execution time for each step is 3.44 seconds, with the robot execution accounting for approximately 50% of the total time. In the active grasp detection system, the majority of the time consumed on NBV planning, while updating the NGF (mapping) and grasp detection take a relatively small proportion of the time. By investing some time in NBV planning, we achieve a trade-off between the performance and time cost compared to scanning the entire scene.

---

Scene
2-Views
5-Views
10-Views
Figure 8: Visualization the geometry and graspness extracted from NGF different planning steps.
4.4 Visualization of the Neural Graspness Field
Fig. g visualizes the NGF with different perception views, where the yellow region represents higher grasp probability. It can be observed that the NGF can not only reconstruct the 3D geometry of the scene but also jointly model the graspness. With approximately 5 views planned using active grasp detection, the NGF can effectively model the grasp distribution of objects. As more steps are taken, the details of the geometry and grasp distribution of the scene can be incrementally refined.
4.5 Visualization of the Planned Camera Trajectories
Close-loop NBV
ACE-NBV
Ours
Figure 9: Visualization of the camera trajectories generated from different active grasp detection methods.
Fig. iillustrates the view trajectories obtained by different view planning strategies. The close-loop NBV (5) approach, which employs unobserved space as the metric, guides the camera view path to scan regions with minimal overlap with the currently observed areas, aiming to maximize scene coverage. However, this method does not prioritize the graspable regions the objects. contrast, ACE-NBV 32) incorporates grasp detection network to guide view planning by selecting views with the highest grasp affordance. Nevertheless, this approach tends to repeatedly sean limited region, potentially leading to sub-optimal local results. Compared to these methods, proposed

---

approach efficiently scans the grasp- corse-correlated regions of the scene while ensuring comprehensive scene reconstruction.
Limitations
Our approach has two limitations. First, the time cost of NBV planning is positively correlated with the number of candidate views. When more refined view sampling is required, the planning time increases. Since our information gain computation is differentiable, this issue may be alleviated by sparse view sampling combined with pose optimization for the selected views. Second, our method cannot handle dynamic scene changes. Although efficient for static scenes, when grasping fails, e.g. objects fall out of the gripper or change pose without being grasped, the robot must re-execute the perception process. Incorporating techniques used in dynamic radiance fields could potentially address this problem.
6 Conclusion
In this paper, we propose an active perception method for grasp detection by introducing the neural graspness field, which models the grasp distribution of scene. By rendering the graspness predicted from a pre-trained network for each view, the NGF can be optimized online and reduce the noise of graspness in each view. Based which, introduce graspness inconsistency- -orddddguddddddug NBV policy to select the view with the largest inconsistency between the rendered graspness and pseudo-graspness graspness introduce neural graspness sampling decode the grasp distribution from the
label. Furthermore, neural representation, which benefits the position sampling of grasp pose synthesis. The experiments conducted the simulation and real-world settings demonstrate the effectiveness of the proposed active grasp detection method.
Acknowledgment
This work is partly supported by the National Natural Science Foundation of China (62022011). the Research Program of State Key Laboratory of Software Development Environment (SKLSDE- 2023ZX- 14), and the Fundamental Research Funds for the Central Universities.
References
[1] Stefan Ainetter and Friedrich Fraundorfer. End-to-end trainable deep neural network for robotic grasp detection and semantic segmentation from RGB. In IEEE International Conference on Robotics and Automation, 2021.
[2] Ermano Arruda, Jeremy L.. Wyatt, and Marek Sewer Kopicki. Active vision for dexterous grasping of novel objects. In JEEE/RSJ International Conference on Intelligent Robots and Systems, 2016.
[3] Ruzena Bajesy, Yiannis Aloimonos, and John K. Tsotsos. Revisiting active perception. Au- tonomous Robots, 42 2018.
[4] Michel Breyer, Jen Jen Chung, Lionel Ott, Roland Siegwart, and Juan I. Nieto. Volumetric grasping network: Real-time 6 DOF grasp detection in clutter. In Conference on Robot Learning, 2020.
[5] Michel Breyer, Lionel Ott, Roland Siegwart, and Jen Jen Chung. Closed-loop next-best-view planning for target-driven grasping. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2022.
[6] Berk Çalli, Arjun Singh, James Bruce, Aaron Walsman, Kurt Konolige, Siddhartha S. Srinivasa, Pieter Abbeel, and Aaron M. Dollar. Yale-cmu-berkee berkelyy dataset for robotic manipulation research. International Journal of Robotics Research, 36(3):261-22) 2017.
[7] Xiangyu Chen, Zelin Ye, Jiankai Sun, Yuda Fan, Fang Hu, Chenxi Wang, and Cewu Lu. Transferable active grasping and real embodied dataset. IEEE International Conference on Robotics and Automation, 2020.
10

---

[8] Qiyu Dai, Yan Zhu, Yiran Geng, Ciyu Ruan, Jiazhao Zhang, and He Wang. Graspnerf: Multiview-basee based 6-dof grasp detection for transparent and specular objects using generalizable nerf. In IEEE International Conference on Robolics and Automation, 2023.
[9] Amaury Depierre, Emmanuel Dellandréa, and Liming Chen. Scoring graspability based on grasp regression for better grasp prediction. In IEEE International Conference on Robotics and Automation, 2021.
[10] Haoshu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet- lbillii:: A large-scale benchmark for general object grasping. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[11] Marcus Gualtieri and Robert Platt Viewpoint selection for grasp detection. In JEFE/RSJ International Conference on Intelligent Robots and Systems, 2017.
[12] Mohammad Mahdi Johari, Camilla Carta, and François Fleuret. ESLAM: efficient dense SLAM system based on hybrid representation of signed distance fields. In /EEE/CVE Conference on Computer Vision and Puttern Recognition, 2023.
[13] Gregory Kahn, Peter Sujan, Sachin Patil, Shaunak D. Bopardikar, Julian Ryde, Kenneth Y. Goldberg, and Pieter Abbeel. Active exploration using trajectory optimization for robotic grasping in the presence of occlusions. In IEEE International Conference on Robotics and Automation, 2015.
[14] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):139:1 139:14 14, 2023.
[15] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems, 2022.
[16] Soomin Lee, Le Chen, Jiahao Wang, Alexander Liniger, Suryansh Kumar, and Fisher Yu. Uncertainty guided policy for active robotic 3d reconstruction using neural radiance fields. IEEE Robotics Automation Letters, 1000:2270-- 2022.
[17] Hengduo Li, Zuxuan Wu, Abhinav Shrivastava, and Larry S. Davis. Rethinking pseudo labels for semi-supervised object detection. AAAI Conference on Artificial Intelligence, 2022.
[18] Hongzhuo Liang. Xiaojian Ma, Shuang Li, Michael Görner, Song Tang, Bin Fang, Fuchun Sun, and Jianwei Zhang. Pointnetgpd: Detecting grasp configurations from point sets. In IEEE International Conference Robotics and Automation, 2019.
[19] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface con- struction algorithm. In Annual Conference on Computer Graphics and Interactive Techniques, 1987.
[20] Haoxiang Ma, Modi Shi, Boyang Gao, and Di Huang. Generalizing 6-dof grasp detection via domain prior knowledge. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[21] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision, 2020.
[22] Douglas Morrison, Peter Corke, and Jürgen Leitner. Multi-view picking: Next-best-view reaching for improved grasping in clutter. In IEEE International Conference on Robotics and Automation, 2019.
[23] Arsalan Mousavian, Clemens Eppner, and Dieter Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In JEEE/CVE International Conference on Computer Vision, 2019.

---

[24] Rhys Newbury, Morris Gu, Lachlan Chumbley, Arsalan Mousavian, Clemens Eppner, Jürgen Leitner, Jeannette Bohg. Antonio Morales, Tamim Asfour, Danica Kragic, Dieter Fox, and Akansel Cosgun. Deep learning approaches to grasp synthesis: A review. IEEE Transactions on Robotics, 39(5):39944405, 2023.
[25] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. Stylesdf: High-resolution 3d-consistent image and geometry generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[26] Xuran Pan, Zihang Lai, Shiji Song, and Gao Huang. Activenerf: uncertainty estimation. In European Conference on Computer Vision, 2022.
Learning where to see with
[27] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[28] Yuzhe Qin, Rui Chen, Hao Zhu, Meng Song, Jing Xu, and Hao Su. S4G: amodal single-view single-shot SE(3) grasp detection in cluttered scenes. In Conference Robot Learning, 2019.
[29] Liangchen Song, Yonghao Xu, Lefei Zhang, Bo Du, Qian Zhang, and Xinggang Wang. Learning from synthetic images via active pseudo-labeling. IEEE Transactions on Image Processing, 29:6452 6465, 2020.
[30] Andreas ten Pas, Marcus Gualtieri, Kate Saenko, and Robert Platt Jr. Grasp pose detection in point clouds. International Journal of Robolics Research, 36( 36(13-14):1555-4473 2017.
[31] Chenxi Wang, Haoshu Fang, Minghao Gou, Hongjie Fang, Jin Gao, and Cewu Lu. Grasp- ness discovery in clutters for fast and accurate grasp detection. IEEE/CVP International Conference Computer Vision, 2021.
[32] Xuechao Zhang, Dong Wang, Sun Han, Weichuang Li, Bin Zhao, Zhigang Wang, Xiaoming Duan, Chongrong Fang, Xuelong Li, and Jiamping He. Affordance-driven next-best-view planning for robotic grasping. In Conference Robot Learning, 2023.
[33] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. IEEE/CVF International Conference on Computer Vision, 2021.
[34] Xinwen Zhou, Xuguang Lan, Hanbo Zhang, Zhiqiang Tian, Yang Zhang, and Nanning Zheng. Fully convolutional grasp detection network with oriented anchor box. In IEEE International Conference on Intelligent Robots and Systems, 2018
12

---

A
Appendix
Seen
Similar
Novel
Methods
AP
APo.s
A...4
AP
A.o.s
APo.4
AP
APo.s
APo.4
Close-loop
S
40.07
48.06
32.05
34.74
42.32
27.78
8.68
10.61
2.93
ACE-NBV
32
44.74
54.23
36.32
37.72
45.69
31.65
13.56
16.51
7.46
Ours
52.35
61.64
45.86
44.50
51.76
39.82
13.94
18.02
5.97
views
61.35
70.45
55.76
55.12
62.07
49.85
19.54
23.75
9.89
Table Kinect results compared to the state-of-the-an active grasp detection methods.
Overall performance the kinect camera We compare the overall performance of kinect camera on the PisspNet--Billion benchmark, as shown in Table 4 Our active grasp detection method improves the performance by 7.61% 6.78%, 0.38% the seen, similar and novel sets compared to ACE-NBV (32 for grasp detection in clutter.
scenel
scotie2
Scenoó
scened
Figure 10: Object setting of the real-world experiment.
Scenel
Scene2
Scene3
Scene4
Scene5
Methods
Success Rate
1
3
1
3
1
2
3
2
3
2
3
Close-loop 5
4
3
3
3
4
4
3
4
3
3
4
3
3
53/75
ACE-NBV
32
4
3
4
3
3
4
2
2
4
3
47/75
Ours
5
4
4
3
4
3
4
4
3
56/75
Table 5: Detailed results for each scene in real-world experiments.
Details of the real-world experiment The scene setting for real-world experiment shown in Fig. 10 In total, constructed scenes, each containing 5 objects. For each scene, repeated the experiment 3 times by changing the poses of the objects within the scene. The number of the success attempts for each scene is provided in Table
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction in this paper reflect the paper's contribution accurately.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We discuss the limitations of our paper in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: Our paper doesn't 't contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide necessary information to reproduce the experimental results,
including hyper-parameters and the related papers.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We used a public dataset in the simulation experiments, and the related code will be open sourced after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http:///cc/giiee//eeeuuubiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the necessary training and test details of the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the standard deviation in the simulation experiments by inference 5 times each scene.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information about the computer resources for simulation and real-world experiments.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: This paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is purely academic work and has no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
18

---

Justification:
This paper doesn't 't release any data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual- should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The assets used this paper are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "RWi95Ftoc3", "reviewer": "Reviewer_DQft", "summary": "This work proposes an active perception method for grasp detection composed of two parts: neural graspness field mapping and next-best-view planning with a graspness inconsistency-guided strategy. And a corresponding inference strategy is also proposed by decoding the graspness score from NGF to generate grasp samples. The evaluation benchmark is constructed based on the GraspNet-1B benchmark and the experiment results indicate consistent improvements on the seen, similar and novel sets.", "questions": "1. Could you account for why with plan step increases, for the novel set, your method encounters a significant drop in performance?\n2. Could you provide further generalization and distraction experiments to verify your model's robustness?", "limitations": "1. The NGF-based method may be constrained to static environments and cannot adapt to dynamic scenarios.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is well motivated: aiming to settle both the negative effect of incomplete 3D geometry information for learning-based grasp detection methods and the time costs of scanning the whole scene, and find a trade-off.\n2. Experiments on GraspNet-1B demonstrate performance improvements compared to previous methods.", "weaknesses": "1. Relatively small performance improvement on the novel set compared to ACE-NBV.\n2. Generalization and distraction experiments are not provided to verify model's robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer DQft**\n\nThanks for your valuable feedback. We appreciate your acknowledgment of our active perception method based on neural graspness field and the performance improvements achieved. We address your concerns below:\n\n**Q1: The relative weak improvement on novel set.**\n\nIn our experiments, the improvement of the proposed method compared to other NBV strategies is less pronounced on the novel set than on the seen and similar sets. We attribute this to several factors: (1) The grasp detection model for inference demonstrates relatively lower performance on the novel set compared to the seen and similar sets. This inherent limitation constrains the potential improvements through active perception. (2) The graspness prediction network we employed has not been exposed to objects from the novel set during training. Consequently, the view graspness prediction used for training the NGF may lack accuracy for novel objects. This potential inaccuracy in graspness prediction may impact the performance of our active perception approach. \n\nOur method also experiences a performance drop between step 3 and 4 in Figure 4 (c) of the original paper. We attribute this primarily to the grasp detection model's suboptimal performance on the novel set, which leads to a scarcity of positive grasp samples and makes the final results unstable.  As the number of view gradually increases, our active perception method still achieves stable performance improvement on novel set.\n\n**Q2: Generalization and distraction experiments.**\n\nThanks for your suggestion. In future work, we will strive to supplement our study with ablation experiments on different modules and hyperparameters of our method to demonstrate its robustness and reliability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GH1bo8tjWG", "reviewer": "Reviewer_peVt", "summary": "The paper introduces a novel framework utilizing a Neural Graspness Field (NGF) in conjunction with a pre-trained graspness prediction network to enhance active grasp detection. It applies online training to the NGF upon encountering a new scene view, producing RGB, depth, and graspness maps. The method involves computing information gains from potential views by assessing discrepancies between the NGF-generated and pre-trained network graspness maps to select the most informative views. During grasp inference, the framework samples grasp poses guided by NGF-derived graspness scores and samples them using the Farthest Point Sampling (FPS) method.", "questions": "1. Despite the reported rapid mapping operation time of 0.45 seconds in Table 3, the paper notes extensive training iterations for both initial and subsequent views (100 and 50 iterations). Further details on achieving such fast training times would be beneficial for understanding the feasibility of the NGF training process.\n\n2. Equation (6) introduces confusion regarding the role of the summation symbol and the meaning of the variable $r$. Clarification of these components would aid in a better comprehension of the equation's intent and application.", "limitations": "See weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The approach of leveraging graspness inconsistency to define information gain offers a targeted advancement for the robotic grasping task, potentially surpassing traditional methods that focus on uncovering occluded regions, as referenced in [1].\n2. Comprehensive evaluations are conducted through both simulated and real-world experiments, with comparisons against multiple baselines. These experiments demonstrate the framework's effectiveness in practical scenarios.\n\n\n[1] Breyer, Michel, et al. \"Closed-loop next-best-view planning for target-driven grasping.\" 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.", "weaknesses": "1. A primary concern is the incremental training approach of the NGF. Since the NGF is trained incrementally, the rendered graspness maps \\hat{g} and depth maps might be inaccurate for novel views at the initial stage, and thus are inappropriate to be used to compute information gain. \n2. The NGF's grasp knowledge, being distilled from the pretrained graspness network, suggests that its maximum achievable performance may be inherently limited to that of the teacher network. How can you demonstrate that the proposed framework is better than the single-shot method? \n3. According to Eqn(1), the graspness score of a position is defined as the mean grasp quality scores of different orientations on that position. When sampling grasp poses at the inference stage, it seems that they only sample positions for the grippers. How to determine the orientations?\n4. Visual results (Figure 3) indicate that initial high information gains may be attributed more to background inconsistencies rather than the target object regions, potentially skewing the focus of planning efforts. The inconsistency of the background region should be neglected when planning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Response to Reviewer peVt**\n\nThanks for your valuable feedback. We understand your concerns regarding the computation of information gain with NGP and the other issues you have raised. We address your concerns below:\n\n**Q1: Incremental training approach of the NGF.**\n\nIn the initial stage of NGF, there are indeed errors in both the rendered depth $\\hat{d}$  and graspness $\\hat{g}$ but have few influence on the calculation of information gain.  Our inconsistency-guided NBV policy adopts the pseudo-label paradigm by substituting the ground-truth grasp distribution $g(d)$ with the pseudo-graspness $g(\\hat{d})$, which is widely employed in other semi-supervised and active learning vision tasks [1,2]. The effectiveness of our NBV policy relies on the premise that the $g(\\hat{d})$ predicted by the graspness network is closer to the ground-truth $g(d)$ compared to $\\hat{g}$. The smaller error in $g(\\hat{d})$ can be attributed to two factors: (1) The robot-mounted camera moves continuously in small steps, resulting in minimal differences in the observed geometric information between views, leading to insignificant errors in the rendered depth $\\hat{d}$. (2) The graspness prediction network, trained on a large dataset of real point clouds, inherently provides robustness to depth noise. As a supplementary, we visualize the rendered graspness error $E_\\hat{g} = |g(d)-\\hat{g}|$ and the pseudo-graspness error $E_{g(\\hat{d})} = |g(d)-g(\\hat{d})|$ in **Figure 1** of the rebuttal PDF, where $E_\\hat{g}$ is significantly larger than $E_{g(\\hat{d})}$ and the difference decreases with more steps. \n\n**Q2: The maximum achievable performance of our method.**\n\nWe are not certain the definition of \"single-shot\":\n\nIf \"single-shot\" refers to using a single-view depth map for grasp detection, our active perception method aims to reconstruct the scene geometry and grasping representation with as few steps as possible. The scene point-cloud reconstructed by active perception is used for final grasp poses prediction,, resulting in significantly better performance.\n\nIf \"single-shot\" refers to not using neural graspness sampling,  it's important to note that while the grasping knowledge of NGF originates from the teacher network's output, NGF enhances the multi-view consistency and smoothness of the graspness distribution through implicit multi-view fusion. This is particularly beneficial when the single-view output is noisy and views are sparse, as demonstrated in similar scenarios such as 3D segmentation [3]. Therefore, sampling grasping positions from NGF yields better results compared to directly sampling from the network's graspness prediction.\n\n**Q3: The inference pipeline with neural graspness sampling.**\n\nWe are sorry for the confusion caused. To clarify, we still rely on a grasp detection network to perform inference on the reconstructed scene point-cloud after active perception, rather than directly sampling grasp poses from NGF. For neural graspness sampling, we replace the graspness output of the grasp detection network with values obtained from NGF based on the position, while parameters such as rotation and gripper width are still inferred using the grasp detection network.\n\n**Q4:  Visualization in Figure 3 in original paper.**\n\nIn robotic grasping scenarios, it is difficult to divide the objects and background in 3D space from a single view perception. Therefore, given a candidate view, the graspness values of both foreground and background regions are used to compute the information gain. For our methods, not all selected views with high information gain arise from background errors. We provide more visualization results in the **Figure 2** of rebuttal PDF for illustration, where we use yellow rectangle box to highlight the forground inconsistency region.\n\n**Q5: The mapping time for our framework.**\n\nThe mapping method is an extension of an efficient dense SLAM method [4] (citation [12] in our paper) that exploits axis-aligned feature planes and implicit Truncated Signed Distance Fields to reduce the number of training iterations, whereas previous dense neural mapping methods require more iterations for convergence.\n\n**Q6: Explaination about Equation(6).**\n\nWe are sorry for the confusion caused. $g(r)$ means the rendered graspness of sampled ray $r$ and the summation symbol means the rendered graspness of the whole view. The definition follows Equation (3) in our paper and we will clarify it in the camera-ready version.\n\n**References**\n\n[1] Rethinking pseudo labels for semi-supervised object detection, Li et al., AAAI2022.\n\n[2] Learning From Synthetic Images via Active Pseudo-Labeling, Song et al., TIP2020\n\n[3] In-Place Scene Labelling and Understanding with Implicit Scene Representation, Zhi et al., ICCV2021.\n\n[4] ESLAM: efficient dense SLAM system based on hybrid representation of signed distance fields, Johari et al., CVPR2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QWOlW3Wqcu", "reviewer": "Reviewer_4AgP", "summary": "This paper studies active perception for robotic grasp detection. It proposes an active grasp detection framework based on the Neural Graspness Field (NGF), which incrementally models the scene and facilitates next-best-view planning. For next-best-view planning, it aims to reduce the uncertainty of the NGF through a graspness inconsistency-guided policy, selecting views based on discrepancies between NGF outputs and a pre-trained graspness network. Additionally, it presents a neural graspness sampling method that decodes graspness values from the NGF to improve grasp pose detection results.", "questions": "See Weaknesses.", "limitations": "The author mention the limitations of the paper and suggest possible solutions.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper considers how robotic arms can better select the next position to move towards the target object, which is rarely considered in current static two-finger grasping. I believe the authors' claim is important for the development of the robotics community. The authors have effectively combined NeRF with the significant concept of Graspness in recent years to study two-finger grasping, which I find to be an interesting topic.", "weaknesses": "Main concern：\n1. First of all, I regret that the authors do not submit video materials in the supplementary materials. Although the authors analyze the real-world performance in the paper, the lack of videos makes it difficult to further verify the authenticity and effectiveness of the real-world experiments\n2. The GraspNet-1Billion dataset includes two cameras, as mentioned in line L196 of the paper. However, in Table 1, the authors only provide experimental results for one camera, which I find insufficient. Testing on different cameras can more comprehensively demonstrate the effectiveness of the method.\n3. Although the authors conduct sufficient comparison experiments to demonstrate the effectiveness of their method, I think they lack analysis experiments on some hyperparameters. For example, the authors set the maximum step to 10. I would like to ask the authors what the basis for setting it to 10 is, and what would happen if I used 7 steps or even fewer?\n4. I think the recording of real-world experiments is insufficient. The authors should at least document the number of scenarios designed, the objects used in each scenario, and the average accuracy for each scenario. Simply recording an average success rate in Table 2 without detailing the experimental setup is, in my opinion, not rigorous. Additionally, there are no example images of cluttered scenes, making it difficult to assess the complexity of these scenarios. Typically, five objects do not constitute a cluttered scene; cluttered scenes usually consist of 10-15 or even more objects.\n5. In section 4.5, I cannot intuitively see the clear performance advantages or disadvantages between Close-loop NBV and \"Ours\". In my opinion, the differences are due to the distinct training methods of the two planners, resulting in varied decision-making. I suggest the authors provide more distinguishable trajectory visualizations (e.g., whether the trajectory planning of Close-loop NBV causes collisions with objects, thereby disrupting the scene).\n\nOthers：\n1. In equation (3), the formula for Pn​ is missing a closing parenthesis. Additionally, equation (3) does not clearly indicate the elements of ray tracing that are being focused on. For example, in NeRF, estimating density only requires the 3D location, while estimating color involves both the 3D location and the 2D viewing direction. The authors do not clearly explain in equation (3) which information c(r), d(r), and g(r) utilize.\n2. In Table 1, the term \"All views\" is unclear, and there is no corresponding explanation provided in lines L245-L250.\n3. Although the authors conduct an ablation study on GraspnessSample, I find the ablation experiments insufficient. As cited in the paper, the effectiveness of Graspness is already thoroughly validated in the GSNet paper. Conversely, the authors do not provide sufficient ablation studies on other parts of the planner. I believe these other parts are the more critical aspects of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Response to Reviewer 4AgP**\n\nThanks for your valuable feedback and we address your concerns below:\n\n**Q1: Video recording of real-world experiments.**\n\nThanks for your suggestion. In **Figure 3** of the rebuttal PDF, we provide keyframe screenshots of one execution, including the active perception part and robotic grasping part. The videos will also be attached then.\n\n**Q2: Experiments on data captured from Kinect camera.**\n\nThanks for your suggestion. We conduct additional experiments with the Kinect camera and make comparison with the previous SOTA method (i.e. ACE-NBV), as shown in **Figure 5** of the rebuttal PDF, demonstrating the effectiveness of our method under different cameras. Due to time and page limitation of the rebuttal, we only finish the experiments on the seen set and will supplement the results in the camer-ready version.\n\n**Q3: Analysis experiments on hyperparameters, such as the maximum NBV step.**\n\nIn fact, we have presented the results with a varying number of steps for both the NBV policy comparison (Figure 4 in the original paper) and the Neural Graspness Sampling experiments (Figure 5 in the original paper). The horizontal axis denotes the number of planning steps. As the complete planning policy greedily chooses the view yielding the highest information gain at each step, the results with 7 or fewer steps can be directly inferred from the graph. Regarding the other hyperparameters, such as the number of iterations for NGF, we emprically know that they do not substantially influence the performance of our method. Anyway, we will improve our experiments on hyperparameter sensitivity in the next version.\n\n**Q4: More detials about the real-world experiment setting.**\n\nIn this setting, we consider 5 scenes, each containing 5 objects, and for each scene, we repeat the experiment 3 times by changing the poses of the objects within the scene. We provide the detail recording of the real-world experiment in the table below.\n\n| Scene_id-Pose_id | 1-1  | 1-2  | 1-3  | 2-1  | 2-2  | 2-3  | 3-1  | 3-2  | 3-3  | 4-1  | 4-2  | 4-3  | 5-1  | 5-2  | 5-3  | Overall |\n| ---------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ------- |\n| Close-loop       | 4  | 4  | 3  | 3  | 5  | 3  | 4  | 4  | 3  | 4  | 3  | 3  | 4  | 3  | 3  | 53/75   |\n| ACE-NBV          | 4  | 2  | 3  | 4  | 3  | 3  | 3  | 3  | 4  | 3  | 2  | 2  | 4  | 4  | 3  | 47/75   |\n| Ours             | 5  | 3  | 4  | 4  | 3  | 4  | 3  | 3  | 4  | 4  | 3  | 4  | 5  | 4  | 3  | 56/75   |\n\nAlthough we use a relatively small number of objects (5) to construct each scene, the objects in the scene are arranged in a crowded manner, with the presence of view occlusions. Therefore, our scene setting reflects the characteristics of cluttered scenes. In the rebuttal PDF, we provide the object setting of the real-world experiment scenarios in **Figure 4**.\n\n**Q5: Differences between the trajectories generated by Close-loop NBV and ours .**\n\nClose-loop NBV is a heuristic active perception method based on TSDF Fusion that calculates the number of potentially unobserved voxels as information gain without involving a training process. However, this type of methods do not directly relate to grasp detection, as the view selection primarily depends on the overlap between the candidate views and the observed region. Therefore, the trajectory generated by the Close-loop NBV planner tends to uniformly surround the scene to maximize the observation area. More examples are shown in **Figure 6** of the rebuttal PDF. \n\n**Q6: Problem about Equation (3).**\n\nWe apologize for the missing closing parenthesis and any confusion caused by Equation (3). Regarding the calculation for ray color $\\hat{c}(r)$, depth $\\hat{d}(r)$, and graspness $\\hat{g}(r)$, the NGF follows the previous NeRF-SLAM mapping method [1], where only the 3D location is involved. However, unlike the vanilla NeRF, which directly decodes the color and depth from a single MLP, the mapping framework that we employ first queries the position-corresponding feature from the learnable axis-aligned feature planes and uses separate MLPs to decode the color, depth, and graspness. This approach can significantly enhance the convergence speed. Although not introducing view directions when rendering RGB may lead to a slight decrease in quality, we typically only focus on depth and graspness for grasp detection. Therefore, the mapping system based on the NeRF-SLAM framework is suitable for NGF.\n\n**Q7: Meaning of \"All views\" in Table 1.**\n\nIn Table 1 of the original paper, \"All views\" refers to using all the 256 views captured for a scene in the Graspnet-1Billion dataset to perform a complete reconstruction of the scene and then infer the grasping results. This result serves as an upper-bound reference for active perception methods. \n\n**Q8: Ablation study about Neural Graspness Sampling.**\n\nWe would like to clarify although the definition of graspness is proposed in GSNet, our method significantly differs in the way of graspness generation and achieves better results. In GSNet, a grasp detection network is used to infer the graspness value of each point from the input point cloud, which depends on the quality of the point cloud and does not effectively utilize multi-view information. Our proposed neural graspness sampling directly queries the graspness value by the position from the NGF, which fully leverages the NGF's capability to accurately model the scene's grasp distribution, leading to improved results. Therefore, it is essential to conduct an ablation study on neural graspness sampling to validate its contribution. For other parts of the NBV planner, we will strive to provide ablation studies in the next version.\n\n**References**\n\n[1] ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields, Johari et al., CVPR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SRTemfLuY8", "reviewer": "Reviewer_DbvE", "summary": "This paper proposes an next-best-view planning method for grasp perception. The authors use neural field to model the grasp distribution of a scene, which is learned from the graspness detection result of different views. Then the NBV is designed as the view with the largest inconsistency between the rendered graspness from the neural field and pseudo-graspness label predicted by the network on the rendered depth. Simulation and real-world experiements show that the proposed method is slightly bettern than exising methods.", "questions": "1. Sec 3.4 is not clear. Why do the authors need FPS? There is no ablation study on this design. Is the graspness score generated by the network or neural field?\n2. Colorbar. should be added in Fig3, 8\n\n3. How is the deviation between NGF and predicted graspness computed?", "limitations": "Yes, limitations are addressed in the paper: the time cost and the method cannot be used for dynamic scenes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of modelling the graspness of the scene as a neural field is novel to my knowledge. \nThe paper is well structured and the visualization is good for readers. The method and the experiment detials are well presented and most parts are easy to understand and reproduce.", "weaknesses": "First of all, the meaning of NBV for grasp percetpion is not very clear to me. The goal of grasp percetpion is to find ONE successful and executable grasp in the scene, rather than detecing all the feasible grasps. \n\nSecondly, the gain of the proposed method is not well validated by the experiments. In both simulation and real-world experiments, the proposed method is only slightly better than baselines. The margin is not significant.\n\nThirdly, the grasp network that is used to predict the graspness for neural field may produce inconsistent results across views. How do the authors address this problem during field optimization. The depth of the field for unseen views can also be errorneous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer DbvE**\n\nThanks for your valuable feedback. We understand your concerns about the goal of active grasp detection, the performance gain of our method and some other details. We address your concerns below:\n\n**Q1: Meaning of NBV for grasp perception.**\n\nWhile finding one feasible grasp pose allows a robot to pick up objects, grasp detection often serves as a part of semantically diverse downstream manipulation tasks [1, 2]. In such cases, it is crucial to provide diverse feasible grasp poses in the scene. For instance, grasping the cap area of a water bottle does not enable the pouring task. Similarly, when tidying up a table, the robot must select the grasping pose based on the object's placement pose. Therefore, it is reasonable for active perception to set the goal of grasp detection as finding all feasible grasps in the scene.\n\n**Q2: Performance gain of our method.**\n\nTo evaluate the performance for active perception, we consider the grasp detection results with different perception steps, where our method totally show significantly superior results compared to other active perception method given same steps. As shown in Figure 4 of our original paper, our method consistently outperforms other NBV strategies by at least 1 AP starting from the 6th step across the seen, similar, and novel sets. In the ablation study of neural graspness sampling illustrated in Figure 5 of our original paper, the incorporation of this sampling method demonstrates significant improvements across almost all planning steps. Furthermore, Table 1 of our original paper presents the results of different active grasp detection methods under a 10-step planning scenario. Our method shows notable improvements compared to the previous state-of-the-art ACE-NBV, with increases of 8.38, 6.71, and 2.98 AP on the seen, similar, and novel sets, respectively. In the context of the Graspnet-1billion benchmark, these improvements can be considered substantial.\n\n**Q3: Problem about the process of NGF optimization.**\n\nIn the optimization process of NGF, there indeed exists inconsistent graspness results for each observed view, which can be primarily controlled through two aspects: (1) During the optimization process, both the new and previously observed views are jointly involved. This approach fully utilizes multi-view information and promotes consistency in graspness across different views given the same spatial location. (2) According to previous research [3], neural representations inherently possess the capability to reconstruct relatively smooth distributions from sparse and noisy multi-view information. \n\nAs for the depth error in the unseen views, becuase the camera moves continuously in small distance for each step, it is relative small in most cases and won't influence the calculation of the information gain. Figure1 in the rebuttal PDF also illustrate the effectiveness of using the rendered depth for the calculation of grasp information gain.\n\n**Q4: Explaination about Section3.4.**\n\nWe apologize for any confusion. In Section 3.4, we introduce neural graspness sampling, which directly queries the graspness from NGF rather than predicting it from a grasp detection network. Following previous work [4], we incorporate FPS (Farthest Point Sampling) for two reasons: (1) To ensure that grasp positions are distributed throughout the entire scene. (2)To control the number of generated grasp poses, as there can be numerous positions with high graspness values. \n\nThe graspness scores are generated by the neural field, and we generate grasp poses at positions where the graspness score exceeds a threshold.\n\n**Q5: Missing of the colorbar.**\n\nThanks for your suggestion. In Figure 3 and Figure 8 of the original paper, blue means low graspness value while yellow represents high value. We will add it in the next version.\n\n**Q6: Deviation between NGF and predicted graspness.**\n\nDuring the training of NGF, the rendering process of graspness is similar to the color. We sample rays $r \\in R$ from the given view and use L2 loss to calculate the deviation, which can be formulated as:\n$$\nL_g = \\frac{1}{|R|}\\sum_{r\\in R}(\\hat{g}(r)-g(r))^2\n$$\nwhere $\\hat{g}(r)$ is the graspness rendered by NGF and $g(r)$ is the predicted graspness. During NBV planning, we employ Equation (6) in the original paper to calculate the deviation as the information gain.\n\n**References**\n\n[1] Learning task-oriented grasping for tool manipulation from simulated self-supervision, Fang et al., IJRR2020\n\n[2] Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping, Rashid et al., CoRL 2023\n\n[3] In-Place Scene Labelling and Understanding with Implicit Scene Representation, Zhi et al., ICCV2021\n\n[4] Graspness discovery in clutters for fast and accurate grasp detection, Wang et al., ICCV2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their insightful feedback on our submission. We are grateful for their acknowledgment of our paper's contribution to active perception in robotic grasping. To address the questions raised regarding our method design and experimental work, we have provided comprehensive explanations aimed at clarifying the reviewers' concerns. We have also included a rebuttal PDF containing the figures referenced in our response.

We summarize the key points of our response below:

1. **The calculation of information gain:** Reviewer peVt expressed concern about the accuracy of the information gain calculation, given potential errors in rendered depth and graspness. Reviewer DbvE also mentioned possible errors in rendered depth for unseen views. In our response, we have provided a detailed illustration addressing the influence of these potential errors to view planning and included a numerical analysis in the  rebuttal PDF.
2. **Neural graspness sampling**: Reviewer 4AgP and DbvE expressed confusion about how we use neural graspness sampling for grasp poses inference. Our original paper may be unclear, leading to misunderstanding. In our response, we have provided a detailed explanation to clarify this point.
3. **Real-world experiment setting:** Reviewer 4AgP provided many valuable questions and suggestions on our real-world experiments. We have supplemented our response with additional details about the real-world scene settings and comprehensive results. Furthermore, we have included keyframe screenshots from the video of our method being executed on a real robot in the rebuttal PDF.

Meta-review: The paper presents an active perception framework for robotic grasp detection, demonstrating improved performance on a large-scale benchmark and real-world validation. Reviewers appreciated the novelty but raised questions about calculating information gain, the clarity of neural grasp sampling, and the details of experimental setups. The authors responded to these concerns with additional explanations and data, satisfactorily addressing the reviewers' issues.
Considering the overall positive reviews, the authors' effective rebuttal, and the potential contributions to the field, the paper is recommended for acceptance.
Decision: Accept (poster)