Affordance-Driven Next-Best-View Planning for Robotic Grasping
Authors: Xuechao Zhang
Dong Wang
Sun Han
Weichuang Li
Bin Zhao
Zhigang Wang
Xiaoming Duan
Chongrong Fang
Xuelong Li
Jianping He
Venue: CoRL 2023 Poster (2023)
Keywords: Grasp Synthesis
Neural SDF
Next-Best-View Planning

Abstract: Grasping occluded objects in cluttered environments is an essential component in complex robotic manipulation tasks. In this paper, we introduce an AffordanCE-driven Next-Best-View planning policy (ACE-NBV) that tries to find a feasible grasp for target object via continuously observing scenes from new viewpoints. This policy is motivated by the observation that the grasp affordances of an occluded object can be better-measured under the view when the view-direction are the same as the grasp view. Specifically, our method leverages the paradigm of novel view imagery to predict the grasps affordances under previously unobserved view, and select next observation view based on the highest imagined grasp quality of the target object. The experimental results in simulation and on a real robot demonstrate the effectiveness of the proposed affordance-driven next-best-view planning policy. Project page: https://sszxc.net/ace-nbv/.

Affordance-Driven Next-Best-View Planning for Robotic Grasping
Xuechao Zhang Dong Wang2 Sun Han Weichuang Li², Bin Zhao?, Zhigang Wang', Xiaoming Duan', Chongrong Fang', Xuelong Li2, Jianping He
!Shanghai Jiao Tong University, SShanghai Artificial Intelligence Laboratory
Abstract: Grasping occluded objects in cluttered environments is an essential component in complex robotic manipulation tasks. In this paper, we introduce an AffordanCE-drrven Next-Best- View planning policy (ACE-NBV) that tries to find a feasible grasp for target object via continuously observing scenes from new viewpoints. This policy is motivated by the observation that the grasp af- fordances of an occluded object can be better-measured under the view when the view-direction are the same as the grasp view. Specifically, our method leverages the paradigm of novel view imagery predict the grasps affordances under pre- viously unobserved view, and select next observation view based on the highest imagined grasp quality of the target object. The experimental results in simula- tion and on a real robot demonstrate the effectiveness of the proposed affordance- driven next-best-vin best-view planning policy. Project page: https://sszxx https://szzxc.metacc-bbbl net/ace--bw..
Keywords: Grasp Synthesis, Neural SDF, Next-Best- View Planning
Introduction
When we aim to pick up an occluded object from an unstructured environment, observations from a single perspective often fail to provide sufficient affordances information, and we spontaneously move our heads obtain new perspectives of the occluded object. The driving force behind the actions that lead abilities. We know in which direction we can better interact with objects, and we will choose to observe in this direction the next time. However, current intelligent robotic systems are not able to perform these tasks efficiently, and a unified framework for addressing this challenge is lacking. In this work, we aim to investigate the feasibility of endowing robots with this capability.
seek the next observation relies
imagination and spatial reasoning
As shown in Fig. 1, we focus the task of grasping specific object in cluttered scenes by robotic arm with parallel-jaw gripper. There are relatively mature approaches for predicting grasp poses for unknown objects in cluttered scenes, and most of them first observe the entire scene from one or more fixed viewpoints [1, 2] and predict grasps of all objects at once. However, these methods may fail to predict a feasible grasp for a specific object due to heavy occlusions between objects. order to design more stable grasp affordances prediction pipeline, some previous works [3, 4] introduced active perception modules to observe the scene from several new selected viewpoints before execut- ing the grasp. They all select the new observation view directions based on the information gain of object geometry reconstruction. However, the improvement of geometry reconstruction does not always indicate a better grasp quality.
In this paper, as shown in Fig. 1, we build on the intuition that the grasp affordances can be better- measured using the observation when the observation view direction is the same as the grasp di- rection. Based on this insight, we leverage the novel view imagery ability of the implicit neural
Work done while the author was intern at Shanghai Artificial Intelligence Laboratory. B Corresponding authors: Dong Wang (wangdong @ pjl...org.e)) Jianping He sjt..c...)
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Imaginal View
Uccloded View
Figure 1:
The task the robotic arm is to grasp the red target object, but the view of its in-hand camera is hindered by a cluttered scene, making it difficult to provide high-quality and collision free grasping pose predictions. In this study, we draw insight that the grasp affordance of occluded target object can be well-measured when the view direction is the same as the grasp direction, and propose framework that plans the next observation based on the increment of grasp affordances find feasible grasp on the target object.
representation to predict the grasp affordances of imagined novel grasps, and set the next best ob- servation view to the imagined grasp view that yields the highest gain the grasp quality rather than object geometry reconstruction. Specifically, we first propose view-aware grasp affordances prediction module effectively exploit the target object geometry information and occlusion lationships between objects for better grasp synthesis. Then, we adopt similar training paradigm as NeRF [5] to enable our model to imagine the scene representation from previously unobserved viewpoints. With this scene imagery ability, we predict the grasp affordances of target object under many imagined views with proposed view-aware grasp affordances prediction module. Last, a next- best-view planning policy designed to continuously observe the scene from selected new views until a feasible grasp on target object found. In summary, the contributions of this work are as follows:
We propose a view-aware grasp affordances prediction module for better grasp synthesis on occluded target object in cluttered environments.
We design sett-best-view planning framework that leverages the implicit neural repre- sentation to jointly predict imagined grasp affordances under unseen views and select the next observation view based on the grasp affordances prediction.
We demonstrate sigmificant improvements of our model over the state-of-the-arr for the grasp task in cluttered scenes in simulation and real robot.
Related Works
2.1 Grasp Detection
Grasping objects is one of the fundamental abilities for robotic manipulators in manipulation tasks. In order to grasp diverse unknown objects in any environment, robotic system must effectively utilize the geometric information gathered from its sensors to calculate the feasible grasping poses. Recent advances deep learning methods have led to rapid developments in robot object grasp- ing [6. 7, 8, 9, 10, 11, 12]. significant portion of these methods do not require object localization and object pose estimation, but instead perform grasp affordances prediction using end-to-end ap- proaches [1. 13]. In particular, Dex-Net [13, 14] adopts two-step generate-and- senerate-and-evallutt approach for top-down antipodal grasping. and VGN [1] introduced a one-step approach for predicting 6- DoF grasping configurations in cluttered environments. GIGA [2] exploits the synergistic relationships between the grasp affordances prediction and 3D reconstruction scenes in cluttered environments for grasp detection. These works all take fixed single or multiple images as input [1, 2, 13, 15], and the robustness of these methods largely influenced by the observation camera viewpoints [16]. When dealing with complex environments with strong occlusions, many works [3, 4, 17] try to grasp objects by dynamically moving the observation sensors obtain additional scenes and object geometry information.

---

2.2 Next-Best- View Planning
Next-Best-Viee (NBV) planning, which aims to recursively plan the next observation position for sensors, is one of the most challenging problems in active vision for robotics [18]. Compared to the passive observation paradigm, active perception with next-best-view planning enables a more flexible way of obtaining environment information. has been applied in various fields, such as object reconstruction [19, 20, 21J, object recognition [22, 23], and grasp detection [3, 4, 17]. NBV planning is typically divided into two categories: synthesis methods and search methods. Synthesis methods directly calculate the next observation position based current observations and task constraints [24]. with some methods [25] working within the paradigm of reinforcement learning. On the other hand, search methods first generate a certain number of candidate viewpoints and then select viewpoints based on human-desipn designed criteria. Most search-based approaches use the gain of 3D geometry reconstruction as the metric to select next viewpoints [3, 4, 19]. In partieular, Arruda al. [3] propose next-best-view planning policy that maximizes object surface reconstruction quality between the object and given grasp. Breyer et al. [4] design closed-loop pext-best-view planner based volumetric reconstruction of the target object. Recent work based on neural radiance fields has also proposed some uncertainty-drivee methods [26, 27, 28, 29, 30]. However, for grasp tasks, evaluating viewpoints from grasp affordances is more direct approach [17]. In this article, we mainly explore how to use grasp affordances information for NBV planning.
3 Problem Formulation
We consider the same active grasp problem as in [4]: picking up occluded target object in cluttered scenes using a robotic arm with eye-in-hand depth camera. As shown in Fig. 2, the target object is partly visible within the ini- tial camera view field and a 3D bounding box
Update Perception
Compute Grasp
given to locate the target object. Our goal is
Stepping stisfed!
design policy that moves the robotic arm to find a feasible grasp for the target object.
Execute Grusp
An overview of the whole system is shown in Fig. 2. Given a cluttered scene on a tabletop and an occluded target object with corresponding bounding box T hbox: we aim to predict feasi- ble 6-DoF grasp G for the target object T. Specifically, at each time t, we obtain the observation D, and integrate it into Truncated Signed Distance Function (TSDF) M, Then predict sev- eral possible grasps G1, G.,..., GN for the target object based on current M,, Next, we use stopping criterion to determine whether feasible grasp on the target object has been found. If the stopping criterion is satisfied, we select the grasp of G with the highest predicted quality to exe- cute. Otherwise, our proposed model computes a Next-Best- View O.+1 and moves the robotic arm to this viewpoint to get a new observation D which will be integrated into M Then, a set of new grasps are predicted using M4+1. This observe-predict-plee closed-loop policy is continuously running until the stopping criterion is met.
Plan View
Figure 2: Overview of the next-best-view plan- ning policy for grasping.
Method
We now present the AffordanCE-driven Next-Best-View planning policy (ACE-NBV), a learning framework that leverages the paradigm of novel view imagery to predict the grasp affordances for the unseen views and achieve the closed-loop next-best-view planning according to predicted grasp affordances. As shown in Fig. 3, our model composed of two modules: 1) a view-aware grasp affordances prediction module, and 2) an affordance imagery of unseen novel views module. We discuss these two modules in Sec. 4.1 and Sec. 4.2, respectively, followed by the proposed next- best-view planning policy in Sec. 4.3 and training details in Sec. 4.4.
3

---

direction V
Graap Quality
2.
G
Cray
Grasp Rotation
ggo
Grpperr Withh
3D Conv Encoder
Depth Renderer
TSDF M
3D Feature Volume C
Figure 3: Architecture of the proposed ACE-NBV. The input is a TSDF voxel field M obtained from the depth image. The upper branch predicts the grasp affordances for the target object and the lower branch synthesizes the depth image of different views, including the previously unseen views. Both branches share the same tri-plane feature volume C.
4.1 View-Aware Grasp Affordances Prediction
In this work, we define the grasp affordances in the form of grasp quality Gg. grasp center Gp (z. y, 2), grasp orientation GR E (O))), and opening width G of the parallel-jaw grip- per. Following the grasp pose representation in [31], we decouple the grasp orientation GR as the grasp view Gy and in-plane rotation G,. Because of the heavy occlusions in the cluttered scenes, we draw insight that the grasp affordances can be better estimated using the observation whose the view-direction Oy is the same as the grasp direction G., Motivated by this, we propose a novel view-aware grasp affordances prediction module to predict the in-plane rotation G,, grasp quality Gg and gripper width G given specific grasp center Gp and grasp direction G.,
Specifically, given depth image D; E & captured by the depth camera on robotic arm, we first integrate it into a TSDF M, E 4-00400400 which represents a cubic workspace of size L and can be incrementally updated with the new observed depth images. As shown in Fig. 3, at each time step f, our model takes the current TSDF voxel M2 as input and processes it with a 3D CNN network to obtain tri-plane feature volume E R2xxwx3 as in [32], which is shared for view-aware grasp affordances prediction and novel view depth synthesis.
For grasp affordances prediction of the occluded target object, we first uniformly sample N points in 3D as grasp centers Gp., Gp....., Gpx in the given 3D bounding box, and set the current obser- vation view O, to grasp view G. To predict grasp affordance for a specific grasp center Gp. we cast a ray r (Gp: Gv) ) from orthographic cameras [33] origins o along the direction Gv passing through the grasp center Gp. In particular, as shown in Fig. 3, these rays are cast into the tri-plane feature volume C and n points S {s1, S2; { S, } uniformly sampled along each ray. Then, query the local features C., of these 3D points from the shared tri tri-plane feature volume, and these local features are integrated together ray feature Cray via max pooling, i.e.,
Cryy maxpool Css, Cay...,,Cs..
(1)
This ray feature captures the occlusion relationships between objects along the ray direction, which is essential for grasp affordances prediction in cluttered scenes.
In addition, local geometry information around the grasp center another key factor for grasp af- fordances prediction on a target object. Therefore, as shown in Fig. 3, we draw small 3D bounding box along the diewdirrection O, with fixed length and width around grasp center Gp- Then we obtain the tri-plane features of the eight vertices of this cuboid Cverty Cverty .... Cverts and catenate these features with the feature of grasp center CG, This concatenated feature is denoted as local geometry feature C geu i.e.,
C goo concat( Cvert Cverta ..., Cverts CG.).
(2)

---

Based on the above ray feature and local geometry feature, we implement the grasp affordances prediction module as a small fully-connected neural network fG that takes Gv: Cray Cgso as input and outputs the in-plane rotation G,, grasp quality G2 and gripper width G
G, Gy G (Gv; Cray Cgoo).
(3)
In (3), Gy is a 3-dimensional unit vector that represents the view direction, G [0, Wmax] where Wmax is the maximum gripper width, and grasp quality Gg E [0, 1].
4.2 Affordance Imagery with Implicit Neural Representation
Inspired by the impressive performance of neural radiance fields in the new view synthesis, we adopt the same paradigm to enable our model imagine the scene geometry from previously unobserved viewpoints. With this scene imagery ability, model can predict reasonable grasp affordances under unseen viewpoints, and the imagined grasp affordances tion. Specifically, as in Fig. 3, we share the same tri- tri-plane feature volume C' for novel view depth synthesis and grasp affordances prediction, and the network is trained with two tasks simultaneously.
used for the next-best-vice selec-
First, we build geometry decoder upon the shared tri-plane feature volume C for novel view depth synthesis. We implement this geometry decoder as MLP network that takes local feature Cays of 3D point p (zz y, 2) input and predict its signed distance function (SDF) value. Then, for given view direction Dv, we sample series of 3D points along the ray and synthesize depth images D using their corresponding SDF values, following the approach described in NeuS [34], i.e.,
D - (C,
where Fs denotes the whole network branch for novel view depth synthesis. Note that here we only utilize depth images supervision, which differs from the approach described in the original NeuS paper. The optimization with (4) makes the shared tri-plane feature volume able to reason scene geometry under unseen viewpoints, which aids us in grasping affordances imagery.
For grasp affordances imagery, we utilize the method described in (3) to predict grasps at points in the bounding box Tabox from given direction Gy. The model takes current feature volume C as input and imagine grasp affordance map for novel view. Let FG represent the whole network branch for grasp affordances prediction, the affordances imagery pipeline is formulated as:
G + Fa bbox:
(Tbbb C, Gv).
(5)
4.3 Next-Best- View Planning for Grasping
We design closed-loop next-best-view planning policy to determine the next observation view which is most beneficial for grasping the target object when feasible grasp is found. Let G, be view from set of potential next observation views Gy C SE(3).. The goal of our next-best-view planning policy is to find the next observation view Ov, with the highest predicted grasp quality G from set of imagined grasp affordances for the target object, i.e.,
O O <- arg max (Tabox Cr, G.). G.EG.
We adopt methodology similar to that presented in [4] to generate potential next grasping views Gy. and predict the imagined grasp affordances under these potential views with the above affor- dances imagery module. In addition, we use two simple stopping criteria to decide whether to stop or to continue to find the next observation view. First, the policy is terminated the highest grasp quality G of currently predicted grasps above a given threshold Qmax. Second, we impose a maximum number of next-best- view planning steps Tmax. We summarize the overall next-best-vvew planning policy for grasping as an algorithm in the appendix.
4.4 Training
The network is trained end-to-end using ground-truth grasps obtained through simulated trials. To achieve generalizable grasp affordances imagery, we generate three types of input-output data
5

---

pairs: front-observe- soont--obrrre-froot-graap front-observessid--rraas and multi-observe-frot-- The wront-observe-fron--eee means that our model takes one front view depth image as input and predicts grasp affordances and depth image under the same front view. Similarly, the other two types of data pairs represent different input and prediction task pairs under different views. Note that multi-observe means the imput TSDF is fused from several depth images from different views, aiming construct data that closely resembles the input distribution during the reasoning process of the closed-loop grasping. By incorporating such data pairs into the dataset, we enable the model to predict the affordance of objects in unobserved directions, thus allowing for a more accurate evaluation of candidate observation directions.
The training loss consists of two components: the grasp affordances prediction loss LA and the novel view depth synthesis loss Ls. For the grasp affordances prediction loss, we adopt similar training objective as VGN [1]:
= LG Go) Gr G.) Lw (G Gw)).
In (7), G represents the ground-truth grasp, and G represents the predicted grasp. The ground-truth grasp quality is denoted by G., which takes on value of 0 (representing failure) or 1 (representing success). The binary cross-entropy loss between the predicted and ground-truie grasp quality is represented by Ly. The cosine similarity between the predicted rotation G, and the ground-truth rotation G, is denoted as f.. while Lu represents the fz-distance between the predicted gripper width GhD and the ground-truth gripper width G.,. The supervision of the grasp rotation and gripper width is only applied when the grasp is successful (i.c., G3 = 1).
The geometry loss is calculated using the standard & loss between the synthesized depth image and the actual depth image, and is denoted by Lg. The final loss C is obtained by adding the affordances loss and the geometry loss together, i.e., L = LA + Ls.
5 Experiments
We evaluate the performance of our algorithm by grasping an occluded target object in simula- tion and real-world environments. We use 7-DoF Panda robotic arm from Franka Emika, with RealSense D435 attached the end effector. Our algorithm was implemented in Python, using PyTorch for neural network inference and ROS the hardware interface. We use Open3D [35] for TSDF fusion, and all experiments use TRAC-IK [36] for IK computations, and Movelt [37] for motion planning and are run on same computer.
In our experiments, we evaluate the performance of our method and existing methods with the following metrics. Success Rate (SR): the proportion of successful grasps. Failure Rate (FR): the proportion of failed grasps. Abort Rate (AR): the proportion of cases where valid grasp was found even after the maximum number of views was reached. #Views: the average number of views planned by the algorithm for each round. Time (only in real world experiments): the total time consumed, including observation, planning, and execution.
We compare the performance of our algorithm with the following baselines: 1) initial-view: most work in visual grasp detection considers a single viewpoint for grasp detection. this baseline the robot detects grasp using only the initial view. 2) top-view: the robot detects grasps from single top-down image captured on the top of the workspace center, which is typical setting for tabletop robotic manipulation. 3) fixed-traj.: the robot captures 4 images by moving along a circular trajectory centered on the target object, looking down at 30,, with uniform intervals. The images are subsequently used for TSDF fusion and grasp detection. 4) GIGA: a simple text-best-view policy based on GIGA [2], which uses the predicted best grasp direction as the next view direction.
5) Breyer's: the state-of-the-ar closed-loop next-best-view policy from [4], using the geometry- based information gain approach to plan next- next-best- for target object grasping. All baselines use the same controller described above to generate the robot motion.

---

5.1 Simulated Experiments
As shown in Fig. 4, simulation environments, we generate simulation scenes in PyBullet [38] us- ing the "packed" approach described in [1] and the object with the smallest amount of visible pixels in initial view is selected as the grasping target. Table 1 shows the results of the 400 experiment trials with each method in simulation environments.
We observe that the success rate of the initial-view method is the lowest since the strong occlusion of the target object in the initial view leads to difficult grasp affordance prediction. Top-view re- sults in high success rate because the target can always be grasped from the top in the generated scenes, making simple and effective strategy find feasible grasps. The success rate of the fixed-traj. algorithm is higher than the initial-view it collects more scene information from predefined viewpoints. On the other hand, the state-of-the-ar closed-loop next-best-vien planning method Breyer's 's achieves superior grasping performance and it requires only few new observa- tions. Finally, compared to Breyer's, our method achieves comparable success rate with fewer new observations and obtains a significant improvement on the success rate when only one new obser- vation (2- Views SR) is allowed. This indicates that our model can find more informative views for grasping the target object. Moreover, the qualitative results of next-best-view planning of model shown in Fig. 4, and the examples verify the effectiivness next- next-best-view planning ability of our proposed method.
To investigate the influence different components within our model, we test two variants in Ta- ble I: (i) ours w/o feature Cgoo and Cray where the features Czea and Cray are replaced with feature CG, of the grasp center, and (ii) ours w/o novel view synthesis branch that removes the novel view depth synthesis in Sec. 4.2. We find that ours w/o features Cgen and Cray results in significantly worse grasp affordances prediction, and ours w/o novel view synthesis branch needs more observation views to achieve comparable performance. Moreover, the results of 2- Vieww SR suggests their importance in finding informative views.
Table 1: Results from the simulation experiments
Method
SR
FR
AR
#Views
2- Views SR
initial-view
71 %
7%
22%
1.00
N/A
top-view
79%
6%
15%
1.00
N/A
fixed-traj.
77%
6%
17%
4.00
N/A
Breyer's [4]
81%
8%
11%
4.31
77%
Our w/o feature Cgeo and Cray
74%
8%
18%
2.54
72%
Ours w/o novel view synthesis branch
80%
10%
10%
3.86
75%
Ours
83%
7%
10%
2.97
80%
Figure The above images illustrate the next-best-view planning in two simulation scenes. Red and blue pixels in the depth images represent randomly sampled grasp candidates, with red indicating successful grasps and blue indicating unsuccessful grasps as predicted by the model.
7

---

5.2 Real Robot Experiments
We test our model with 7-DoF Panda robotic arm in real-world cluttered scenes shown in Fig. 1. We initialize the robotic view. Note that each scene is tested 5 times with small perturbations in the initial robotic arm position and object locations.
position where the target object is partially visible in the initial
The results from 5 grasping trials are reported Table 2. Intuitively, the difficulty varies among different scenarios. Some objects are heavily occluded, requiring the robot find suitable directions for observation. Additionally, some objects have unique shapes, limiting stable grasping pose to specific directions. In the relatively easier scenes 4 and our method outperforms the initial-view and fixed-traj. baselines, while achieving comparable grasping performance the state-of-the- art GIGA and Breyer's s method. Furthermore, our algorithm has advantages in much more difficult scenes and 2, where it finds a feasible grasp for the occluded target object with fewer additional observations. For more results, please refer to the supplementary appendix and videos.
Table 2: Real-world experiments setup and results. The first column shows the arrangement of the and the second column displays view the initial position of the robotic arm. The target object has been circled with red dashed line. Our method is capable of achieving comparable grasp success rates (SR) using fewer views (iews)..
Sctup
Initial
Method
SR
FR
AR
#Views
Time's
top-down
0/5
3/5
2/5
1.0
18.3
lixed-traj.
3/5
2/5
0/5
4.0
29.4
GIGA [2]
3/5
1/5
1/5
5.4
32.1
Breyer's [4]
4/5
0/5
1/5
4.8
24.7
Ours
3/5
2/5
0/5
3.4
23.1
top-down
2/5
1.0
fixed-traj.
4.0
30.5
GIGA [2]
0/5
3.6
31.8
Breyer's [4]
0/5
2/5
5.2
25.9
3.0
top-down
2/5
1/5
1.0
16.5
lixed-traj.
4/5
1/5
015
4.0
29.8
GIGA [2]
4/5
0/5
1/5
3.8
30.2
Breyer's [4]
3/5
2/5
0/5
4.8
23.0
Ours
5/5
0/5
0/5
3.2
24.7
top-down
0/5
0/5
5/5
1.0
19.1
trad-trajj
2/5
3/5
015
4.0
30,5
GIGA [2]
3/5
1/5
1/5
5.0
28.6
Breyer's [4]
2/5
1/5
2/5
4.4
23.2
Ours
3/5
2/5
0/5
2.8
23.7
top-down
1.0
fixed-traj.
0/5
4.0
29.7
GIGA [2]
0/5
1.8
23.5
Breyer's [4]
0/5
3.0
21.7
2.6
Conclusion and Limitations
In this paper, we introduce next-best-view planning framework that leverages the imagined grasp affordances plan the robotic arm's new observation views for grasping target object in occluded environments. This framework is motivated by the idea that the grasp affordances can be well- predicted when the observation direction aligned with the grasping direction. Through both simu- lated and real-world experiments, demonstrate the effectiveness and robustness of our approach compared to previous works.
Our next-best-view planning framework involves neural network inference and re- quires the sampling of multiple views, leading to high computational cost. In addition, the robotic arm motion planning is not considered in our method, and some unsatisfactory grasp executions exist in real robot experiments. In the future, we plan integrate motion planning into our method to perform more complex tasks in more challenging environments.
Limitations:

---

Acknowledgments
This work supported by the Shanghai Artificial Intelligence Laboratory, National Key R&D Pro- gram of China (2022ZD0160100) and the National Natural Science Foundation of China (62106183 and 62376222).
References
[1] M. Breyer, J. J. Chung, L. Ott, R. Siegwart, and J. Nieto. Volumetric Grasping Network: Real- time 6 DOF Grasp Detection in Clutter. In Conference Robot Learning, pages 1602-1611. PMLR, Oct. 2021.
[2] Z. Jiang, Y. Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies Between Affordance and Ge- ometry: 6-DoF Grasp Detection via Implicit Representations. Robotics: Science and Systems (RSS), 2021.
[3] E. Arruda, J. Wyatt, and M. Kopicki. Active Vision for Dexterous Grasping of Novel Objects. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2881- 2888. IEEE, Oct. 2016.
[4] M. Breyer, L. Ott, R. Siegwart, and J. J. Chung. Closed-Loop Next- Bext-Best- View Planning for Target-Driven Grasping. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1411-1416 Oct. 2022.
[5] B. Mildenhall, Srinivasan, Tancik, T. Barron, R. Ramamoortii, and R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Communications of the ACM, 65(1):99-106, 2021.
[6] R. Newbury, M. Gu, L. Chumbley, A. Mousavian, C. Eppner, J. Leitner, J. Bohg, A. Morales, Asfour, D. Kragic, and others. Deep Learning Approaches Grasp Synthesis: A Review. arXiv e-prints, pages arXiv-2207, 2022.
[7] G. Du, K. Wang. S. Lian, and K. Zhao. Vision-based Robotic Grasping from Object Localiza- tion, Object Pose Estimation to Grasp Estimation for Parallel Grippers: A Review. Artificial Intelligence Review, 54(3):1677-1744 2021.
[8] A. Mousavian, C. Eppner, and D. Fox. 6-DoF GraspNet: Variational Grasp Generation for Object Manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2901-2910, 2019.
[9] K. Fang, Y. Zhu, A. Garg, A. Kurenkov, V. Mehta, L. Fei-Fei, and S. Savarese. Learning Task- Oriented Grasping for Tool Manipulation from Simulated Self-Supervision. The International Journal of Robotics Research, 39(2-3):202-2111 2020.
[10] Q. Dai, Y. Zhu, Y. Geng, C. Ruan, J. Zhang, and H. Wang. GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1757-1763. JEEE, 2023.
[11) S. Jauhri, I. Lunawat, and G. Chalvatzaki. Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Rendering, arXiv preprint arXiv:2306.07392 2023,
[12] H. Huang, D. Wang, X. Zhu, R. Walters, and R. Platt. Edge Grasp Network: A Graph-Based SE(3)-invariant Approach to Grasp Detection. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3882-3888, 2023.
[13] Mahler, J. Liang, Niyaz, M. Laskey, Doan, X. Liu, A. Ojea, and K. Goldberg. Dex- net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics. Robotics: Science and Systems (RSS), 2017.

---

[14] J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg. Learning Ambidextrous Robot Grasping Policies. Science Robotics, 4(26)xeau44984, 2019.
[15] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-GraspNett Efficient 6-DoF Grasp Generation in Cluttered Scenes. In IEEE International Conference on Robotics and Automation (ICRA), pages 13438-13444. IEEE, 2021.
[16] M. Gualtieri, A. Ten Pas, K. Saenko, and R. Platt. High Precision Grasp Pose Detection in Dense Clutter. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 598-605. IEEE, 2016.
[17] D. Morrison, P. Corke, and J. Leitner. Multi-View Picking: Next-best-view Reaching for Im- proved Grasping in Clutter. In International Conference on Robotics and Automation (ICRA), pages 8762-8768, 2019.
[18) R. Zeng, Y. Wen, W. Zhao, and Y.-. Liu. View Planning in Robot Active Vision: A Survey of Systems, Algorithms, and Applications. Computational Visual Media, 6(3):225-244) 245, Sept. 2020. ISSN 2096-0433, 2096-0662.
[19] J. Delmerico, S. Isler, R. Sabzevari, and D. Scaramuzza. A Comparison of Volumetric Informa- tion Gain Metrics for Active 3D Object Reconstruction. Autonomous Robots, 42(2):197-208, 2018.
[20] S. Kriegel, C. Rink, T. Bodenmüiller, and M. Suppa. Efficient Next-best-scan Planning for Autonomous 3D Surface Reconstruction of Unknown Objects. Journal of Real-Time Image Processing, 10:61 11611-631, 2015.
[21] W. R. Scott, G. Roth, and J.-F. Rivest. View Plamning for Automated Three threc-dimensional Object Reconstruction and Inspection. ACM Computing Surveys (CSUR), 35(1):64- 96, 2003.
[22] A. Hutchinson, R. L. Cromwell, and A. C. Kak. Planning Sensing Strategies in Robot Work Cell with Multi-sensor Capabilities. In Proceedings. 1988 IEEE International Conference on Robotics and Automation, pages 1068-1075. IEEE, 1988.
[23] E. Johns, S. Leutenegger, and A. J. Davison. Pairwise Decomposition of Image Sequences for Active Multi- Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3813-3822, 2016.
[24] M. Mendoza, J. I. Vasquez-Gomez. H. Taud, L. E. Sucar, and C. Reta. Supervised Learning of the Next-Beste View for 3D Object Reconstruction. Pattern Recognition Letters, 133:224-231, May 2020. ISSN 01678655.
[25] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the Wild:Le 6DoF Closed- Loop Grasping from Low-Cost Demonstrations. IEEE Robotics and Automation Letters, 5(3); 4978-4985, 2020.
[26] S. Lee, L. Chen, J. Wang, A. Liniger, S. Kumar, and F. Yu. Uncertainty Guided Policy for Ac- tive Robotic 3D Reconstruction using Neural Radiance Fields. IEEE Robotics and Automation Letters, 7(4):12070-12777 2022.
[27] J. M. Drozdzal, D. Nowmuzezahrai, D. Meger, and A. Romero-Soriano, Uncertainty- Driven Active Vision for Implicit Scene Reconstruction. arXiv preprint arXiv:2210.0097 2022.
[28] L. Jin, X. Chen, J. Rückin, and M. Popović. NeU-NBV: Next Best View Planning Using Uncertainty Estimation in Image-Based Neural Rendering. arXiv preprint arXiv:2303.01 2844, 2023.
[29] K. Lin and B. Yi. Active View Planning for Radiance Fields. In Robotics Science and Systems, 2022.
10

---

[30] X. Pan, Z. Lai, S. Song, and G. Huang. ActiveNeRF: Learning where to See with Uncertainty Estimation. ECCV 2022: 17th European Conference, Oct. 2022, Proceedings, Part XXXIII, pages 230-246. Springer, 2022.
[31] H.-S. Fang. C. Wang, M. Gou, and C. Lu. GraspNet-IBillion: A Large-Scale Benchmark for General Object Grasping. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11441-11453 11400. IEEE, June 2020.
[32] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger. Convolutional Oceu- pancy Networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part III 16, pages 523-540. Springer, 2020.
[33] Y.-C. Lin, P. Florence, A. Zeng, Barron, Y. Du, W.-C. Ma, Simeonov, Garcia, and Isola. MIRA: Mental Imagery for Robotic Affordances. In Conference Robot Learning. pages 1916-198111 1927. PMLR, 2023.
[34] Wang. L. Liu, Liu, C. Theobalt, T. Komura, and Wang. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Advances Neural Information Processing Systems, 2021.
[35] Q. Zhou, J. Park, and V. Koltun. Open3D: Modern Library for 3D Data Processing. arXiv preprint arXiv:1801 10009477 2018.
[36] P. Beeson and B. Ames. TRAC-IK: An Open-source Library for Improved Solving of Generic Inverse Kinematics. In IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 928-935. IEEE, 2015.
[37] S. Chitta, I. Sucan, and S. Cousins. Moveit![ros topics]. IEEE Robotics & Automation Magazine, 191):18--9, 2012.
[38] E. Coumans and Bai. Pybullet, Python Module for Physics Simulation for Games, Robotics and Machine Learning. 2016.

---

Appendix
A Pseudo Code of the Proposed ACE-NBV Policy
We summarize the overall NBV planning policy for grasping target object in the following algo- rithm 1. In the experiment, we set max to 8 and 4max to 0.95.
Algorithm Grasp Affordance Prediction and Next-Best View Planning
Input: A cluttered scene, an occluded target object given its 3D bounding box bbox
Output: A feasible grasp G of the target object
for t Σ Tmax do
M f Dr
Intergrate depth image into TSDF
C₂ < 3D (M
Encode feature
if (Tobox: C₁, Ov,1) 9max then
0v,1+1 arg maXG.EG, Ggy (Tbbox: Ct Gv)
Evaluate candidate next views
Move camera to Ov,t+1
> Go to the next-best-view
else
Execute grasp G (T Ct Ov,t)
Break
end if end for
B Network Architecture and Implementation Details
We adopt the same encoder as in GIGA that takes TSDF M, TR400x40x40 as input and outputs feature embedding for each voxel with a 3D CNN layer. Then, the tri-plane feature grids is con- structed by projecting each input voxel on a canonical feature plane via orthographic projection. Then, three feature planes are processed with a 2D U-Net that consists of series of down-sampling and up-sumpppin sampling 2D convolution layers with skip connections. The output is formulated as the shared tri-plane feature volume C E IR 3x 10x32 where 32 is the dimension of the feature embed- ding.
Based on the shared tri-plane feature volume, the local feature Cp of a 3D point p (I, y, 2) is obtained by projecting it to each feature plane and querying three features Cp. Cp,, Cp. at the projected locations using bilinear interpolation, and the local feature Cp is the concatenated feature of these queried features, i.e., Cp concat(Cp. Cpy: C..). We implement our grasp affordance prediction network with five layer fully-connected network with residual connections. The input dimension of this MLP network is 3 + 96 + 9 x 96 963 which is composed of view direction unit vector v E RR3 the ray feature Cuay E IR and the local geometry feature Cgeo E R9996 The output dimension for grasp affordance prediction is 1 + + = 3 which includes the grasp quality Gy: in-plane rotation G,, and gripper width G..
As for the novel view depth synthesis, we employ a MLP network that takes the 3D point feature Cp E I9 input and output the SDF value of this point, and adpot the same rendering technique with NeuS synthesize depth images (17 1 = 5). We sample 128 rays in depth image in each batch, each ray consisting of 64 uniformly sampled points and extra 4 x 32 points following the importance sampling rule. We set the near and far range close to the ground truth depth at the beginning of training, and then gradually relax the range to the maximum range of the implicit feature volume.
For experiments in simulation and real word, the size of cubic workspace L 30cm. The size of the $2...., uniformly sampled with step of 1. The sizes of the three datasets fromt-observe-fronte observe-side- -isegre- grasp and multi-obserre- front-gg-a grasp are IM, IM and 2M grasps, respectively. Each scene contains 240 grasps and = 12 ground-trum -tutt depth images with the resolution of 480 640. After data cleaning and balancing, there are about 40% data left. We separate the datasets randomly into 90% training and 10% validation. We train the models with the Adam optimizer and learning rate of 2 10- and batch sizes of 128. All experiments run computer equipped with an Intel Core 1390KK and a GeForce RTX 4090.
cubic for Cvert is 0.25, which is 7.5cm in the real world. The points S = {si, S2, 8.,) for Cray is front-
12

---

C Extra Experiments for Intuition
We conducted extra experiments in simulation to justify our intuition that the grasp affordances can better-measured using the observation when the observation view direction is the same as the grasp view. each randomly selected case, the network receives depth image from different directions and is then required to predict grasping pose of the target object in the same direction. The results quite evident: the prediction is much better when those two directions are the same.
(b) case
(c) case 3
Figure 5: The network receives TSDF fused from depth image captured either from the front (lower left) or the side (lower right) as input. It is required to predict grasp in the frontal direction, which aligns with the direction of the depth image used for visualization in the upper row. The color red indicates high-quality grasps, while blue represents low-quality ones.
13

---

D Qualitative Results of Real Robot Experiments
We present qualitative results in Fig. and 7 and recommend readers watch the supplementary video for more comprehensive real robot experimental results. Note that our model can select reasonable next-best-view observe the occluded target object. We show representative failure case in Fig. 7, where small errors in grasp affordance prediction leads to unsuccessful grasp. This small pre- diction inaccuracy occurs in most failure experiments. Therefore, in the future, we plan to exploit better grasp affordance prediction module to improve the success rate of our method.
(a) Initial View
(b) Selected Next-Best View
(c) Execute Grasp
(d) Grasp Success
Figure 6: Success Case. The robot planned one view to observe the target box and successfully grasped it.
(a) Initial View
Selected Next-Best-Viee
(c) Exccute Grasp
(d) Grasp Fail
Figure 7: Failure Case. The robot failed predict accurate grasp affordances of the target object after obtaining a new observation. As a result, the grasping failed.
14


Reviews:
[{"review_id": "q6OPNpQ5pA", "reviewer": "Reviewer_hPBU", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces ACE-NBV, a method for next-best-view (NBV) planning in cluttered environment. The proposed architecture is similar to GIGA, with the exception of using affordance module to jointly predict and grasping pose and the next-best-view pose. Specifically, the model integrate depth images into TSDF and then converted into a tri-plane feature volume using a 3D CNN encoder. Novel view depth synthesis following NeRF paradigm is used to aid in learning the encoder. Grasp affordance and grasping params (rotation, gripper width) are predicted given the learned feature volume, along with randomly sampled grasp candidates within the given object bounding boxes of the target object. When the proposed grasp fails to meet the criteria, the next camera poses are generated using the proposed grasp pose.", "summary_of_recommendation": "Based on the strengths and weaknesses section, I would currently recommend weak rejection for the paper, despite the idea of using NeRF for NBV planning is interesting. More experiments are needed to demonstrate the advantages of the proposed method.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- In Eq.1, why the ray feature captures the occlusion relationships? A feature that captures occlusion relationships should be sensitive to the order of input points. However, swapping the order of two neighboring points (S_i, S_{i+1}) may not affect the maxpool results, but the occlusion relationship between S_i and S_{i+1} is changed after swapping.\n- Please provide more details explaining what is front-observe / side-observe in Eq.5 and why using them.\n- What's the definition of Gq*? It appears to be inconsistent in the paper: Gq*(Tbbox,Ct,Ov,t) and Gq*(Tbbox,Gv) (Algorithm 1 in Supplementary)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength\n\n- The idea of using NeRF for NBV planning is interesting\n- Related work is sufficiently addressed\n- Real-world experiments are conducted and limitations are discussed\n\nWeakness\n\n- The evaluation cases are overly simplified, and limited to objects with basic geometry (box and cylinder) in the scene. The author may consider incorporating more complex objects (e.g. toy) similar to those used in Breyer's paper. \n- Given the simplicity of the evaluation scenarios, a top-view or a randomly sampled fixed 2 or 3 near-top-view might be sufficient to predict the grasping pose. However only fixed 4-view are discussed (in both simulation and real-world experiments), top-view is not discussed in real-world experiments. Moreover, The author may consider including more details on how the fixed views are generated. Well-generated fixed views, such as those randomly generated around top-view, have the potential to perform effectively across different scenarios. In contrast, poorly generated fixed views, like multiple side-views from the same direction and close to each other, may work poorly.\n- The use of bounding box of the occluded objects seems to be not realistic - if the target object is fully (or mostly) occluded, the bounding box cannot be detected from the current observation. However, information-based NBV method could still work in these scenarios by simply finding the next view that maximizes the information, until observing the occluded object. The author may consider evaluation scenarios with fully occluded target objects.\n- The proposed method is not showing significant advantages over baselines given the current result in simple scenarios, and considering the baseline's effectiveness in fully occluded scenarios. The author may consider conducting further experiments in more complex scenarios to demonstrate the superiority of the proposed method.\n- The author claims \"it is intuition that the grasp affordances can be better-measured using the observation when the observation view direction is the same as the grasp view\". However, it doesn't seem to be intuitive, as human rarely use observation view same as grasp view (e.g. grasping a bottle). The author may consider conducting experiments to justify this, for example, using two affordance networks with shared weights on local features to separately predict the grasping pose and next-best-view pose.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "NRkP9Wa7cFe", "reviewer": "Reviewer_6CeP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies closed-loop view selection for grasp planning, where the goal is to grasp a target object which might be partially occluded initially. To address this, they use a continually updating TSDF scene representation, which is converted to a feature field via a 3D CNN. To compute a new view, this feature volume is rendered from multiple viewpoints and grasps are sampled with the grasp axis facing these viewpoints. If grasp confidence is sufficiently high, a grasp is executed, otherwise the next highest confidence view is used as a new viewpoint.", "summary_of_recommendation": "The approach is a reasonable strategy for affordance-based view selection, however it is not, as the authors claim, \"the first\" to propose affordance-driven next-best-view policy. I have some additional concerns with the marginal benefits shown in experiments compared to single view top-down planning in simulation (which is missing in real). If top-view planning is successful in all these scenes, it's difficult to justify the added complexity of this system over a heuristic top-down approach. In that case, more test scenes should be evaluated on to illustrate the benefits of multi-view planning. I also feel the choice of describing this method as a NeRF is erroneous, since in fact the method doesn't represent radiance anywhere, or reconstruct the scene from multi-view RGB images, but rather uses TSDF fusion from a depth camera.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- The writing should clearly describe the difference in approach between this work and the \"grasping in the wild\" paper provided above.\n- In the videos it seems like many times the first view selected is a top-down view, after which a grasp is carried out. Indeed, in the simulated results the difference between top-view and this method is only 4%, which is hardly significant. The difference between 2-view and top-view is even less (1%). In addition, top-view is not provided in the physical experimental results. Does this method actually present an improvement over a single top view in the physical experiments provided? \n- What is Tmax? Providing it would be helpful in contextualizing the number of views taken during experiments.\n- Additional experiments with more cluttered scenes where top-down grasping fails would more convincingly show the benefit of this method.\n- Including the time taken during experiments would be a useful metric to compare, since lower number of actions doesn't necessarily always mean faster.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- The idea of using grasp affordances from imagined views to guide visual exploration of scene before executing a grasp is a good idea\n- The figures in this paper were easy to follow\n\nWeaknesses\n- An important missing citation: https://arxiv.org/abs/1912.04344, this paper from 2020 takes a very similar approach to the problem by rendering a TSDF from different views and predicting grasp affordances from those novel views, with which a closed-loop grasp is made. I feel that the wording in this submission that \"we, for the first time, introduce an affordance-driven NBV planning policy\" is inaccurate. The idea of using grasp affordances from novel views is precisely the idea this prior work proposes.\n- I'm confused why the authors refer to their method as using a Neural Radiance Field. The 3D representation used is a TSDF, and the feature volume output by the CNN encoder is not a NeRF, but rather a feature volume from which grasp properties and SDF values are rendered. Rendering this volume from different views does not make it a NeRF, as there is no  radiance represented anywhere in the pipeline. \n- It seems to me that taking in a 3D bounding box as input breaks the assumption that the object is occluded, as constructing such a bounding box would require full knowledge of the extents of the object, defeating the purpose of the occlusion-tolerant capabilities. This should be discussed in the limitations section, how could such a box be obtained automatically without observing the occluded part of the object? Could the system be adapted to work with other methods of specifying a target object? \n- The experimental evaluation in physical settings is lacking, since only 4 scenes are considered (each scene is repeated 10 times, but it's unclear how significant the \"small perturbations\" in arm and object positions have on the experiment diversity). In addition, the method provides fairly marginal success improvements over baselines, even over a single-view top-down baseline. \n\nPage 2 line 84: \"neural radiation fields\"", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "7YHsuanH3_", "reviewer": "Reviewer_3RPC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method for using novel view synthesis to plan the best grasp of an object in a cluttered scene. Given a 3D bounding box of the target object in the cluttered scene, first a TSDF is built from the RGBD observation. Then a learned grasp success model is queried. If it finds a grasp it predicts will be successful, the robot executes the grasp and the trial is over. If not, the learned Next-Best-View model is queried to predict the next view to move the camera to. The RGBD image is integrated into the TSDF and the loop is repeated. The results show that the proposed method outperforms the baseline in either success rate or number of required views or both.\n", "summary_of_recommendation": "This paper presents an interesting and novel methodology, and does an excellent job comparing to several baselines. If the authors are able to address my comments in the rebuttal section, my recommendation would be weak accept. The clarity of section 4.2 and the number of real world scenes are the biggest issues to address.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "For the rebuttal, the authors should make the following changes:\n\nSection 4.2 is quite unclear. How are depth images generated by sampling SDF values at individual points in the tri-plane? Why/how are the number of images downsampled from \\eta to \\gamma and then what exactly is F_s doing to return to \\eta number of images? Also, the observe-grasp data pair described in the last paragraph don’t seem to be related to the rest of the section. Is this a data collection detail? Should this paragraph belong in section 4.4? I would strongly recommend having a colleague who is not already familiar with this work proofread this section and then rewrite it for clarity.\n\nThere are only 4 real-world scenes, and they’re all pretty similar: the target object is hidden behind one or more large objects. There really should be more scenes with more variation. For a sanity check, this should be tested on a scene with the target object in front of other objects just to make sure it works in simpler cases. It’s also worth testing having the target object completely surrounded on all sides, requiring the robot to reach down from above. Additionally, other tests could include creating a “U” shape of objects around the target, forcing the robot to pick from either above or the opening of the “U”. There are many other potential variations that could be run and I leave it up to the authors to determine them, but I strongly recommend adding at least 4 scenes, if not more, with significant variation.\n\nFinally, this last issue is more of a pedantic one: This paper doesn’t use NeRFs (neural radiance fields), key word here being “radiance”. The methodology here, as far as I can tell, exclusively uses depth images, but radiance is never computed or used. The correct term for what this paper uses would be Neural SDF. Please fix the terminology.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While grasping is not a new task, and in fact has been one of the most studied robotic tasks, combining it with a NeRF-based next-best-view method is. Grasping in cluttered scenes is hard, and this paper proposes a novel way to solve that task. By utilizing the underlying 3D representation to predict where the next best grasp might be, the robot is able to move the camera to that location to better estimate the grasp quality. This is an interesting idea. Furthermore, a major strength of this paper is the extensive use of baselines including ablations of the proposed methodology as well as a completely different method from prior work. Results are shown across all methods and for both sim and real environments. They make a convincing argument as to why the proposed method is superior to alternatives.\n\nThe biggest weakness of the paper is that a few of the components are relatively simplistic. The grasp policy assumes the gripper travels along a straight line, closes the gripper, and presumably travels directly back. While this might work for some cluttered scenes, it won’t work in many more, e.g., a cluttered kitchen cabinet. Additionally, the next-best-view planner utilizes a greedy approach, i.e., it picks the view that predicts the highest likelihood of a successful grasp. Why is this the best way to pick the next view? Why not use another metric like information gain? Or maybe even a planner that predicts several views? Furthermore, the results only evaluate on 4 fairly similar scenes on the real robot (detailed more in the rebuttal section). To be clear, none of these are deal-breakers, merely observations.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "5LcT_aRDVS", "reviewer": "Reviewer_pq4R", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors propose an active perception pipeline for grasping in a cluttered scene. Building on the work of Breyer et al. [1] and Jiang et al. [2], the authors propose using a view-dependant grasp affordance function to detect grasps and use an SDF-based reconstruction head to encourage reconstruction for the prediction of grasp affordance from unseen views. The proposed approach ACE-NBV compares favorably to the work of Breyer et al. [1] and some other simple next-best-view strategies.", "summary_of_recommendation": "While the authors propose some interesting ideas, there are some odd choices used in the methodology of novel-view depth prediction that have not been justified in the paper. Moreover, a comparison with a significant baseline is missing due to which the paper cannot already be accepted in its current form.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- How well does the method perform when directly trained with ground-truth SDF values for the SDF reconstruction?\n\n- How expensive is the NeuS volume rendering step?\n\n- How well does the method perform compared to a simple GIGA[2]-based next-best-view setup?\ni.e. 1. Predict best grasp with GIGA -> 2. Move the franka gripper (and camera) to a view direction that is the same as the predicted best grasp’s orientation -> 3. Execute the grasp if it has high-quality else continue\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n- The extension of GIGA [2] to a view-dependant grasp affordance function is interesting to the community\n- Apart from the methodology section, the paper is well written\n- The figures in the paper are useful for understanding the approach\n- The authors have carried out real-robot experiments with a Franka robot with a cluttered table-top scene\n- The paper clearly mentions some limitations\n\nWeaknesses:\n\n- The methodology section is not so well-written, making the overall approach difficult to understand. For example, the authors should explain clearly that the network is jointly trained for reconstruction (depth synthesis) and grasp affordance prediction to leverage synergies between these two objectives (as shown in [2]). Moreover, it should be made clear in section 4.2 that the depth samples are taken during the data-generation phase.\n- There are issues with the Affordance Imagery part of the methodology:\n    - First, the authors should refrain from using the term Neural “Radiance” since the approach does not use radiance and only geometry or surface.\n    - Secondly, the authors mention the rendering approach NeuS by Wang et al. [28], but the NeuS approach was mainly meant for 2D RGB images without any 3D supervision. The authors should clarify that they use the renderer with depth images instead.\n    - Thirdly, the authors should justify why they opted to use a Volume Rendering technique instead of a Surface Rendering technique, which is typically faster and better suited to depth images.\n    - It is unclear why the authors use depth-image supervision to train the SDF instead of directly using ground-truth SDF values from the simulation. Using ground truth SDF values OR occupancy values (as in GIGA [2]) would directly solve the problem of learning appropriate geometric features in the scene. It would also be more computationally efficient since the expensive volume rendering step would be avoided.\n- While a comparison has been made with the method from Breyer et al. [1], a comparison with GIGA [2] is missing. Testing the method against GIGA is important since it also uses the scene reconstruction head. A simple next-best-view method using GIGA can be devised by just using a view that “looks at” GIGA’s predicted best grasp i.e. view direction is the same as grasp view orientation G_v and centered at the grasp’s position center.\n- Missing citations:\n    - Recent related grasping methods:\n\n    Huang, H., Wang, D., Zhu, X., Walters, R., & Platt, R. (2023, May). Edge grasp network: A graph-based se (3)-invariant approach to grasp detection. ICRA 2023\n\n    Jauhri, S., Lunawat, I., & Chalvatzaki, G. (2023). Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Rendering. arXiv preprint arXiv:2306.07392.\n\n    Dai, Qiyu, et al. \"GraspNeRF: multiview-based 6-DoF grasp detection for transparent and specular objects using generalizable NeRF.\" ICRA 2023\n    - Given that a tri-plane feature volume is used, also cite Peng et al.\n\n    Peng, Songyou, et al. \"Convolutional occupancy networks.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer International Publishing, 2020.\n- Typo: “Neural radiation fields”", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a grasp planner for initially partially occluded object. It continually updates a scene representation and replans grasps based on updates.

Reviewers appreciated the premise of using imagined views to guide exploration, using NeRF for next-best-view planning, clean figures, some real-world evaluation, clear writing (with a few exceptions), and evaluation baselines.

Reviewers raised concerns about missing related work, some method details (such as the bounding volume), simplified/lacking evaluation, and an unclear method section.

The reviewers appreciated the improvements made by the rebuttal, particularly the additional experiments, causing several to raise their assessment, resulting in an accept. The authors are strongly encouraged to ensure all updates make it to the final camera-ready version--this will likely involve making room for the additional experiments in the main content of the paper (and not leaving it to appendices).
Decision: Accept (Poster)