Region-aware Grasp Framework with Normalized Grasp Space for Efficient 6-DoF Grasping
Authors: Siang Chen
Pengwei Xie
Wei Tang
Dingchang Hu
Yixiang Dai
Guijin Wang
Venue: CoRL 2024 (2024)
Keywords: 6-DoF Grasping
RGBD Perception
Normalized Space
Heatmap
TL;DR: For 6-DoF grasping, we propose Region-aware Grasp Framework consisting of Normalized Grasp Space and a efficient Region-aware Normalized Grasp Network, achieving best grasp detection performance with high efficiency and generalization capability.

Abstract: A series of region-based methods succeed in extracting regional features and enhancing grasp detection quality. However, faced with a cluttered scene with potential collision, the definition of the grasp-relevant region stays inconsistent. In this paper, we propose Normalized Grasp Space (NGS) from a novel region-aware viewpoint, unifying the grasp representation within a normalized regional space and benefiting the generalizability of methods. Leveraging the NGS, we find that CNNs are underestimated for 3D feature extraction and 6-DoF grasp detection in clutter scenes and build a highly efficient Region-aware Normalized Grasp Network (RNGNet). Experiments on the public benchmark show that our method achieves significant >20 % performance gains while attaining a real-time inference speed of approximately 50 FPS. Real-world cluttered scene clearance experiments underscore the effectiveness of our method. Further, human-to-robot handover and dynamic object grasping experiments demonstrate the potential of our proposed method for closed-loop grasping in dynamic scenarios.

Region-aware Grasp Framework with Normalized Grasp Space for Efficient 6-DoF Grasping
Siang Chen'r, Pengwei Xie Wei Tang", Dingchang Hu', Yixiang Dai', Guijin Wang!. 1.3,3
Department of Electronic Engineering, Tsinghua University SSenzhen International Graduate School, Tsinghua University Shanghai AI Laboratory, China Equal Contribution
Abstract:
A series of region- based methods succeed in extracting regional features and en- hancing grasp detection quality. However, faced with cluttered scene with po- tential collision, the definition of the grasp-relevant region stays inconsistent. In this paper, we propose Normalized Grasp Space (NGS) from novel region-aware viewpoint, unifying the grasp representation within a normalized regional space and benefiting the generalizability of methods. Leveraging the NGS, we find that CNNs are underestimated for 3D feature extraction and 6-DoF grasp detection clutter scenes and build highly efficient Region-aware aware Normalized Grasp Net- work (RNGNet). Experiments on the public benchmark show that our method achieves significant 20% performance gains while attaining real-time infer- ence speed of approximately 50 FPS. Real-world cluttered scene clearance ex- periments underscore the effectiveness of our method. Further, human-to-robee handover and dynamic object grasping experiments demonstrate the potential our proposed method for closed-loop grasping in dynamic scenarios.
Keywords: 6-DoF Grasping, RGBD Perception, Normalized Space, Heatmap
1 Introduction
Grasping, fundamental task for robotic manipulation, has seen significant advancements. Notably, recent strides in deep-lcarning methodologies have paved the way for data-driven grasp detection techniques. These techniques, capable directly deducing the positions and rotations of avail- able grasps, offer superior generalization to novel scenes compared to the traditional template-base methods. Meanwhile, 6-DoF grasping has grown significantly because of its effectiveness in appli- cations and scenarios that require accurate and reasonable manipulation of objects.
Towards efficient 6-DoF grasping, most grasp detection methods treat the task as a detection-style problem involving the acquisition of optimal grasp positions and rotations from single-view RGBD image or point cloud. Most of them process the global scene information to generate all possible grasps in the scene. Typical works like [1, 2] extract scene geometric features and regress grasps directly. The crucial problem the redundancy and ambiguity of the entire scene information, which is time-consume consumnns.,nng Facing the challenge of grasping in cluttered scenes, recent methods [3, 4, 6, 7, 8] encode global per-points features to infer potential grasp centers and segment graspable regions by integrating the regional points with the global features. While global-to-local schemes have demonstrated effectiveness, leveraging regional information without a consistent representation may undermine the generalization capability for novel scenes. The primary challenge lies in devising a method to extract regional information consistent representation.
In this work, we propose novel Region-aware Grasp Framework decoupling grasp detection into graspable region extraction and follow-up regional grasp generation. Specifically, we propose Nor-
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

malized Grasp Space (NGS), unified representation bridging the patch extraction and regional grasp detection. With extracted region information, the grasp detection problem within NGS can be formulated into a normalized form, which shows the invariance to the gripper size and the ro- bustness to perturbations on the grasp center localization. Thus, considering grasp detection from a region-aware viewpoint, we propose Regional Normalized Grasp Network (RNGNet) extract grasp-relevant geometric features and generate rotation heatmaps, demonstrating enhanced robust- ness and efficiency. Quantitative experiment results show a significant over 20% improvement in the grasp Average Precision (AP) on the public benchmark with 10x faster inference speed than the former SOTA, AnyGrasp [9]. Real-world clutter clearance experiments validate our framework's ss robustness and generalizability. Further demonstration of grasping in dynamic scenes proves the ef- ficiency and flexibility of our proposed region-aware framework and the potential for more complex tasks like target-oriented manipulation. The contributions of the paper mainly include:
1) A novel Region-aware Grasp Framework for 6-DoF grasp detection in cluttered scenes, decou- pling grasp detection tasks into two-stage heatmap prediction and enhancing method efficiency.
2) Normalized Grasp Space (NGS), a novel unified space defined for general parallel gripper grasp detection, empowering aligned and scale-invariant grasp detection in regions.
3) A highly efficient Region Normalized Grasp Network (RNGNet) aiming at extracting region- aware features and predicting high-quality 6-DoF grasps via rotation heatmap prediction.
2 Related Work
Vision-based 6-DoF Grasp Detection: Research has focused on detecting 6-DoF grasps primarily relying on single-view RGBD images or point clouds. Pioneer works ten Pas et al. [10], Liang al. [11] mainly follow a sample-evaluation strategy, generating numerous grasp proposals and selecting optimal grasps through an evaluation model. Qin et al. [2] employ PointNet+ [12] for per-point feature extraction and direct regression of grasp parameters. Recent advancements, benefiting from large-scale datasets [3, 13, 14], adopt an end-to-end strategy. Wang et al. [6] introduces point- wise graspness to represent grasp location probabilities and aggregate regional features for rotation prediction. HGGD [15] employs 2D CNNs to generate heatmaps for grasp locations and integrate 2D semantics with 3D geometric features configure 6D grasps. Tang al. [16] investigates scene regions for qigh-tuality grasp detection but ignores helpful regional characteristics. Diverging from existing approaches, our method employs region-aware scheme forcing the network to capture scene-independent regional features, enhancing efficiency and generalization capability.
Region Normalization for Grasp Detection: Traditional 4-DoF grasp detection approaches [17, 18] conduct image-based normalization or crop the image patches with the exact sizes [19] to improve model performance. However, such 2D frameworks are limited, overlooking critical 3D geometric information and sensitive to camera viewpoints. For 6-DoF grasp detection, the utiliza- tion of regional 3D geometry is also not unique. However, most methods operate under a fixed gripper setting and extract local regions of predetermined sizes [2, 3, 6, 5]. These methods strug- gle to generalize or require model retraining when applied to grippers with different sizes or novel scenes. We normalize regions and corresponding grasps according to adjustable gripper sizes to ob- tain consistent 3D normalized grasp spaces, allowing more efficient grasp detection and adaptation for challenging applications, such as dynamic grasping.
3 Method
Similar to prior works [15, 20, 3}, we focus on the problem of 6-DoF grasp detection of parallel grippers in clutter scenes from single-view RGBD image
E R "x x4 as input. Inspired by [15), we adopt the grasp representation as g = (x. y, z, 0, Y, B, wr, which (z. y, 2) is the 3-DoF translations of grasp centers, w is the grasp width and (0, 7, 8) compose the 3-DoF intrinsic rotation as Euler angles. w is predicted within the range of [0, Wgripper] to avoid collisions in clutter scenes.
(0, n, () are all constrained in 2 4]

---

Location
Heatmap
GHM
Patch Center
Normalized Grasp Space
RNGNet
patch
Rotation
Noraaization
Heatmap
Adaptive
Grid Generator
Point-wise Gate (PG)
Regional
RGBD
Grasps
Scene Points
Scene Grasps
Figure 1: Region-aware Grasp Framework. With RGBD image input, Grasp Heatmap Model (GHM) [15] predicts grasp location heatmap. Subsequently, graspable patches are extracted through an adaptive grid generator, which samples mesh grids based on the location heatmap and the gripper scale. Local patches are then converted into Normalized Grasp Space and fed into the Regional Normalized Grasp Network predict regional rotation heatmaps and grasps. Finally, we apply the inverse of the normalization process and obtain the scene-level grasps.
3.1 Normalized Grasp Space
Depth-Adaptive Patch Extraction: As shown in Fig. 2(A)(B), we utilize the RGBD image E RHxWxl and patch centers p; = (Iy Mb: 2,) sampled from grasp location heatmap predicted by [15] as input, aiming acquiring corresponding patches Praw e 8888x4 To better leverage geometric features and avoid camera viewpoint variations, rather than using fixed patch size, we design an adaptive grid generator that effi- ciently generates sampling grids with different sizes centered at the centers. Inspired by the setting in [21], the principle to generate adaptive receptive fields for different patches and keep the receptive fields invari- ant real-world distances. Thus, the patch sizes can be formulated as:
Raw Patches
Deptradaptive Patches
Nomelizdd Region Patches
Normalized
Grasps
Grap Ungper
Figure 2: Depth-Adaptive Patch Extrac- tion and grasp representation.
Wref F
(1)
where r; is the patch image side length for the 7th patch, 2; is the depth to the 7th center the camera frame, and is the camera focal length. noteworthy that Wref is the reference receptive field size the real world. ensure the local patch size is adequate infer grasps and avoid potential collisions, fref is
Wgripper- Then the regular spatial grids will be rescaled and applied to the input RGBD image, resulting in the depth-adaptive local patches.
Space Normalization: Normalized Space has been proved helpful for neural network learning. During Space Normalization, our goal to acquire a more structural and aligned representation among patches and regional grasps. Thus, firstly we convert the 4-channel raw RGBD image patches into the 6-channel RGBXYZ form P: Sx5x6 using the camera intrinsics and imaging model, in which XYZ denotes the 3D coordinate maps. For clarity, because RGB channels require extra normalization, we omit the RGB channels and only consider the XYZ channels in the following discussion. Then as is shown in Fig. 2(C), the normalized region patch P, can be obtained with the patch centers and the given reference receptive field Wref as below:
P. P
p
Wref

---

Raw Image
Translation(Af)
Scale(a, (a 4z)
Rotation, (A8)
Raw
Patches
& Grasps
GgT
Normalized
RA00
Patches
Grasps
(z.y,2.08B.y.w))
(A)) = (x,y,z,0 A0.B.y,w) (x..
Figure 3: Visualization for the characteristics of Normalized Grasp Space.
Notably, input normalization is only part of the Normalization Grasp Space. We filter and normalize the grasp labels inside P Our 6-DoF grasp pose representation = (I. y, 2, 0, 7, B, w) is equiva- lent to the representation in the SE(3) space g = (t, R. w), where t E R3x1 denotes the translation, R R3x3 is the rotation matrix in the camera frame, w denotes the grasp width. Thus, can be represented Euler angles: R R. (3)) R,,)).. where (g) represents the rotation radians around the z-axis and similar notations apply for the other rotations.
We limit our scope to grasp labels located within a spherical boundary of 0.1 * Wref centered at P;- Such distance boundary is consistent with the widely adopted grasp coverage criterion in [22, 4, 15], in which two grasps are regarded as equivalent if their center distance is less than a threshold. The grasps, denoted G {g where G. the set of normalized grasp labels g):
9 = (t), Ri. w5) = 1 P
W
R
Wref
Wref
G = [ || < 0.1).
(3)
As Fig. 2(D) indicates, region-aware framework only focuses the grasps near the patch centers, making the problem statement clearer and more concentrated. Then, for the regional 6-DoF grasp detection problem, the defined grasp function is f P; + G; Our goal is learn network fx P G with parameters À fit By inverting the normalizing process, can finally get the grasp predictions G; for this patch in the camera frame.
Characteristics of Normalized Grasp Space: As shown in Fig. 3, constraining the grasp detection problem in the Normalized Grasp Space brings beneficial characteristics as follows. For clarity, we omit the patch index and denotes grasp labels within the transformed patches as G.y.
1) Translation-invarinncs When the patch P: translates with At, we get the normalized patch P and the grasps inside the patch keep invariant:
Gi f = + At)) ((Norm())) f(P") G
2) SE(() rrotation invariance for (3,7) and equivariance for 6: When the patch P, rotates along the z-axis in the camera frame with A6, the grasps inside the patch rotates equivariantly:
RG = (A R. R.
(3) R (
(5)
3) Scale-invariance: With the coordinates (XYZ) of the patches or the scene scaled by a factor E R+ we get the same normalized patch P and the grasps inside the patch keep invariant if the receptive field for normalization is scaled by the same factor a:
G f(P7) (Norm((PP Plwrer = aw))) f( a(P P:)
1(P))
(6)
awo

---

Thus, based on the characteristics above, the latter trained grasp function fx to fit the ground truth grasp function f is able to generate consistent results across scenes and grippers different scales.
Above, we consider the ideal scenario for applying Normalized Grasp Space. However, in the real world, noise from depth sensors is non-negligible and may cause un- stable patch extraction. Furthermore, the gripper sizes in grasp detection datasets are usually fixed, which means that though we train our model through a normalized scheme, it senses the regions with a fixed receptive field. Thus, we add randomization to the receptive field Wree during training. The receptive field randomly selected in the range [Weef 2 Wref], which can improve the robustness and generalization capability of our grasp network.
Patch Scale Randomization:
3.2 Regional Normalized Grasp Network
Region-aware Patch Feature Extraction: To ef- ficiently detect grasps in regions, our approach ex- tracts patch-wise features in a region-aware man-
ner. The XYZ coordinates of the patches are eru- cial for understanding object contact and collision. Hence, it is imperative incorporate the over- all 3D XYZ coordinates instead of only the depth channel for feature extraction. For geometric fea- ture extraction from point-wise coordinates, previ- ous methods utilize shared MLP (Multi-layer Per- ceptron) combined with sampling and grouping [23, 12, 24], which is effective but introduces a significant computation overhead. Thus, combin- ing the advantages of point-wise shared MLPs and CNNs, depicted in Fig. 4, we design an efficient network to process the normalized patches.
RGB
Comm
basthack
PG
ResBlock
Point-wise Gate (PG)
Summation
ROLL
Come
ReLu
Com
DownSample
Sigmaid
Figure 4: Detailed structure of RNGNet.
Inspired by the gated CNN in [25, 26]. we establish a multi-stage Gated CNN with a novel Point- wise Gate (PG) Module. Point-wise Gate Module is built with shared MLP operating 3D point coordinate XYZ maps, increasing the dimensions of XYZ maps in different resolutions to obtain spatial feature gate values in each convolution stage. By multiplying the original feature maps with the gate values, Point-wise Gate enables selective amplification or attenuation of filters within convolution layers.
Rotation Heatmap Predictor: Based on the regional machor-based grasp generator in [3, 6, 15], with the region-aware features extracted, we predefine series of non-uniform rotation anchors for (7,8) generated via the Anchor Shifting algorithm in [15], whose Cartesian product forms 2D rotation heatmap (one axis for y, the other of 33. Then, the grasp detection can be considered a two- class semantic segmentation problem, while the normalized grasp widths and extra center offsets are regressed for each possible rotation (pixel in the heatmap).
4 Dataset Experiments
Dataset and Metrics:
GraspNet- -oapNNet-BBiliionn BBillion [3] a large-scale grasp dataset widely adopted in recent 6-DoF grasp detection research [9, 15, 8, 20], which provides large-scale training data and standard experiments, we follow the
evaluation platform for the task of general robotic grasping. During official evaluation pipeline and code of the GraspNet- Billlion dataset, in which detected grasp poses are first filtered with non-maximum suppression, and then the top 50 grasp poses are evaluated with force-closure [28] metrics under a series of friction coefficients condition.
1
Quantitative Results:
We compare the overall performance of our RNGNet with multiple typical grasp detection algorithms trained on the same dataset training split. As Table indicates, RNGNet achieves excellent performance, reaching average 58. 06....4/2224 AP by RealSense/Kinect cameras.
5

---

Method
Average
Seen 1
Similar 1
Novel f
Paras 1
Time /ms
GPD [10]
17.48 19.05
22.87 24.38
21.33/ 23.18
8.24 9958
PointneGGPD [1]]
19.29 20.88
25.96 / 27.59
22.68 24.38
9.23/ 10.66
GraspNet-hasliine [3]
21.41 / 23.08
27.56 29.88
26.11 27.884
10.55 / 11.51
126
TransGrasp [27]
27.65 25.70
39.81 / 35.97
29.32/ 29.71
13.83 11.41
HGGD! [15)
42.79 39.79
58.35 / 56.85
47.93/43.95
22.10/ 18.59
3.42M
34
Scale Balanced Grasp [8]
44.85
58.95 /
$2.97
22.63/
2421
GSNet [6]
47.92 42.53
65.70 / 61.19
53.75 47.39
24.31 / 19.01
15.4M
61
AnyGrasp w/ CD [9]
49.01
66.
56.09 /
24.81
24.7M
1981
RNGNNet
58.06
75.20 72.23
666.2/ 58.43
32.38 26.05
3.66M
RNGNet w/ CD
59.13 99.11
76.28 / 72.89 68.26 59,42
32.84/ 26.12
3.66M
20
"-"; Result Unavailable; CD:Collisin Detection, the grasp post- proccesscsiing algorithm proposed in [3] and utilized by [9]. Evaluated with batch_siz size = 1 on Ubuntu20.04 with AMD 5600x CPU and single NVIDIA RTX 3060Ti GPU.
Reimplemented or tested with the provided codebases. More detailed results are shown in Table 7 of the Appendices.
Table I: Results on GraspNet dataset. Showing APs RealSense/Kineet split and model efficiency.
boco)
Sexae (aniba))
(eard)
KNGNO
60
Average AP
HGGD
30
AnyGroup
20
HGGD
68N
12
1.0
0.8
0.5
0.4
02
Friction Coefficient
Figure 5: Qualitative results covering seen/similar/novve test Figure 6: Performance (Average set. Top 50 grasps after grasp-NMS [3] are displayed. Color AP) curve. Tested under different implies the predicted grasp confidence (red: high, blue: low). difficulties (friction coefficient).
Compared with other methods, RNGNet outperforms current ofate-of-the-art by a large margin on all dataset splits, especially on the unseen (similar and novel) splits, proving the efficacy and general- izability of our proposed region-aware grasp framework. Furthermore, considering the computation cost for one shot of grasp detection, method achieves the fastest speed with the best grasp de- tection quality, which implies the potential of our approach to be applied in dynamic scenes or on widely used compputation-rssuurcc-ooossttttt
platforms in robot deployment.
As shown in Fig. 6, we conduct grasp evaluation using the force-closure metric under different difficulties with different object friction coefficients. With lower object friction, the object is more likely to drop from the gripper and thus requires more high-quality grasp detection results. It can be clearly seen in the figure that RNGNet surpasses former SOTAs by a large margin with different object friction coefficients, especially for the settings with lower friction coefficients.
As is indicated Table 2, following the setting in [8], we evaluate the grasp quality at scales (S: Small, M: Medium, L: Large). RNGNet significantly improves performance with all the metrics, especially in the small-scale grasps and the novel split, proving the proposed NGS's effectiveness. RNGNet can learn robust and generalizable region features across different scales with NGS.
Seen
Similar
Novel
Method
APs
APM m
AP:
APs
AP
AP,
APs
AP
APr
Scale Balanced Grasp [8]
13.47
48.12
61.81
6.23
37.90
53.89
7.60
17.04
23.10
Scale Balanced Grasp OBS [8]
18.29
52.60
64.34
10.03
42.77
57.09
9.29
18.74
24.36
RNGNet
22.94
64.06
68.50
19.34
59.55
64.58
21.51
29.84
28.57
Table 2: Multi-scale results. Grasps with different scales are tested separately on RealSense split. OBS: Object Balanced Sampling proposed in [8] which requires extra instance segmentation.

---

Components
Seen
Similar
Novel
Norm
Adapt
Rand
Components
Seen
Similar
Novel
66.71
51,66
26.97
Z
XY
RGB
PG
69.93
53.15
28.71
72.68
61.49
28.31
73.98
64.89
30.99
4
73.76
63.43
30.15
75.20
66.62
32.38
74.31
65.41
31.72
Norm: Patch Normalization
75.20
66.62
32.38
Adapt: Depth-adaptive Patch Extraction
PG: Point-wise Gate Module
Rand: Scale Randomization
Table 3: Normalized Grasp Space ablation.
Table 4: Input modality and network ablation.
Tested on Realsense Split.
Tested on Realsense Split.
Qualitative Results: We also conduct grasp detection visualization with HGGD [15] and GSNet
[6] on three parts of the test set (seen: scene scene100-125) 100-129, similar: scene 130-159, novel: scene scene160-1800 180- As depicted in Fig. 5, RNGNet exhibits superior grasp detection quality and achieves a higher grasp coverage rate in cluttered scenes than prior works, even in scenarios involving occlusion or partial observation. This enhancement provides the robot with more feasible action alternatives, benefiting scene-level grasp planning and execution. Furthermore, when compared with [15, 9], it is note- worthy that our proposed region-aware framework enables the generation of grasps better aligned with object centers and surfaces, consequently enhancing grasp stability. The overall qualitative re- sults underscore the validity of our representation and approach in cluttered scenes, offering more comprchensive perspective.
Normalized Grasp Space Ablation: We perform ablation studies on our proposed Normalized Grasp Space, which primarily consists of Patch Normalization, Depth-adaptive Patch Extraction, and Scale Randomization. The results Table 3 illustrate the effects of these three components on Normalized Grasp Space. As expected, both Patch Normalization and Depth-adaptive Patch Extraction significantly enhance performance across all scenes, proving the effectiveness of the novel viewpoint about generating grasps in the region-aware and grasp-centric spaces. Additionally, Scale Randomization also boosts the performance of our methods on the dataset.
Region Normalized Grasp Network Ablation: Table presents the ablation experiments con- ducted on the RNGNet with different input modalities. Remarkably, our method achieves a com- mendable performance even with only depth maps as input. Introducing XY maps RGB images supplementary inputs leads to considerable improvements. While geometric information from Z maps plays crucial role in 6-DoF grasp detection, the additional positional information from XY maps and the inclusion of colors contribute to enhanced feature extraction and are especially benefi- cial for generalization to previously unseen (similar and novel) scenes. Finally, applying Point-wise Gate Module to feature maps in all stages yields further improvements across all test splits, with a negligible increase in the number of parameters (approximately 36k, or about 1% of the total pa- rameters). These findings support the efficacy of our approach, which built with combination of point-based networks and CNNs.
Realworld Experiments
Cluttered Scene Clearance: To conduct a comprehensive comparison of grasp detection methods in real-world cluttered scenes, we constructed multiple cluttered scenes consisting of randomly placed objects within 50cm 50cm workspace, including novel objects, object stacking and occlusion, shown in Fig. We deployed multiple representative grasp detection methods [15, 9] on real-world robot platform. Subsequently, akin the metrics utilized in [15], we assessed the overall Success Rate and Completion Rate to facilitate overall comparison.
Compared with other methods, results presented in Table demonstrate the generalization capability to real-world grasping of our framework. However, is worth noting that HGGD struggles to generate grasps for objects with small scales, and AnyGrasp exhibits imprecise rotation predictions, resulting in unstable grasping and frequent drops. particularly evident when objects are stacked. Both methods are prone to failure when the captured point cloud becomes more noisy only partial
7

---

Scene
Objects
RNGNet
HGGD
AnyGrasp
8
8 8/9
71
8/10
URBe
7
7/9
7/9
7/9
9
9/10
9/11
9/10
6
6/6
6/6
6/6
50cmn Workssade
7
7/8
7/9
7/9
8
819
8/10
8/10
9
9/9
9/11
9/11
Success Rate
90 / 60)
82%((3 64)
83%(55 65)
Completion Rate
100%(7/7)
86%(6/7)
100%
Table 5: Real-world clutter clearance results. Robot performs grasp detection and executes grasping for each until grasp is de- tected or 15 attempts are tried." means failed to grasp all the objects.
Figure 7: Robot settings
and object samples.
Setting
Success Rate
82.5
Handover
(33 / 40)
67.5%
Conveyor
(27 / /00)
(a) Handover
(b) Conveyor
Figure Real-world closed-loop experiment settings.
Table 6: Closed-loop results.
object observations available. In contrast, RNGNet, with its robustness to noise originating from the depth sensor, demonstrates the ability to predict more stable grasps with precise rotations. This effectively reduces collisions and object drops, showcasing the potential of this method.
With RNGNet, we are able to detect grasps any region regardless of patch location the scene structure. This implies a practical downstream application potential when there is need to generate grasps across the scene. To evaluate the efficiency of framework in more challenging scenes, build two dynamic settings illustrated in Fig. 8: (a) Human-to- robot handover, (b) Moving object on the conveyor. Experiment results in Table 6 confirm that our framework can accomplish dynamic grasping at a satisfactory success rate by detecting high-quality grasps a real-time frame rate and executing the closed- controlling. The proposed closed-loop grasping pipeline can effectively handle more complex nontable-top grasping scenarios. Thus, our proposed method proves robust to much noisier environment, which underscores the potential of our method for addressing challenging dynamic scenarios. The closed-loop grasping algorithm is demonstrated in Algorithm. 1 of the Appendices, and more demonstrations of the dynamic grasping experiments can be found in the Supplementary Videos.
Closed-loop Grasping:
Limitation and Conclusion
Normalized Grasp Space (NGS), unified space for region-based grasp detection, is proposed to obtain consistent region representation. With the representation, we further build an efficient convolution- based Region Normalized Grasp Network (RNGNet) to extract features and predict ro- tation heatmap. Though efficient, our method, relying on single-view images, is vulnerable to noises and can be only applied with rigid bodies. We plan to explore integrating scene-level information from the multi-view image during the process of closed-loop grasping, which is critical for precise manipulation in complex scenes with specific instructions. More limitations are discussed in the Section C and D of the Appendices.

---

References
[T] Ni, W. Zhang, X. Zhu, and Q Cao. Pointnet++ grasping: Learning end-to-end spatial grasp generation algorithm from sparse point clouds. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.
[2] Y. Qin, R. Chen, H. Zhu, M. Song. J. Xu, and H. Su. S4g: Amodal single-view single-shot se (3) grasp detection in cluttered scenes. In Conference robot learning. PMLR, 2020.
[3] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Graspnet-bbilioon A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11444-114 11453, 2020.
[4] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[5] B. Zhao, Zhang, X. Lan, H. Wang. Z. Tian, and Zheng. Regnet: Region-based grasp network end-to-end grasp detection in point clouds. In 2021 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2021.
[6] C. Wang, H.-S. Fang, M. Gou, H. Fang. J. Gao, and C. Lu. Graspness discovery in clutters for fast and accurate grasp detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15964-15973, 2021.
[7] X. Liu, Y. Zhang, H. Cao, Shan, and Zhao. Joint segmentation and grasp pose detection with multi-modal feature fusion network. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 1751-1756. IEEE, 2023.
[8] H. Ma and D. Huang. Towards scale balanced 6-dof grasp detection in cluttered scenes. In Conference on Robot Learning, pages 2004-2013, PMLR, 2023.
[9] H.-S. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y. Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions Robotics, 2023.
[10] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research, 36(13-14):455-1477, 2017.
[11] H. Liang, X. Ma. S. Li, M. Görner, S. Tang. B. Fang. F. Sun, and J. Zhang. Pointnetgpd: Detecting grasp configurations from point sets. In 2079 International Conference on Robotics and Automation (ICRA). IEEE, 2019.
[12] C. R. Qi, L. Yi, H. Su, and J. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets metric space, Advances neural information processing systems, 30, 2017.
[13] C. Eppner, A. Mousavian, and D. Fox. Acronym: A large-scale grasp dataset based on sim- ulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[14] D. Morrison, P. Corke, and J. Leitner. Egad! evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. IEEE Robotics and Automation Letters, 5(3): 2020.
[15] S. Chen, W. Tang, P. Xie, W. Yang, and G. Wang. Efficient heatmap- guided 6-dof grasp detection in cluttered scenes. IEEE Robotics and Automation Letters, 2023.
[16] W. Tang, S. Chen, P. Xie, D. Hu, W. Yang, and G. Wang. Rethinking 6-dof grasp detection: A flexible framework for high-quality grasping. arXiv preprint arXiv:2403.1505. 2024.

---

[17] D. Park and S. Y. Chun. Classification based grasp detection using spatial transformer network. arXiv preprint Xrv:v: /803.01356, 2018.
[18] A. Gariépy, J.-C. Rucl, B. Chaib-Draa, and P. Giguère. Gq-stn: Optimizing one-shot grasp detection based on robustness classifier. In 2019 JEEE/RSJ International Conference on Intel- ligent Robots and Systems (IROS), pages 3996-4003. IEEE, 2019.
[19] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and R. Platt. Sample efficient grasp learning using equivariant models. arXiv preprint arXiv:2202.094668 2022.
[20] Z. Chen, Z. Liu, S. Xie, and W.-S. Zheng. Grasp region exploration for 7-dof robotic grasping cluttered scenes. 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3169-3175. IEEE, 2023.
[21] B. Kang, Y. Lee, and T. Q. Nguyen. Depth-adaptive deep neural network for semantic segmen- tation. IEEE Transactions Multimedia, 20(9):2478-244 2490, 2018.
[22] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2901-2910, 2019.
[23] C. R. Qi, H. Su, K. Mo, and J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[24] X. Ma, C. Qin, H. You, H. Ran, and Y. Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. arXiv preprint 22Xi::2202.07123: 2022.
[25] J. Chen, T. He, W. Zhuo, L. Ma, S. Ha, and S.-H. G. Chan. Tyconv: Efficient translation variant convolution for layout- aware visual processing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12548-12558, 2022.
[26] S. Howard, P. Norreys, and A. Döpp. Coordgate: Efficiently computing apatially-varying convolutions in convolutional neural networks. arXiv preprint arXiv:2401.04688. 2024.
[27] Z. Liu, Z. Chen, S. Xie, and W.-S. Zheng. Transgrasp: A multi-scale hierarchical point trans- former for 7-dof grasp detection. 2022 International Conference on Robotics and Automa- tion (ICRA). IEEE, 2022.
[28] Mahler, J. Liang, S. Niyaz, M. Laskey, Doan, X. Liu, A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv: / 1703.09312 2017.
[29] Z. Liu, Z. Chen, and W.-S. Zheng. Simulating complete points representations for single-view 6-dof grasp detection. IEEE Robotics and Automation Letters, 2024.
[30] C. Choy, J. Gwak, and S. Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 3075-3084, 2019.
[31] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environ- ment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[32] H. Zhang, D. Yang, H. Wang, B. Zhao, X. Lan, J. Ding, and N. Zheng. Regrad: large-scale relational grasp dataset for safe and object-specific robotic grasping in clutter. IEEE Robotics and Automation Letters, 7(2):2929-20) 2936, 2022.
10

---

[33] R. Qin, H. Ma, B. Gao, and D. Huang. Rgb-d grasp detection via depth guided learning with cross-modal attention. arXiv preprint arXiv:2302. 14264, 2023.
[34] D. Coleman, I. Sucan, S. Chitta, and N. Correll. Reducing the barrier entry of complex robotic software: a moveit! case study. arXiv preprint arXiv:1404.3785, 2014.
[35] Valle. Rapidly-exploring random trees: A new tool for path planning. Research Report 9811, 1998.
[36] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[37] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[38] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02633 2023.
[39] Liu, Z. Zeng, Ren, Li, H. Zhang. J. Yang, C. Li, J. Yang, Su, J. Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint 20X:v:3303005499 2023.

---

Appendices
A
Additional Results
Detailed Results on the GraspNet-IBillion Dataset: In addition to the overall grasp AP (Average Precision) metric, we also provide the evaluated grasp quality based on force-closure metric [28] with varying friction coefficients, denoting AP representing the average precision at given friction coefficient μ. As shown Table 7, our proposed RNGNet outperforms previous state-of-the- art methods in terms of all metrics under different friction conditions, especially the difficult scenarios (similar/novel splits and A.4.4 with lower friction coefficient).
Method
Sera
Similar
Novel
AP
AP.
AP.
AP
AF.
APb.a
AP
AP,
APr.
GPD [10)
22.87/24.38
26.53/3011 16
12.84/13.46
21,33/23.18
27.83/28.64 64
9.64/11.22
8.24/9.58
8.89/10.14
2.67/3.16
PointeeGiPD [1]]
25.96/27.59
33.01/34.21
15,37/17.83
22.08/24.38
29.15/3088
10.76/2288
9:23/10.66
9.89/11244
2.74/3.21
CrueNet-buelline [3]
27.56/2988 58
33.43/3.19 19
16.95/19.31
26.11/27.88
34.1833.11 19
14.23/16.62
10.55/11.51
11,25/12922
3.98/3.56
TransGrrupp
39.8/335997
47.54441.09 69
36,42/31.86
29.32/29.71
34,8035.00 67
25.19/34.19
13.82/11.41
12.11/14.42
7.6715.84
HGGGD' [15]
$8.35/56.85
66 66.54/54.00 60
55, 55.96/52.94
47.93/43.90 93
56.91/52.73 73
41.8836.8 88
22.10/18.59
27,37/22.96 98
14,31/11.91
Balanced Grasp
68.18/
54.88/
46.991
22.63/
GSNit
65.70661.19
76.2571146
53.75/47.39
65 04/50/6688
45. 45.97740:43
23,98/19.01
29. 29.93/3377
14,05/10.00
Anytitap o/ CID [9]
66.12
77,27/
61.02/
56.09/
68.29
47,317
24.817
31.00/
1588//
KNGNet
23.20/2.223
05.02/22.0
75.71/5.955
66.0258.43
78.5368.00
59.23/5187
32.3836605
40.48/52.2
19.8416.24
RNGNNet w/ CD
76.28/72.89
86.5X
72.2668.11
68.2659.42
90.6/7016
60.01/52.13
32.84/26.12
41,17/32.455
19.68/5.92
: Result Unavailable, CD: Collision Detection Reimplemented with official codebase without CD.
Table 7: Detailed results on GraspNet Dataset. Showing APs on RealSense/Kinect split under different friction coefficient.
Extra Real-world Results: For more detailed performance evaluation, we conduct extra real-world experiments at different noise levels to illustrate the practical benefits and robustness of our method. In the extra real-world experiments, for more challenging grasping, we introduce more novel objects (50 objects in total vs 24 used in the manuscript) in the experiments, and more objects (6 to 12 now vs 6 to in the manuscript) may be randomly placed in a clutter. Each method is tested with extra scenes under different settings. The other settings (grasp order, collision check, robot control, etc.) are kept the same as in HGGD [15] and the manuscript.
Noise Deviation (m)
Method
Attempt Success Rate
Scene Clearance Rate
AnyGrasp
/ = 77.6 %
5/6 = 83.3 %
RNGNet
53/ 60 88.3 %
6/6= 100 %
AnyGrasp
41 89 46.1 %
2/6 33.3
0.01
RNGNet
48 / 80 60.0 %
4/6 66.7
AnyGrasp
8/ 8.9 %
0/ 0 %
0.02
RNGNet
39 / 88 44.3 %
2/6 = 33.3
Table 8: Additional Real-world Clutter Clearance Experiments.
As is shown in the table above, our method achieves an impressive success rate in attempts without additional noise, efficiently clearing all scenes. In comparison, AnyGrasp exhibits lower attempt success and scene clearance rates and may struggle to clear scenes due to repeated failures specific objects. Our method, on the other hand, offers more viable grasps across scenes with multiple sampled patches, which benefit the scene clearance process. This is also illustrated in Figure 5 of the manuscript (AnyGrasp mainly uses GSNet for one-frame grasp detection).
As for grasp detection with noisy inputs, AnyGrasp shows significant decline in performance as noise increases, while our method performs more consistent grasp detection results in noisy envi- ronments and is capable of clearing scenes with relatively high noise levels (0.02 m). These results
12

---

demonstrate the superior robustness and reliability of our method across varying noise levels. Its ability to maintain high performance in the presence of noise makes a more reliable choice, espe- cially in environments where noise is a factor.
Real-world Ablations: One of the purposes of the proposed Normalized Grasp Space (NGS) is to better leverage geometric features and avoid camera viewpoint variations, keeping the receptive fields invariant to real-world camera positions. Thus, we conduct more extensive testing across various camera viewpoint to ablate the proposed NGS in the manuscript. We conduct robot grasp- ing with a variable camera viewpoint rather than a fixed camera position by changing the camera distance to the table. All other experiment settings are the same as above.
Camera Viewpoint
Method
Attempt Success Rate
Scene Clearance Rate
RNGNet w/o NGS
52 66 = 78.8
5/6 6 = 83.3 %
close: m (default)
RNGNet
53 60 = 88.3
6 100
RNGNet w/o NGS
48 / 79 = 60.8
4/ 6 = 66.7 %
medium: 0.6 to 0.8 m
RNGNet
53 62 = 85.5 %
6/ = 100 %
RNGNet w/o NGS
18 98/90-200 = 20.0 %
0/ 6 = 0 %
far: 0.8 to 1.0 m
RNGNet
53 65 81.5 %
6 6 100 %
Camera Viewpoint: camera distance to table surface
Attempt Success Rate = Successful Attempt Number / Total Attempt Number Scene Clearance Rate Cleared Scene Number / Total Scene Number
Table 9: Real-world Clutter Clearance Ablations proposed Normalized Grasp Space.
As the results show, RNGNet with NGS demonstrates superior robustness and adaptability across different camera viewpoints. The inclusion of NGS significantly enhances the method's ability to maintain high performance, even as the camera viewpoint moves further away. Without NGS, the performance drops considerably, especially at greater distances, making the NGS component crucial for reliable and consistent performance in varying conditions.
Backbone Ablation: To further prove the efficiency and robustness of our AD-convolution-base.
network, in Table 10, we replace the backbone with the widely used [6, 20, 29] high-dimensionaal sparse convolution backbone, MinkowskiFCNN [30]. As is shown in the results, our simple 2D convolution network is more efficient than the more complicated sparse convolution network in experiments, especially with high-resolution input. When facing low-resolution input, SparseCNN performs slightly better due to its particular design for sparse input. Significantly, both RNGNet and SparseCNN show quite consistent performance with input patches of different resolutions, which proves the robustness of our region-aware framework.
Backbone
Size
Scen
Similar
Novel
Time/ms
64 x 64
72.17
61.90
29.85
74
SparseCNN [30]
32 x 32
71.76
61.63
29.89
29
16 x 16
68.16
56.68
25.76
18
64
75.20
66.62
32.38
RNGNet
32
72.13
62.81
29.86
16
67.04
55.35
25.63
Table 10: Backbone ablation. Showing APs on Realsense split and network inference time.
Guidance Ablation: Though RNGNet utilizes the Grasp Heatmap Model as part of the patch extractor by default. However, we only introduce the Grasp Heatmap Model for locating graspable regions, and our model can be easily used without the Grasp Heatmap Model. To better investigate the ability of method to detect high-quality grasp in regions, compare the performance of method under different settings (with or without Grasp Heatmap Model). Without the Grasp
13

---

Heatmap Model, we filter the table pointeloud by z coordinates and use farthest sampling on the foreground pointeloud to acquire region centers. As shown in the table below, even without the guidance the Grasp Heatmap Model, our method can still perform high-quality grasp detection relatively slower speed (more patches are processed).
Localization
Seen
Similar
Novel
Time' (ms)
Heatmap 75.20
66.62
32.38
17
Foreground 74.75
66.06
33.59
30
Same settings as in Table 1.
Table 11: Ablation Experiments of Patch Localization Algorithm. Showing APs on Realsense split.
Grasp Inference with Multi-scale Grippers: As clarified in Section. 3.1, the Normalized Grasp Space (NGS) demonstrates scale invariance. In other words, our grasp detection model can be easily scaled to accommodate different range from the fixed gripper width defined in the training dataset without any extra training. Fig. 9, we illustrate this by enlarging the gripper scale from Wgripper 0.05 m to Wggripper = 0.2 m, showcasing grasps in cluttered scene. These results affirm that our framework can seamlessly adapt grippers of varying sizes. As is shown in Fig. 10, extend the application of framework much larger indoor scene than the typical table- top settings. The proposed RNGNet successfully generates large-scale grasps for large objects, validating the effectiveness of the Normalized Grasp Space.
To further investigate the effectiveness our Normalized Grasping Space for object grasping with grippers of different scales, We build scenes with objects with significantly different scales in the Sapien [31] simulator and conduct grasp detection and robot grasping. Experiments prove that Normalized Grasping Space can generalize well to grippers with different scales and has the potential to apply to industrial scenes, in which data and labels are hard to obtain. Detailed results and visualization can be found in our Supplementary Video.
(a) Wgrtyper
Mirispur
(c) Wyrtoper=0.1mm
[d] #0.15m
(e) Wirrtpper 2.2m
Figure 9: Visualizations of grasp predictions of different gripper scales. Different colors mean different gripper widths. From (a) to (e), as the gripper scale enlarges, it shows adaptive grasp predictions under different configurations.
B Implemetation Details
Normalized Patch Dataset Preparation: Currently, most grasp datasets focus on providing object- level grasp labels [13, 14] or scene-level grasp label [3, 32], which are not directly suitable for our CraspNet-IBillion 1Billion offers a large number of RGBD images and grasps annotations, based on the proposed Normalized Grasp Space, we need construct patch-based grasp dataset providing normalized patches and grasps annotations. As is shown in Fig. 11, utilizing the scene-level annotations provided in the GraspNet- -esspNeet-Biilioon dataset, gener- ate new patch-based grasp dataset for Regional Normalized Grasp Network training. Firstly
normalized scheme training. Although GraspNet-
14

---

(a)
(b)
(c)
Figure 10: Grasp detection in a room-scale scene. (a) Target RGB image and depth image of an ordinary indoor scene with sofa placed. (b) Extracted patch RGB and normalized image. (c) Detected grasps of the sofa.
obtain the object segmentation mask in the original dataset and adopt Gaussian kernel to dilate the mask image for covering possible graspable regions much possible. Then fixed number of patch center candidates are randomly sampled on the dilated mask image and cropped. Also, as is mentioned in [33], depth information is important to generate collision-fre grasp detection results. Thus, we add domain randomization to the patch center depth on the RGBD image planes, which provides patch-based dataset covering more the possible circumstances and improves the robustness of our training method.
Random Sample
Objert Mask
Centers
Patch Extraction
Scene-level Annotations
Distance Filter
Putch-level Annotations
Figure 11: Pipeline of Normalized Patch Dataset Preparation.
Grasping Order and Execution in Clutters: In cluttered scene clearance experiments, mainly adopt the perception-plamning plaannnn-grrasping open-loop scheme. For each grasp attempt, firstly, the robot moves to the initial home pose and captures the single-view RGBD image, from which multiple normalized grasp regions are extracted. Then, grasps are detected in each region to compose the scene-leve grasps for later grasp execution. During the grasping part, the grasp with the highest score is selected as the current target pose. We utilize the Movelt [34] framework to plan the robot's trajectory and execute the grasp poses using the UR5e equipped with the Robotiq gripper. RRT [35] serves as path planner plan feasible trajectory the target pose and execute the final grasp action. To avoid potential collisions and unsafe planning trajectories, Movelt's 's collision box construction tool is employed to define collision geometry. In case of failed grasp execution resulting from collisions or inaccurate grasps, the scene will be restored, and the objects will be re-placed.
Target-oriented Grasping: Actually, by utilizing the proposed region-aware grasp framework, we successfully decouple the grasp location problem and grasp detection problem. As state in the able to detect grasps in any
closed-loop grasping section in the manuscript, with RNGNet, region regardless of the patch location the scene structure. This implies practical downstream
15

---

application potential when there is no need to generate grasps across the scene. In short, Region- aware Grasp Framework allows a more flexible patch localization for different purposes.
Closed-loop Grasping Pipeline: As demostrated in the Algorithm. 1, we integrate grasp detection with simple closed-loop robot controlling and keep tracking the real-time grasp detection results to achieve closed-loop grasping in dynamic scenes.
Algorithm Closed-loop Grasping in Dynamic Scenes
Inputs:
M workspace workspace mask
min distance threshold
Python-Style Pseudocode:
1: ## Tracking phrase
2: while dist(grasps)) -poee, robot.pose) > min_dis do
3: image = get Ret.RRBB.Lnnt Reg.RGiiamema)
4: centers = uniform uniform-sample/ Mworkspace)
# sample patch centers within the mask
5:
patches = normalize(pan (patch extract(centern image))
# extract and normalize patch regions
6: grasps = RNGNet (patches)
# detect grasps for objects in the workspace
7:
grasps.sort.by bysoorrryooooe( sebot.set.speed_grges (0].pose- - robot.pose)
8:
# robot moving towards the best grasp pose
9: end while
10:
## Grasping phrase
11: move pose (graspss))) (00ppsssssppep)) pose)
robot.grasp(
# move robot to the final grasp pose and grasp
Hyperparameters and Loss: In the patch extraction, extract normalized patches with size S x S = 64 64 by default. We process the input image the same size 640 360 and adopt the pre-trained Grasp Heatmap Model checkpoint in [15] for fair performance comparison. In the feature extraction part, we directly adopt the ResNet-18 18 [36] with half width as the mainstream of our backbone in RNGNet. For the implementation of the Point-wise Gate module, a 3-layer shared MLP with the same width as the corresponding feature maps is adopted to generate gate value for feature maps of each resolution. our rotation heatmap predictor, the settings about rotation anchor are kept unchanged from those in the [15]. AdamW optimizer is adopted to train our RNGNet.
The training loss of RNGNet is the weighted sum of the regression losses and the anchor classifica- tion losses:
L Lo Lo + Lyg + L + Lw:
where Logete and T.g are the losses for Fuler angle classification, calculated using focal loss [37] and La.reg- L, and Law are the losses for 0 refinement, translation regression and width regression, calculated using smoothed LI loss.
C Discussion
Data Efficiency: To further confirm the data efficiency of our framework, we conduct extra experi- ments by training methods with different portions of data. For fair comparisons, we test our method without Grasp Hcatmap Model (pretrained Grasp Heatmap Model from HGGD uses all the training data and may lead to higher test results), that is, using the farthest sampling in the filtered foreground pointcloud extract local regions. As shown in the Table 12 below, method achieves bet- ter data efficiency than other baselines. Even with only 10% training data, method performs competitive grasp detection performance.
16

---

Method
1/50' scenes
1/20' scenes
1/10' scenes
Full scenes
GSNet
8.64
30.65
36.15
47.81
AnyGrasp
7F
351
49.01
RNGNet w/o Heatmap
25.60
41.61
47.43
58.13
I/XX scenes" means annotations from X scenes are used out of total 100 training scenes. Approximate results from Fig. (Evaluation results the GraspNet- Billion test when training with different portions of real-world data) in the AnyGrasp paper[9].
Table 12: Experiments of Method Data Efficiency. Showing APs on all Realsense test split with different portions of training data.
Equivariant Neural Networks: The proposed RNGNet is not inherently equivariant. We have incorporated rotation augmentation techniques to reduce the burden on the network to learn equiv- ariance from scratch, thereby improving generalization and reducing the amount of training data required. Experiments above also show the efficacy of our proposed Region-aware Grasp Frame- work for offering better sample efficiency than former methods. In the future, to further improve the performance and efficiency of our approach, we may use equivariant networks in our framework.
Introduction of Scene Priors: Actually, our approach is designed to be easily integrated with widely adopted preprocessing modules to enhance results and extend functionality. Such shape completion and instance segmentations can serve as the data processing module for the input of our Region Normalized Grasp Network. Shape completion enhances object surface information, contributing collision-free and stable grasp detection, while instance segmentation can provide valuable object pose information for more precise grasp detection. practice, we have also ex- plored integrating SAM (Segment Anything Model) [38] the instance segmentation model and Grounding Dino [39] as the object detection model, enabling our method to address target-oriented and instruction instruction-guided grasping tasks seamlessly.
D Limitations
Although our proposed region-aware framework demonstrates excellent performance and flexibilityy across diverse grasping tasks, several limitations remain. Primarily, our method relies on single-view RGBD images, which are vulnerable to degradation from severe noise and occlusion. Secondly, our current design focuses exclusively on common rigid bodies, and does not address the challenges posed by transparent, reflective, or deformable objects. Lastly, while our closed-loop algorithm addresses grasping tasks in some dynamic scenes, online grasp generation and selection methods could be further optimized. For example, integrating it with reinforcement learning could enhance adaptability, facilitating smoother task execution.
17


Reviews:
[{"review_id": "25KPtfRI52", "reviewer": "Reviewer_GvsS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to normalize SE(3) grasps in table-top scenes based on local regions and uses a learning-based method to generate normalized grasps based on local features. Specifically, a heatmap indicating potential grasps is generated using a prior method [15], and the candidate regions are sampled from the heatmap and further normalized based on the gripper size, distance to the camera, and focal length of the camera. The RGBXYZ image of each candidate region is fed to a neural network to predict grasp rotations, widths, and translational offsets.", "summary_of_recommendation": "This paper shows good quantitative improvements over existing methods. However, leveraging local geometric regions has been widely adopted. The technical novelty needs to be further discussed.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Besides the concerns above I hope the authors can address, I have the following questions:\n1. I hope the authors can clarify why the method has equivariance for theta because the patch seems to be a square which is not symmetric with respect to the z-axis. \n2. In Table 5, why are the different methods tested on different numbers of trials?\n3. Why are grasping methods that leverage shape completion, equivariant neural networks, and instance segmentations not discussed or compared against? They are also important for grasping from cluttered table-top scenes.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qiSOZoJZ2d", "reviewer": "Reviewer_aAbj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper identifies an inherent shortcoming of existing point cloud-based grasp pose detection neural networks, namely that the area of interest can vary in the presence of clutter. It proposes a Normalized Grasp Space to represent regions of interest and a Regional Normalized Grasp Network that operates on this space. RNGNet is trained on the GraspNet-1Billion dataset, and achieves state-of-the-art performance, both in precision and runtime on the evaluation set. It also outperforms baseline systems when evaluated on a real robot, and achieves high performance in more difficult dynamic tasks.", "summary_of_recommendation": "RNGNet's network design is based on an analysis of the drawbacks of existing systems. It achieves a new state-of-the-art on the GraspNet dataset in grasp detection precision while outperforming other baselines at detection speed. It is also shown to be more effective than several baseline systems in real robot experiments. The exhaustive ablation studies support the design decisions.", "technical_quality": 5, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "The only suggestion I have for improving this paper is to cite Rethinking 6-Dof Grasp Detection: A Flexible Framework for High-Quality Grasping by Tang et al., that outperforms the other baselines but underperforms RNGNet.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RDrZYFpAZD", "reviewer": "Reviewer_yaiB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Authors address the problem of generating grasps for a gripper in a cluttered scene. The grasps are represented as a pose and a gripper width. The main novelty is in the way the RGBD input is processed. The authors \"normalize\" the patches in the input to be scale, translation, and rotation invariant. Ablations on a chosen dataset show that this normalization yields significant improvement in average precision (AP) over other methods. The authors validate the method on the real robot and show they can clear the cluttered scenes, as well as pick up a moving object due to fast inference time of the proposed method.", "summary_of_recommendation": "My recommendation is based on real world experiments showing strength of the proposed approach: compared to SoTA as well as ablations", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Can you provide real-world experiments/ablations for either of the proposed novelties to show that the novelties translate to improvement on the real robot, and if so, by how much? No AP, but PPM (picks per minute) or number of attempts to clear the table.\n\nFurther, in real-world experiments, can you show significant improvement over the AnyGrasp baseline?\n\nHow do you ensure the reachability of your grasps? How is this accounted for in the experiments?\n\nWhat are the real limitations of your method? When does your method mostly fail?\n\nEDIT: all questions are answered, thank you", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are in agreement that the paper addresses an important and relevant problem. They also agree that the experiments in simulation are thorough and the experiments show some improvement over the existing literature.

However, the specific novelty of the paper appears minimal to two of the reviewers. This coupled with the only minor increase in performance on real world tasks, makes it difficult to draw a strong conclusion from the paper. There's also some question as to how the equivariance stated as a contribution is ensured.

---Update Post Rebuttal---
Thanks to the authors and reviewers for a thorough discussion of the concerns above and in the initial reviews.

Based on the updated information the reviewers and I agree that we can now recommend the paper for acceptance.

As noted by one reviewer there is remaining concern about the lack of comparison with "grasping methods that leverage shape completion, equivalent neural networks, and instance segmentation."

We also strongly suggest the authors incorporate their answers form the rebuttal phase into the paper to improve the clarity of discussions on the contributions and the technical approach.
Decision: Accept