Region-aware Grasp Framework with Normalized Grasp Space for Efficient 6-DoF Grasping

Authors: siang  chen
pengwei  xie
wei  tang
dingchang  hu
yixiang  dai
guijin  wang
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 04:17:55

Abstract: A series of region-based methods succeed in extracting regional features and enhancing grasp detection quality. However, faced with a cluttered scene with potential collision, the definition of the grasp-relevant region stays inconsistent. In this paper, we propose Normalized Grasp Space (NGS) from a novel region-aware viewpoint, unifying the grasp representation within a normalized regional space and benefiting the generalizability of methods. Leveraging the NGS, we find that CNNs are underestimated for 3D feature extraction and 6-DoF grasp detection in clutter scenes and build a highly efficient Region-aware Normalized Grasp Network (RNGNet). Experiments on the public benchmark show that our method achieves significant >20 % performance gains while attaining a real-time inference speed of approximately 50 FPS. Real-world cluttered scene clearance experiments underscore the effectiveness of our method. Further, human-to-robot handover and dynamic object grasping experiments demonstrate the potential of our proposed method for closed-loop grasping in dynamic scenarios.

Main paper: 
 introduction: 
 Grasping, a fundamental task for robotic manipulation, has seen significant advancements. Notably, recent strides in deep-learning methodologies have paved the way for data-driven grasp detection techniques. These techniques, capable of directly deducing the positions and rotations of available grasps, offer superior generalization to novel scenes compared to the traditional template-based methods. Meanwhile, 6-DoF grasping has grown significantly because of its effectiveness in applications and scenarios that require accurate and reasonable manipulation of objects.Towards efficient 6-DoF grasping, most grasp detection methods treat the task as a detection-style problem involving the acquisition of optimal grasp positions and rotations from a single-view RGBD image or point cloud. Most of them process the global scene information to generate all possible grasps in the scene. Typical works like [1, 2] extract scene geometric features and regress grasps directly. The crucial problem is the redundancy and ambiguity of the entire scene information, which is time-consuming. Facing the challenge of grasping in cluttered scenes, recent methods [3, 4, 5, 6, 7, 8] encode global per-points features to infer potential grasp centers and segment graspable regions by integrating the regional points with the global features. While global-to-local schemes have demonstrated effectiveness, leveraging regional information without a consistent representation may undermine the generalization capability for novel scenes. The primary challenge lies in devising a method to extract regional information in a consistent representation.In this work, we propose a novel Region-aware Grasp Framework decoupling grasp detection into graspable region extraction and follow-up regional grasp generation. Specifically, we propose Nor-malized Grasp Space (NGS), a unified representation bridging the patch extraction and regional grasp detection. With extracted region information, the grasp detection problem within NGS can be formulated into a normalized form, which shows the invariance to the gripper size and the robustness to perturbations on the grasp center localization. Thus, considering grasp detection from a region-aware viewpoint, we propose Regional Normalized Grasp Network (RNGNet) to extract grasp-relevant geometric features and generate rotation heatmaps, demonstrating enhanced robustness and efficiency. Quantitative experiment results show a significant over 20% improvement in the grasp Average Precision (AP) on the public benchmark with 10× faster inference speed than the former SOTA, AnyGrasp [9] . Real-world clutter clearance experiments validate our framework's robustness and generalizability. Further demonstration of grasping in dynamic scenes proves the efficiency and flexibility of our proposed region-aware framework and the potential for more complex tasks like target-oriented manipulation. The contributions of the paper mainly include:1) A novel Region-aware Grasp Framework for 6-DoF grasp detection in cluttered scenes, decoupling grasp detection tasks into two-stage heatmap prediction and enhancing method efficiency.2) Normalized Grasp Space (NGS), a novel unified space defined for general parallel gripper grasp detection, empowering aligned and scale-invariant grasp detection in regions.3) A highly efficient Region Normalized Grasp Network (RNGNet) aiming at extracting regionaware features and predicting high-quality 6-DoF grasps via rotation heatmap prediction. 
related work: 
 Vision-based 6-DoF Grasp Detection: Research has focused on detecting 6-DoF grasps primarily relying on single-view RGBD images or point clouds. Pioneer works ten Pas et al. [10] , Liang et al. [11] mainly follow a sample-evaluation strategy, generating numerous grasp proposals and selecting optimal grasps through an evaluation model. Qin et al. [2] employ PointNet++ [12] for per-point feature extraction and direct regression of grasp parameters. Recent advancements, benefiting from large-scale datasets [3, 13, 14] , adopt an end-to-end strategy. Wang et al. [6] introduces pointwise graspness to represent grasp location probabilities and aggregate regional features for rotation prediction. HGGD [15] employs 2D CNNs to generate heatmaps for grasp locations and integrate 2D semantics with 3D geometric features to configure 6D grasps. Tang et al. [16] investigates scene regions for high-quality grasp detection but ignores helpful regional characteristics. Diverging from existing approaches, our method employs a region-aware scheme forcing the network to capture scene-independent regional features, enhancing efficiency and generalization capability.Region Normalization for Grasp Detection: Traditional 4-DoF grasp detection approaches [17, 18] conduct image-based normalization or crop the image patches with the exact sizes [19] to improve model performance. However, such 2D frameworks are limited, overlooking critical 3D geometric information and sensitive to camera viewpoints. For 6-DoF grasp detection, the utilization of regional 3D geometry is also not unique. However, most methods operate under a fixed gripper setting and extract local regions of predetermined sizes [2, 3, 6, 5] . These methods struggle to generalize or require model retraining when applied to grippers with different sizes or novel scenes. We normalize regions and corresponding grasps according to adjustable gripper sizes to obtain consistent 3D normalized grasp spaces, allowing more efficient grasp detection and adaptation for challenging applications, such as dynamic grasping. 
method: 
 Similar to prior works [15, 20, 3] , we focus on the problem of 6-DoF grasp detection of parallel grippers in clutter scenes from a single-view RGBD image χ ∈ R H×W ×4 as input. Inspired by [15] , we adopt the grasp representation as g = (x, y, z, θ, γ, β, w), in which (x, y, z) is the 3-DoF translations of grasp centers, w is the grasp width and (θ, γ, β) compose the 3-DoF intrinsic rotation as Euler angles. w is predicted within the range of [0, w gripper ] to avoid collisions in clutter scenes. With an RGBD image input, Grasp Heatmap Model (GHM) [15] predicts grasp location heatmap. Subsequently, graspable patches are extracted through an adaptive grid generator, which samples mesh grids based on the location heatmap and the gripper scale. Local patches are then converted into Normalized Grasp Space and fed into the Regional Normalized Grasp Network to predict regional rotation heatmaps and grasps. Finally, we apply the inverse of the normalization process and obtain the scene-level grasps. and patch centers p i = (x i , y i , z i ) sampled from grasp location heatmap predicted by [15] as input, aiming at acquiring corresponding patches P raw i ∈ R S×S×4 . To better leverage geometric features and avoid camera viewpoint variations, rather than using a fixed patch size, we design an adaptive grid generator that efficiently generates sampling grids with different sizes centered at the centers. Inspired by the setting in [21] , the principle is to generate adaptive receptive fields for different patches and keep the receptive fields invariant to real-world distances. Thus, the patch sizes can be formulated as:(θ, γ, β) are all constrained in [-π 2 , π 2 ].EQUATIONwhere r i is the patch image side length for the i th patch, z i is the depth to the i th center in the camera frame, and F is the camera focal length. It is noteworthy that w ref is the reference receptive field size in the real world. To ensure the local patch size is adequate to infer grasps and avoid potential collisions, w ref is set to 2 * w gripper . Then the regular spatial grids will be rescaled and applied to the input RGBD image, resulting in the depth-adaptive local patches.Space Normalization: Normalized Space has been proved helpful for neural network learning. During Space Normalization, our goal is to acquire a more structural and aligned representation among patches and regional grasps. Thus, firstly we convert the 4-channel raw RGBD image patches into the 6-channel RGBXYZ form P i ∈ R S×S×6 using the camera intrinsics and imaging model, in which XYZ denotes the 3D coordinate maps. For clarity, because RGB channels require no extra normalization, we omit the RGB channels and only consider the XYZ channels in the following discussion. Then as is shown in Fig. 2 (C), the normalized region patch P * i can be obtained with the patch centers p i and the given reference receptive field w ref as below:EQUATION)Figure 3 : Visualization for the characteristics of Normalized Grasp Space.Notably, input normalization is only part of the Normalization Grasp Space. We filter and normalize the grasp labels inside P * i . Our 6-DoF grasp pose representation g = (x, y, z, θ, γ, β, w) is equivalent to the representation in the SE(3) space g = (t, R, w), where t ∈ R 3×1 denotes the translation, R ∈ R 3×3 is the rotation matrix in the camera frame, w denotes the grasp width. Thus, R can be represented as Euler angles: R = R z (θ)R x (β)R y (γ), where R z (θ) represents the rotation of θ radians around the z-axis and similar notations apply for the other two rotations.We limit our scope to grasp labels located within a spherical boundary of 0.1 * w ref centered at p i . Such distance boundary is consistent with the widely adopted grasp coverage criterion in [22, 4, 15] , in which two grasps are regarded as equivalent if their center distance is less than a threshold. The grasps, denoted asG * i = {g * j },where G * i is the set of normalized grasp labels g * j :EQUATIONAs Fig. 2 (D) indicates, our region-aware framework only focuses on the grasps near the patch centers, making the problem statement clearer and more concentrated. Then, for the regional 6-DoF grasp detection problem, the defined grasp function is f :P * i → G * i .Our goal is learn a network fλ : P * i → Ĝ * i with parameters λ to fit f . By inverting the normalizing process, we can finally get the grasp predictions Ĝi for this patch in the camera frame.As shown in Fig. 3 , constraining the grasp detection problem in the Normalized Grasp Space brings beneficial characteristics as follows. For clarity, we omit the patch index i and denotes the grasp labels within the transformed patches as G * T . 1) Translation-invariance: When the patch P i translates with ∆t, we get the same normalized patch P * i and the grasps inside the patch keep invariant:EQUATION2) SE(2)-rotation invariance for (β, γ) and equivariance for θ: When the patch P i rotates along the z-axis in the camera frame with ∆θ, the grasps inside the patch rotates equivariantly:EQUATION3) Scale-invariance: With the coordinates (XYZ) of the patches or the scene scaled by a factor a ∈ R + , we get the same normalized patch P * i and the grasps inside the patch keep invariant if the receptive field for normalization is scaled by the same factor a:EQUATIONThus, based on the characteristics above, the latter trained grasp function fλ to fit the ground truth grasp function f is able to generate consistent results across scenes and grippers in different scales.Patch Scale Randomization: Above, we consider the ideal scenario for applying Normalized Grasp Space. However, in the real world, noise from depth sensors is non-negligible and may cause unstable patch extraction. Furthermore, the gripper sizes in grasp detection datasets are usually fixed, which means that though we train our model through a normalized scheme, it senses the regions with a fixed receptive field. Thus, we add randomization to the receptive field w ref during training. The receptive field is randomly selected in the range [w ref 2 , 2 * w ref ],which can improve the robustness and generalization capability of our grasp network. Region-aware Patch Feature Extraction: To efficiently detect grasps in regions, our approach extracts patch-wise features in a region-aware manner. The XYZ coordinates of the patches are crucial for understanding object contact and collision. Hence, it is imperative to incorporate the overall 3D XYZ coordinates instead of only the depth channel for feature extraction. For geometric feature extraction from point-wise coordinates, previous methods utilize shared MLP (Multi-layer Perceptron) combined with sampling and grouping [23, 12, 24] , which is effective but introduces a significant computation overhead. Thus, combining the advantages of point-wise shared MLPs and CNNs, as depicted in Fig. 4 , we design an efficient network to process the normalized patches.Inspired by the gated CNN in [25, 26] , we establish a multi-stage Gated CNN with a novel Pointwise Gate (PG) Module. Point-wise Gate Module is built with shared MLP operating on 3D point coordinate XYZ maps, increasing the dimensions of XYZ maps in different resolutions to obtain spatial feature gate values in each convolution stage. By multiplying the original feature maps with the gate values, Point-wise Gate enables selective amplification or attenuation of filters within convolution layers.Rotation Heatmap Predictor: Based on the regional anchor-based grasp generator in [3, 6, 15] , with the region-aware features extracted, we predefine a series of non-uniform rotation anchors for (γ, β) generated via the Anchor Shifting algorithm in [15] , whose Cartesian product forms a 2D rotation heatmap (one axis for γ, the other of β). Then, the grasp detection can be considered a twoclass semantic segmentation problem, while the normalized grasp widths and extra center offsets are regressed for each possible rotation (pixel in the heatmap). 
dataset experiments: 
 Dataset and Metrics: GraspNet-1Billion [3] is a large-scale grasp dataset widely adopted in recent 6-DoF grasp detection research [9, 15, 8, 20] , which provides large-scale training data and a standard evaluation platform for the task of general robotic grasping. During our experiments, we follow the official evaluation pipeline and code of the GraspNet-1Billion dataset, in which detected grasp poses are first filtered with non-maximum suppression, and then the top 50 grasp poses are evaluated with force-closure [28] metrics under a series of friction coefficients condition.Quantitative Results: We compare the overall performance of our RNGNet with multiple typical grasp detection algorithms trained on the same dataset training split. As Table 1 "-": Result Unavailable; CD:Collision Detection, the grasp post-processing algorithm proposed in [3] and utilized by [9] .1 Evaluated with batch size = 1 on Ubuntu20.04 with AMD 5600x CPU and a single NVIDIA RTX 3060Ti GPU.† Reimplemented or tested with the provided codebases.More detailed results are shown in Table 7 of the Appendices. Compared with other methods, RNGNet outperforms current state-of-the-art by a large margin on all dataset splits, especially on the unseen (similar and novel) splits, proving the efficacy and generalizability of our proposed region-aware grasp framework. Furthermore, considering the computation cost for one shot of grasp detection, our method achieves the fastest speed with the best grasp detection quality, which implies the potential of our approach to be applied in dynamic scenes or on widely used computation-resource-constrained platforms in robot deployment.As shown in Fig. 6 , we conduct grasp evaluation using the force-closure metric under different difficulties with different object friction coefficients. With lower object friction, the object is more likely to drop from the gripper and thus requires more high-quality grasp detection results. It can be clearly seen in the figure that RNGNet surpasses former SOTAs by a large margin with different object friction coefficients, especially for the settings with lower friction coefficients.As is indicated in Table 2 , following the setting in [8] , we evaluate the grasp quality at all scales (S: Small, M: Medium, L: Large). RNGNet significantly improves performance with all the metrics, especially in the small-scale grasps and the novel split, proving the proposed NGS's effectiveness. RNGNet can learn more robust and generalizable region features across different scales with NGS. Qualitative Results: We also conduct grasp detection visualization with HGGD [15] and GSNet [6] on three parts of the test set (seen: scene100-129, similar: scene130-159, novel: scene160-189). As depicted in Fig. 5 , RNGNet exhibits superior grasp detection quality and achieves a higher grasp coverage rate in cluttered scenes than prior works, even in scenarios involving occlusion or partial observation. This enhancement provides the robot with more feasible action alternatives, benefiting scene-level grasp planning and execution. Furthermore, when compared with [15, 9] , it is noteworthy that our proposed region-aware framework enables the generation of grasps better aligned with object centers and surfaces, consequently enhancing grasp stability. The overall qualitative results underscore the validity of our representation and approach in cluttered scenes, offering a more comprehensive perspective.Normalized Grasp Space Ablation: We perform ablation studies on our proposed Normalized Grasp Space, which primarily consists of Patch Normalization, Depth-adaptive Patch Extraction, and Scale Randomization. The results in Table 3 illustrate the effects of these three components on Normalized Grasp Space. As expected, both Patch Normalization and Depth-adaptive Patch Extraction significantly enhance performance across all scenes, proving the effectiveness of the novel viewpoint about generating grasps in the region-aware and grasp-centric spaces. Additionally, Scale Randomization also boosts the performance of our methods on the dataset.Region Normalized Grasp Network Ablation: Table 4 presents the ablation experiments conducted on the RNGNet with different input modalities. Remarkably, our method achieves a commendable performance even with only depth maps as input. Introducing XY maps or RGB images as supplementary inputs leads to considerable improvements. While geometric information from Z maps plays a crucial role in 6-DoF grasp detection, the additional positional information from XY maps and the inclusion of colors contribute to enhanced feature extraction and are especially beneficial for generalization to previously unseen (similar and novel) scenes. Finally, applying Point-wise Gate Module to feature maps in all stages yields further improvements across all test splits, with a negligible increase in the number of parameters (approximately 36k, or about 1% of the total parameters). These findings support the efficacy of our approach, which is built with a combination of point-based networks and CNNs. 
realworld experiments: 
 Cluttered Scene Clearance: To conduct a comprehensive comparison of grasp detection methods in real-world cluttered scenes, we constructed multiple cluttered scenes consisting of 6 to 9 randomly placed objects within a 50cm × 50cm workspace, including novel objects, object stacking and occlusion, as shown in Fig. 7 . We deployed multiple representative grasp detection methods [15, 9] on our real-world robot platform. Subsequently, akin to the metrics utilized in [15] , we assessed the overall Success Rate and Completion Rate to facilitate overall comparison.Compared with other methods, results presented in Table 5 demonstrate the generalization capability to real-world grasping of our framework. However, it is worth noting that HGGD struggles to generate grasps for objects with small scales, and AnyGrasp exhibits imprecise rotation predictions, resulting in unstable grasping and frequent drops, particularly evident when objects are stacked. Both methods are prone to failure when the captured point cloud becomes more noisy or only partial object observations are available. In contrast, RNGNet, with its robustness to noise originating from the depth sensor, demonstrates the ability to predict more stable grasps with precise rotations. This effectively reduces collisions and object drops, showcasing the potential of this method.Closed-loop Grasping: With RNGNet, we are able to detect grasps in any region regardless of the patch location or the scene structure. This implies a practical downstream application potential when there is no need to generate grasps across the scene. To evaluate the efficiency of our framework in more challenging scenes, we build two dynamic settings as illustrated in Fig. 8 : (a) Human-torobot handover; (b) Moving object on the conveyor. Experiment results in Table 6 confirm that our framework can accomplish dynamic grasping at a satisfactory success rate by detecting high-quality grasps in a real-time frame rate and executing the closed-loop controlling. The proposed closed-loop grasping pipeline can effectively handle more complex nontable-top grasping scenarios. Thus, our proposed method proves robust to a much noisier environment, which underscores the potential of our method for addressing challenging dynamic scenarios. The closed-loop grasping algorithm is demonstrated in Algorithm. 1 of the Appendices, and more demonstrations of the dynamic grasping experiments can be found in the Supplementary Videos. 
limitation and conclusion: 
 Normalized Grasp Space (NGS), a unified space for region-based grasp detection, is proposed to obtain consistent region representation. With the representation, we further build an efficient convolution-based Region Normalized Grasp Network (RNGNet) to extract features and predict rotation heatmap. Though efficient, our method, relying on single-view images, is vulnerable to noises and can be only applied with rigid bodies. We plan to explore integrating scene-level information from the multi-view image during the process of closed-loop grasping, which is critical for precise manipulation in complex scenes or with specific instructions. More limitations are discussed in the Section C and D of the Appendices. 
 
 Appendix: Detailed Results on the GraspNet-1Billion Dataset: In addition to the overall grasp AP (Average Precision) metric, we also provide the evaluated grasp quality based on force-closure metric [28] with varying friction coefficients, denoting AP µ , representing the average precision at given friction coefficient µ. As is shown in Table 7 , our proposed RNGNet outperforms previous state-of-theart methods in terms of all metrics under different friction conditions, especially on the difficult scenarios (similar/novel splits and AP 0.4 with lower friction coefficient). Extra Real-world Results: For more detailed performance evaluation, we conduct extra real-world experiments at different noise levels to illustrate the practical benefits and robustness of our method.In the extra real-world experiments, for more challenging grasping, we introduce more novel objects (50 objects in total vs 24 used in the manuscript) in the experiments, and more objects (6 to 12 now vs 6 to 9 in the manuscript) may be randomly placed in a clutter. Each method is tested with extra 6 scenes under different settings. The other settings (grasp order, collision check, robot control, etc.) are kept the same as in HGGD [15] and the manuscript. As is shown in the table above, our method achieves an impressive success rate in attempts without additional noise, efficiently clearing all scenes. In comparison, AnyGrasp exhibits lower attempt success and scene clearance rates and may struggle to clear scenes due to repeated failures on specific objects. Our method, on the other hand, offers more viable grasps across scenes with multiple sampled patches, which benefit the scene clearance process. This is also illustrated in Figure 5 of the manuscript (AnyGrasp mainly uses GSNet for one-frame grasp detection).As for grasp detection with noisy inputs, AnyGrasp shows a significant decline in performance as noise increases, while our method performs more consistent grasp detection results in noisy environments and is capable of clearing scenes with relatively high noise levels (0.02 m). These resultsHeatmap Model, we filter the table pointcloud by z coordinates and use farthest sampling on the foreground pointcloud to acquire region centers. As is shown in the table below, even without the guidance of the Grasp Heatmap Model, our method can still perform high-quality grasp detection for a relatively slower speed (more patches are processed). Grasp Inference with Multi-scale Grippers: As clarified in Section. 3.1, the Normalized Grasp Space (NGS) demonstrates scale invariance. In other words, our grasp detection model can be easily scaled to accommodate a different range from the fixed gripper width defined in the training dataset without any extra training. In Fig. 9 , we illustrate this by enlarging the gripper scale from w gripper = 0.05 m to w gripper = 0.2 m, showcasing grasps in a cluttered scene. These results affirm that our framework can seamlessly adapt to grippers of varying sizes. As is shown in Fig. 10 , we extend the application of our framework to a much larger indoor scene than the typical tabletop settings. The proposed RNGNet successfully generates large-scale grasps for large objects, validating the effectiveness of the Normalized Grasp Space.To further investigate the effectiveness of our Normalized Grasping Space for object grasping with grippers of different scales, We build scenes with objects with significantly different scales in the Sapien [31] simulator and conduct grasp detection and robot grasping. Experiments prove that our Normalized Grasping Space can generalize well to grippers with different scales and has the potential to apply to industrial scenes, in which data and labels are hard to obtain. Detailed results and visualization can be found in our Supplementary Video. Normalized Patch Dataset Preparation: Currently, most grasp datasets focus on providing objectlevel grasp labels [13, 14] or scene-level grasp label [3, 32] , which are not directly suitable for our normalized scheme training. Although GraspNet-1Billion offers a large number of RGBD images and grasps annotations, based on the proposed Normalized Grasp Space, we need to construct a patch-based grasp dataset providing normalized patches and grasps annotations. As is shown in Fig. 11 , utilizing the scene-level annotations provided in the GraspNet-1Billion dataset, we generate a new patch-based grasp dataset for Regional Normalized Grasp Network training. Firstly we obtain the object segmentation mask in the original dataset and adopt a Gaussian kernel to dilate the mask image for covering possible graspable regions as much as possible. Then a fixed number of patch center candidates are randomly sampled on the dilated mask image and cropped. Also, as is mentioned in [33] , depth information is important to generate collision-free grasp detection results. Thus, we add domain randomization to the patch center depth on the RGBD image planes, which provides a patch-based dataset covering more of the possible circumstances and improves the robustness of our training method. Grasping Order and Execution in Clutters: In our cluttered scene clearance experiments, we mainly adopt the perception-planning-grasping open-loop scheme. For each grasp attempt, firstly, the robot moves to the initial home pose and captures the single-view RGBD image, from which multiple normalized grasp regions are extracted. Then, grasps are detected in each region to compose the scene-level grasps for later grasp execution. During the grasping part, the grasp with the highest score is selected as the current target pose. We utilize the MoveIt [34] framework to plan the robot's trajectory and execute the grasp poses using the UR5e equipped with the Robotiq gripper. RRT [35] serves as our path planner to plan a feasible trajectory to the target pose and execute the final grasp action. To avoid potential collisions and unsafe planning trajectories, MoveIt's collision box construction tool is employed to define collision geometry. In case of failed grasp execution resulting from collisions or inaccurate grasps, the scene will be restored, and the objects will be re-placed.Target-oriented Grasping: Actually, by utilizing the proposed region-aware grasp framework, we successfully decouple the grasp location problem and grasp detection problem. As we state in the closed-loop grasping section in the manuscript, with RNGNet, we are able to detect grasps in any region regardless of the patch location or the scene structure. This implies a practical downstream application potential when there is no need to generate grasps across the scene. In short, Regionaware Grasp Framework allows a more flexible patch localization for different purposes.Closed-loop Grasping Pipeline: As demostrated in the Algorithm. 1, we integrate grasp detection with simple closed-loop robot controlling and keep tracking the real-time grasp detection results to achieve closed-loop grasping in dynamic scenes. We process the input image as the same size 640 × 360 and adopt the pre-trained Grasp Heatmap Model checkpoint in [15] for a fair performance comparison. In the feature extraction part, we directly adopt the ResNet-18 [36] with half width as the mainstream of our backbone in RNGNet. For the implementation of the Point-wise Gate module, a 3-layer shared MLP with the same width as the corresponding feature maps is adopted to generate gate value for feature maps of each resolution. In our rotation heatmap predictor, the settings about rotation anchor are kept unchanged from those in the [15] . AdamW optimizer is adopted to train our RNGNet.The training loss of RNGNet is the weighted sum of the regression losses and the anchor classification losses:EQUATIONwhere L θ cls and L γβ are the losses for Euler angle classification, calculated using focal loss [37] and L θ reg , L t and L w are the losses for θ refinement, translation regression and width regression, calculated using smoothed L1 loss.Data Efficiency: To further confirm the data efficiency of our framework, we conduct extra experiments by training methods with different portions of data. For fair comparisons, we test our method without Grasp Heatmap Model (pretrained Grasp Heatmap Model from HGGD uses all the training data and may lead to higher test results), that is, using the farthest sampling in the filtered foreground pointcloud to extract local regions. As is shown in the Table 12 below, our method achieves better data efficiency than other baselines. Even with only 10% training data, our method performs a competitive grasp detection performance. 


Decision: The reviewers are in agreement that the paper addresses an important and relevant problem. They also agree that the experiments in simulation are thorough and the experiments show some improvement over the existing literature.

However, the specific novelty of the paper appears minimal to two of the reviewers. This coupled with the only minor increase in performance on real world tasks, makes it difficult to draw a strong conclusion from the paper. There's also some question as to how the equivariance stated as a contribution is ensured.

---Update Post Rebuttal---
Thanks to the authors and reviewers for a thorough discussion of the concerns above and in the initial reviews.

Based on the updated information the reviewers and I agree that we can now recommend the paper for acceptance.

As noted by one reviewer there is remaining concern about the lack of comparison with "grasping methods that leverage shape completion, equivalent neural networks, and instance segmentation."

We also strongly suggest the authors incorporate their answers form the rebuttal phase into the paper to improve the clarity of discussions on the contributions and the technical approach.

Reviews: review: title: title: Using local regions for grasping. Performance improvement over existing methods.
paper_summary: paper_summary: This paper proposes to normalize SE(3) grasps in table-top scenes based on local regions and uses a learning-based method to generate normalized grasps based on local features. Specifically, a heatmap indicating potential grasps is generated using a prior method [15], and the candidate regions are sampled from the heatmap and further normalized based on the gripper size, distance to the camera, and focal length of the camera. The RGBXYZ image of each candidate region is fed to a neural network to predict grasp rotations, widths, and translational offsets.
main_review: main_review: Strengths:
1. Simulation and real-world experiments suggest that this region-based method is effective and has qualitative improvement over existing methods.
2. Because the scales of the grasps are normalized based on the region sizes, the learned model can generalize to different object sizes. This generalization is illustrated in Figure 9 and in experiments.
3. Overall, the paper is easy to read and presents details for the implementations. 

Weakness:
1. As the authors discuss in the related work section, the idea of utilizing regional 3D geometry has been applied in many grasping works before. The authors argue that most existing methods extract local regions of predetermined sizes. However, [15] extracts local regions based on predicted grasp widths, which are not fixed. As a result, the technical novelty is limited.
2. This method generates the initial heatmap for sampling candidate regions using the Grasp Heatmap Model from [15]. The Grasp Heatmap Model takes in the whole scene as input and is, therefore, not invariant to different grasp sizes, rotations, and translations.
3. Different from existing works such as EdgeGrasp, the neural network itself is not equivariant. Therefore, the neural network model still needs to learn to generate different grasps when the region feature is rotated. This reduces the sample efficiency and generalization of the method to different object orientations.
questions: questions: Besides the concerns above I hope the authors can address, I have the following questions:
1. I hope the authors can clarify why the method has equivariance for theta because the patch seems to be a square which is not symmetric with respect to the z-axis. 
2. In Table 5, why are the different methods tested on different numbers of trials?
3. Why are grasping methods that leverage shape completion, equivariant neural networks, and instance segmentations not discussed or compared against? They are also important for grasping from cluttered table-top scenes.
limitations: limitations: 2
review_summary: review_summary: This paper shows good quantitative improvements over existing methods. However, leveraging local geometric regions has been widely adopted. The technical novelty needs to be further discussed.
title: title: RNGNet: State-of-the-Art Performance in Grasp Pose Detection
paper_summary: paper_summary: This paper identifies an inherent shortcoming of existing point cloud-based grasp pose detection neural networks, namely that the area of interest can vary in the presence of clutter. It proposes a Normalized Grasp Space to represent regions of interest and a Regional Normalized Grasp Network that operates on this space. RNGNet is trained on the GraspNet-1Billion dataset, and achieves state-of-the-art performance, both in precision and runtime on the evaluation set. It also outperforms baseline systems when evaluated on a real robot, and achieves high performance in more difficult dynamic tasks.
main_review: main_review: This paper did many things well, including:
- Introducing the Proposing a new Region-aware Grasp Framework, the Normalized Grasp Space representation, and the Region Normalized Grasp Network.
- Presenting a summary of relevant grasp pose detection systems, as well as recent region normalization approaches for point cloud-based grasp pose detectors.
- Illustrating the normalized grasp region patches, network architecture, and framework well with illustrations.
- Explaining characteristics of the NGS that enable the RNGNet to achieve state-of-the-art performance.
- Proposing a network architecture that fits into the proposed framework to deal with the identified issues, enabling improvements in both accuracy and efficiency.
- Presenting a detailed breakdown of performance of eight baseline systems on the GraspNet dataset for a fair comparison against existing state of the art.
- Providing an additional breakdown of performance at different scales, with a comparison against the relevant state of the art.
- Providing additional sanity check qualitative visualizations to ensure all top grasps proposed by the system appear more reasonable than the baselines.
- Performing ablation studies to ensure each operation of the NGS and each input to the RNGNet aided performance, as well as on the network architecture.
- Verifying that RNGNet outperformed two of the baseline systems when evaluated on a real robot.
- Further demonstrating the system's performance on dynamic handover and conveyor real robot tasks.
- Demonstrating in simulation how the system generalizes with grippers of different sizes.
- Was well written.

This paper could be improved by:
- Citing Rethinking 6-Dof Grasp Detection: A Flexible Framework for High-Quality Grasping by Tang et al. This recent work from 2024 outperforms the other cited works on the GraspNet dataset, but underperforms RNGNet.
questions: questions: The only suggestion I have for improving this paper is to cite Rethinking 6-Dof Grasp Detection: A Flexible Framework for High-Quality Grasping by Tang et al., that outperforms the other baselines but underperforms RNGNet.
limitations: limitations: 3
review_summary: review_summary: RNGNet's network design is based on an analysis of the drawbacks of existing systems. It achieves a new state-of-the-art on the GraspNet dataset in grasp detection precision while outperforming other baselines at detection speed. It is also shown to be more effective than several baseline systems in real robot experiments. The exhaustive ablation studies support the design decisions.
title: title: Interesting idea, yet no real world experiments to support it
paper_summary: paper_summary: Authors address the problem of generating grasps for a gripper in a cluttered scene. The grasps are represented as a pose and a gripper width. The main novelty is in the way the RGBD input is processed. The authors "normalize" the patches in the input to be scale, translation, and rotation invariant. Ablations on a chosen dataset show that this normalization yields significant improvement in average precision (AP) over other methods. The authors validate the method on the real robot and show they can clear the cluttered scenes, as well as pick up a moving object due to fast inference time of the proposed method.
main_review: main_review: [strengths]
(relevance/significance) The paper addresses an important question: quickly generating 6 DoF grasps in cluttered scenes. 
(originality) The main novelty is in normalizing the input image to be scale, translation, and rotation invariant. 
(quality) In the real world, the authors successfully clear cluttered scenes using the proposed approach, and showcase ability to pick up a moving object by running closed loop grasping.
(clarity) The paper is written in a clearly and easy to understand

[weaknesses]
(real world experiments) Authors don't ablate their method on the real world task, and show marginal improvement over baseline (Table 5)

The authors propose two novelties: normalizing space and a pointwise gate (PG) module as a part of neural network.
Through ablation of the normalizing space, authors show significant yields in average precision (AP), while PG has a marginal improvement in AP.

The authors provide no real world experiments/ablations for either of the proposed novelties to show that this translates to improvement on the real robot, and if so, by how much! This is my main problem with this paper. The authors provide no evidence that their novelties translate to improvement in the task of clearing the table. If the authors are able to show this, I believe it would be a much stronger paper. Further, in the real world experiments, the authors show only a marginal improvement over AnyGrasp baseline. Both methods are able to clear 7 tested scenes, but for RNGNet (authors) it took 60 grasp attempts, and 65 for AnyGrasp.

Picking up on conveyor belt was a nice demo that shows fast inference of their approach (the video had a significant multiplier though).
The authors should state limitations of the approach in the main paper, not leave them for appendix.

EDIT: The authors later provided real world experiments and ablations showing their approach is superior to the baseline. Furthemore, all of my below questions are answered and I vote this paper gets accepted.
questions: questions: Can you provide real-world experiments/ablations for either of the proposed novelties to show that the novelties translate to improvement on the real robot, and if so, by how much? No AP, but PPM (picks per minute) or number of attempts to clear the table.

Further, in real-world experiments, can you show significant improvement over the AnyGrasp baseline?

How do you ensure the reachability of your grasps? How is this accounted for in the experiments?

What are the real limitations of your method? When does your method mostly fail?

EDIT: all questions are answered, thank you
limitations: limitations: 1
review_summary: review_summary: My recommendation is based on real world experiments showing strength of the proposed approach: compared to SoTA as well as ablations