Real-to-Sim Grasp: Rethinking the Gap between Simulation and Real World in Grasp Detection

Authors: jia-feng  cai
zibo  chen
xiao-ming  wu
jian-jian  jiang
yi-lin  wei
wei-shi  zheng
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-18 12:42:18

Abstract: For 6-DoF grasp detection, simulated data is expandable to train more powerful model, but it faces the challenge of the large gap between simulation and real world. Previous works bridge this gap with a sim-to-real way. However, this way explicitly or implicitly forces the simulated data to adapt to the noisy real data when training grasp detectors, where the positional drift and structural distortion within the camera noise will harm the grasp learning. In this work, we propose a Real-to-Sim framework for 6-DoF Grasp detection, named R2SGrasp, with the key insight of bridging this gap in a real-to-sim way, which directly bypasses the camera noise in grasp detector training through an inference-time real-to-sim adaption. To achieve this real-to-sim adaptation, our R2SGrasp designs the Real-to-Sim Data Repairer (R2SRepairer) to mitigate the camera noise of real depth maps in data-level, and the Real-to-Sim Feature Enhancer (R2SEnhancer) to enhance real features with precise simulated geometric primitives in feature-level. To endow our framework with the generalization ability, we construct a large-scale simulated dataset cost-efficiently to train our grasp detector, which includes 64,000 RGB-D images with 14.4 million grasp annotations. Sufficient experiments show that R2SGrasp is powerful and our real-to-sim perspective is effective. The real-world experiments further show great generalization ability of R2SGrasp. Project page is available on https://isee-laboratory.github.io/R2SGrasp.

Main paper: 
 introduction: 
 Grasping objects in unstructured environment is fundamental for robots designed to accomplish various complex tasks [1, 2, 3] . Previous works [4, 5, 6, 7, 8, 9, 10] utilize real-world data to achieve impressive performance. However, collecting large-scale datasets in real world is still challenging, which limits the improvement of the grasp and generalization abilities. To address this problem, simulated data provides a feasible alternative, but the grasp performance trained in simulation is hindered by the large gap between simulation and real world when applying to real world scenarios.There exist a few methods attempting to bridge this gap, which use domain randomization [6, 11, 12] or domain adaptation [13, 14, 15, 16, 17] to achieve it. The former adopts various randomized conditions to make the simulated data closer to reality, while the latter narrows the gap by aligning feature distribution between two domains. However, they both include a sim-to-real way that adapts the simulated distribution to the real-world one, which explicitly or implicitly introduces the camera noise in real data when training the grasp detector. This noise appears as positional drift and structural distortion as shown in Figure 1 (a), disrupting the training process of the grasping detector that is highly sensitive to variance in 3D space. To demonstrate the negative effect of camera noise, we compare the upper bound of grasp performance of the grasp detector trained on noiseless data and on noisy data as depicted in Figure 1 (b). The performance of grasp detector trained in data with real-world noise is lower than that of trained in noiseless data, which suggests that the camera noise introduced by the sim-to-real way will disturb the learning of grasping skills.To this end, we propose a novel Real-to-Sim framework for 6-DoF Grasp detection, named R2SGrasp, to bridge the gap between simulation and real world in a real-to-sim way. This way adapts real-world distribution to the simulated one in inference phase to leverage the precise grasping skills learned from the simulation, which bypasses the camera noise in detector training and ensures robust performance. Specifically, a grasp detector is trained with noiseless simulation data to attain accurate grasping capability. During inference, we achieve real-to-sim adaptation by using a Real-to-Sim Repairer (R2SReparier) to preprocess the depth map so as to mitigate positional drift and structural deformation, along with a Real-to-Sim Feature Enhancer (R2SEnhancer) to enhance real-world local features with simulated structural features. Benefiting from our two novel modules, R2SGrasp implements the real-to-sim way in data and feature levels to achieve higher performance.Moreover, benefiting from our real-to-sim perspective, our grasp detector only requires training in simulated data, which endows our framework with the expandable generalization ability by scaling up the simulated data cost-effectively. Therefore, we build a large-scale simulated dataset, named R2Sim, including 256 daily objects, 500 cluttered scenes and 64,000 RGB-D images with total 14.4 million grasp annotations. To improve the efficiency of training on such a large-scale simulated data, we propose a simplified annotation strategy, resulting in a reduction on training time by 73%.We conduct extensive experiments on the GraspNet-1Billion dataset [5] to verify the effectiveness of our R2SGrasp. R2SGrasp achieves superior performance in real world only with simulated data, surpassing methods of using annotated real-world data. Moreover, we further evaluate R2SGrasp through many real-world grasping experiments, verifying the great generalization of our framework. 
related work: 
 Sim-to-real transfer in grasp detection. The gap between simulation and reality is a significant problem in robotics grasping. For those RGB-based grasping methods, most works utilize advanced image processing techniques, such as adversarial training [13, 14] , GAN-based methods [17, 18] or teacher-stundent models [15] to bridge the gap between simulated and real RGB images or features. We focus on the transfer of point cloud-based grasping methods from simulation to reality. Fang et al. [6] adds Gaussian noise to the point cloud for training. Zheng et al. [16] aligns the features Training of grasp detector with R2SEnhancerSimulated point cloud Grasp datasets in simulation. Simulation datasets [18, 19, 20] are the most commonly used data for robot learning. Previous works [21, 22, 23, 24] propose the datasets that define grasp poses within the continuous grasp sample space, which have insufficient annotations, challenging to cover the entire sample space. Recently, [5] uses discrete annotations to fully cover the entire sample sample so as to learn the robustness of grasp abilities. Following that, we decouple the grasp pose parameters and discretize them at fixed intervals within the parameter space. Our annotations are rich enough to cover the entire discrete parameter space. Due to the demand for large-scale simulation training, we also propose a simplified annotation method to enhance training efficiency. Compared to previous simulated datasets, our dataset employs a discrete representation of grasp poses, and provide more dense grasp annotations that cover the entire sample space, which enables our model to achieve great generalization ability.3 R2SGrasp: A Real-to-Sim Framework for 6-DoF Grasp detection 
task definition: 
 We first describe the task of 6-DoF grasp detection. Given the single-view RGB-D image I rgbd ∈ R H×W ×4 , the goal of grasp detector is to predict a set of accurate grasp poses in cluttered scenes. The 6-DoF grasp pose can be denoted as G = [R, T, w], where R ∈ SO(3) is the rotation matrix, T ∈ R 3 is the translation and w is the gripper width. We follow [5] to decouple R into approaching vector v and in-plane rotation θ, T into grasp point p and approaching distance d from grasp points to grasp origin o, as shown in Figure 2 . In this work, we aims to bridge the gap between simulation and real world in 6-DoF grasp detection, where we will train our model in simulation and test it in real world data for employment.Previous works employ the sim-to-real way to bridge the gap between simulation and real world. However, they explicitly or implicitly introduce the camera noise in real data when training the grasp detector. The positional drift and structural distortion caused by camera noise disrupt the training process of the grasp detector, limiting the grasp performance, as depicted in the right hand side of inference phase in Figure 1 (c).To solve the above problem, we propose a novel real-to-sim perspective that bridges the gap between simulation and real world in a real-to-sim way. This way is able to train a robust grasp detector on noiseless simulated data to avoid the interference of noise in grasping skills learning, thereby achieving stronger grasping capabilities. Although this grasp detector performs exceptionally well on noiseless simulated data, its performance degrades on real data due to the presence of noise.To apply the precise grasping ability learned from simulation to the real world, we consider adapting the real world data and feature to the simulated ones. Specifically, we introduce the R2SGrasp framework, including a Real-to-Sim Data Repairer (R2SRepairer) and a Real-to-Sim Feature Enhancer (R2SEnhancer) to achieve real-to-sim adaptation at data and feature level respectively. The overview of R2SGrasp is shown in Figure 2 . Except the two modules, our framework also consist of a grasp detector to generate the aggregated features of the grasp points and a MLP to output the grasp widths and grasp scores for each grasp candidate along the approach direction, following [8] .We adopt the same loss function as [8] to train our grasp detector. More details about our grasp detector and loss function can be seen in Appendix A.2. How the two modules adapt the real world data and feature to the simulated ones will be detailed below. Impact of camera noise. We find that the gap between simulated and real domain primarily stems from camera noise in real data which appears as the positional drift and structural deformation in point clouds. As shown in Figure 3 (a), there is severe noise in depth map captured from the real world. To explore the impact of this noise, we conduct a verified experiment that simply replacing the severe noise regions in the real depth map with accurate depth values to repair the structure. As shown in Figure 3 (b), the average precision increases by 23.61%, even surpassing the baseline [8] whose model trained on real dataset. The results suggest that if we can mitigate this noise and repair the structure, the grasp performance can be improved significantly.Mitigating positional drift and structural deformation. We propose the real-to-sim data repairer (R2SRepairer) to mitigate positional drift and structural deformation by reducing the noise amplitude using an encoder-decoder architecture. It takes RGB-D images I rgbd ∈ R H×W ×4 as input and outputs residual map I r ∈ R H×W . The repaired depth map Îd can be obtained as Eq. ( 1):EQUATIONEQUATIONTo obtain supervision data I * n , we use the object models and object poses from GraspNet-1Billion [5] to efficiently generate the simulated data paired with the real ones in Blenderproc [25] . By setting the camera position in the simulation environment to match the real camera pose, we can render simulated depth maps that are identical to the real depth maps from GraspNet-1Billion. I * n is calculated as Eq.( 2), where I s d and I r n are paired simulated and real-world depth maps. The R2SRepairer is trained with Smooth L1 loss.To further bridge the gap between the simulation and real world, R2SEnhancer conducts the realto-sim adaptation at feature level by using the precise structural features to enhance the real-world local features based on feature similarity. As shown in Figure 2 , memory bank in R2SEnhancer accumulates diverse geometric structure information through clustering during training with simulated data. During inference, this simulated structure information is used to enhance the features of real structures, thereby improving the perception of our grasp detector in real geometric structures.Building the memory bank. Memory bank M = {k j } K j=1 stores K simulated structure features k j during training. At first, we initialize M by sampling from the normal distribution. Given the local features F s l = {f i } Ns i=1 of each batch extracted in simulated data when training, we calculate the distance between local features and stored features based on cosine similarity, and assign each local feature to the closest stored feature. We then update the stored features with the average of all assigned feature clusters using momentum update as Eq.( 3):EQUATION)where fj is the average feature of the cluster corresponding to k j , and α ∈ [0, 1] is the momentum update parameter. After that, we only keep the centers of clutters for the next-batch training.Real-world feature enhancement. After training with extensive simulated data, the memory bank stores a diverse set of structural features that represent common geometric structures. We retrieve similar simulated structural features to enhance the real features through cross-attention mechanism, where the local features F r l = {f i } Ns i=1 extracted from real-world point clouds serve as the queries, and the simulated structural features stored in the memory bank serve as the keys and values. The enhanced real features are obtained using Eq.( 4).EQUATIONwhere ρ denotes the Softmax function, W q , W k , W v ∈ R C×Dm are learnable encoding matrices for query, key and value in cross-attention mechanism. 
r2sim dataset: 
 Motivation of the R2Sim Dataset. Thanks to our real-to-sim perspective, our grasp detector only needs simulated data for training. This endows our framework the expandable generalization ability by scaling up the simulated data cost-effectively. Towards this end, we construct a large-scale dataset named R2Sim in Blenderproc [25] , including 256 daily household objects selected from GSO [26] and GraspNet-1Billion [5] , 500 cluttered desktop scenes, 64,000 RGB-D images taken from different views and approximately 14.4 million grasp annotations. Each frame is also annotated with object segmentation map, object poses, camera pose, graspness heatmap [8] and so on. More information about our R2Sim is available in Appendix A.3.We provide an automated process for generating scene data with various randomization techniques to enhance the diversity, such as rich background textures, diverse lighting conditions, cluttered table scenes, and various random camera views. This process automatically generates RGB-D images, segmentation masks, 6D object poses, and camera poses, providing rich training data for the grasp detector. More details are available in the Appendix A.3.3.Grasp pose annotation for efficient training. Given the object meshes, we use a discrete dense annotation method [5] for object grasp pose labeling, where the grasp points are densely and uniformly distributed on the object surface, with each grasp point having 300 approach directions and 48 grasp candidates per direction. Scene grasp pose annotations can be obtained by projecting the grasp annotations of objects according to their 6-DoF poses. As the data scale expands, the dense annotations significantly increase training burden, including prolonged data loading time and excessive memory consumption. To improve training efficiency, we simplify the grasp pose annotations by reducing the number of negative samples. Specifically, we first filter out grasp points that do not have successful grasp candidates. Then, for the retained grasp points, we select the top 60 approaching vectors from the original predefined 300 vectors based on the proportion of successful grasp candidates. Benefiting from our simplification, we filter out many invalid grasp annotations, which improves the training efficiency and is more suitable for large-scale simulation data training. More details and further experimental validation are illustrated in Appendix A.3.4.5 Experimental Results 
experimental setup and details: 
 Datasets. We use R2Sim dataset to train the grasp detector and use twin datasets based on GraspNet-1Billion [5] to train our R2SRepairer. The grasping performance is validated using the test set from GraspNet-1Billion. GraspNet-1Billion is a real-world dataset comprising 190 scenes, with 100 scenes allocated for training and 90 for testing. The test set is further divided into three subsets: seen, similar, and novel, where each category contains 30 scenes.Metrics. We employ AP µ and AP as the evaluation metric for grasp performance, the metrics introduced by [5] . AP µ represents the average precision of the top 50 grasp poses in terms of grasp scores under different friction coefficient µ. AP denotes the mean of AP µ .Implementation Details. The backbone of R2SRepairer is implemented using a UNet [27] architecture with a ResNet34 [28] encoder, outputting feature vectors with 32 channels. The feature extractor in our grasp detector adopts the cascaded graspness model in GSNet [8] with cylinder grouping operation. The cross-attention mechanism consists of four heads, with a model dimension of 256. The storage capacity of the memory bank K = 120, with the momentum update parameter α = 0.999. Additionally, R2SRepairer is trained with a learning rate of 0.001, using the AdamW optimizer and a batch size of 3 for 30 epochs. And the grasp detector is trained with a learning rate of 0.001, using the Adam optimizer and a batch size of 4 for 30 epochs.Comparison with sim-to-real methods. We first compare our method with current sim-to-real approaches as shown in Table 1 (the methods without using labeled real data). Source-only trains the grasp detector in simulted data without any sim-to-real techniques. ADD-Noise [6] adds Gaussian noise to the simulated point cloud. Global-DA [29] and Local-DA [29] utilize adversarial domain adaptation methods to align the distributions of the simulated and real domains at global and local feature levels respectively. Note that the above methods utilize the same grasp detector [8] as ours for fair comparison. It is evident that our method significantly surpasses others by 9.19/18.64, 3.65/7.62, 5.07/10.14 AP at least in the seen, similar, and novel categories respectively. Additionally, the results of previous methods have little improvement, even decline on grasp performance. We think the reason is that these works explicitly or implicitly introduce the camera noise in real data when training the grasp detector, which disturbs the learning of grasping skills.Comparison with methods trained in real world. To further illustrate the advantages of R2SGrasp, we also compare our method with others [5, 9, 8, 7, 4] trained on real-world datasets, as shown in Table 1 (the methods using labeled real data). The performance of our method is close to that of models trained on real data, and it even surpasses current SOTA methods in most metrics. Specifically, compared to GSNet [8] , our method outperforms it by 4.93, 7.08, and 5.73 AP in the seen, similar, and novel categories on Kinect data. The results demonstrate that our R2SGrasp can leverage the precise grasping skills learned from the simulation in the real world.Ablation Study. We conduct ablation study to validate the effectiveness of each component in our method as shown in 4 : Comparison of camera noise before and after R2SRepairer. Different colors in the noise map represent different noise amplitude ranges, with amplitude measured in millimeters training solely on R2Sim, there remains a noticeable performance gap compared to the control group. After integrating R2SRepairer (see Line 3), the grasp performance increases significantly and surpasses the grasp detector trained in real-world data. This performance increase steps from mitigating the camera noise, which can be seen in Figure 4 . When further adding R2SEnhancer (see Line 5), we notice an increase in AP across the seen, similar, and novel categories by 3.67, 1.63, and 1.49 respectively. These results indicate that our real-to-sim approach bridges the gap between the simulation and real world, where the R2SRepairer and R2SEnhancer modules achieve real-to-sim adaptation at the data and feature levels respectively. Memory bank size in R2SEnhancer. Table 3 illustrates the effect of memory bank size adjustment on grasping performance. As the value of K increases, the grasp performance improves at the beginning, while reducing after K reaches 120. We believe that a large value of K means more structural features are stored in memory bank, but too large K leads to information redundancy. We experimentally find that the optimal value of K is 120. To further verify the effectiveness and generalization ability of R2SGrasp, we conduct real-world experiments on the Franka Emika Panda robotic arm with an Intel RealSense D455 camera and a two-finger gripper as illustrated in Figure 6 . The experiments are performed on six cluttered scenes, each containing 6-8 randomly placed daily objects. We execute the best grasp pose predicted by grasp detectors for each attempt until all the objects are taken away. The grasping capability is measured by the success rate (SR), defined as the proportion of successful grasps to the total number of grasp attempts. We compare the performance of the grasp detectors [8] trained on GraspNet-1Billion dataset [5] , R2Sim dataset with added noise [6] and R2Sim dataset with our method, as shown in Table 4 . We find that the grasp detector trained on real data has a lower success rate than grasp detectors trained in simulated data, due to the insufficient amount of real data, which includes only 40 objects, limiting the generalization ability. Using large-scale simulation data improves the grasping success rate, and the success rate can further increase with R2SGrasp. This indicates that our method can effectively leverage grasping skills learned from simulation data in the real world. 
conclusion & limitations: 
 Conclusion. In this work, we introduce the R2SGrasp framework from a real-to-sim perspective to bridge the gap between simulation and real world in 6-DoF grasp detection. In our framework, we train a robust grasp detector in simulated data, and adapt real world to simulation at both data and feature levels to utilize the skill learned from simulation. Moreover, a large-scale simulated dataset is constructed cost-effectively to enhance the generalization of our methods. By large-scale training and real-to-sim adaptation, the real world generalization ability actually gains great improvement.Limitations. Grasping flat objects or complex shaped objects is suboptimal, and it is a significant challenge in the current 6-DoF grasping community. We think that our real-to-sim idea can help address this issue by generating data in simulated environments that include these types of objects, allowing the network to learn how to grasp them. In the future, we also plan to collect these objects and expand the dataset size to further tackle these challenges. the correct noise image significantly reduce noise compared to the initial noise image. To further demonstrate the performance of R2SRepairer, we compared the single-view point cloud before and after camera noise repair. As shown in Figure S2 , after noise repair, the positional drift and structural deformation of the point cloud are mitigated, bridging the gap between real and simulated data. S3 : Experiment on noise adjustment in residual labels. 
 
 Appendix: The impact of noise in Residual labels. R2SRepairer is robust to the noise in residual labels. As shown in Table S3 , we add Gaussian noise with a mean of 0 and different variance (std) to the residual labels. The overall AP value increases after adding Gaussian noise, with a more significant increase when std=10. We believe that introducing appropriate noise can enhance the robustness of model learning.Qualitative analysis of structural features in memory bank. Our Real-to-Sim Feature Enhancer (R2SEnhancer) uses the precise structural features stored in the memory bank to enhance the realworld features. To visually demonstrate the semantic information of the stored structural features, we visualize the structures represented by these features. First, we calculate the cosine distance between the features of each graspable point and the stored features in the memory bank, assigning each graspable point to the nearest stored structural feature. Then, we obtain a set of graspable points for each stored structural feature. Based on cosine similarity, we select three distinct features from the memory bank and visualize their corresponding sets of graspable points. As depicted inYellow Figure S3 , there are obvious differences in the structure represented by the three features. The first feature represents sharp object structures, as shown in the first row of Figure S3 , which are commonly found on toy legs and heads. The second feature represents planar object structures, as depicted in the second row, primarily seen on square boxes. The third feature represents curved object structures, as illustrated in the last row, appearing on various curved surfaces, with bottles being the most prominent.Qualitative analysis of grasping performance. To demonstrate that our R2SGrasp can adapt to real data, we use R2SGrasp to predict grasp poses on the GraspNet-1Billion test set and visualize the results, as shown in Figure S4 . In seen and similar scenes, R2SGrasp predicts grasping poses with a success rate close to 100%. In novel scenes, there are some failed cases, which occur due to collisions or grasping at empty locations.A.2 Implementation details of grasp detector. [27] architecture with a ResNet14 [28] encoder built upon the Minkowski Engine [31] . Followed by a MLP layer, we predict the object point mask I o and graspness heatmap I h to select the graspable points along with their corresponding point-wise features with the shape of M × (C 1 + 3), where M is the number of graspable points and 3 denotes the cartesian coordinates of the points. We also select the grasp view of the graspable points from the predefined 300 approaching vectors based on the grasp view scores s v which is also predicted by a MLP layer. Then, we perform cylinder grouping operation along the grasp view for each grasp point to aggregate the features of G neighboring points, followed by the MLP and max pooling operations to extract local structural features with C 2 dimension. Finally, our Real-to-Sim Feature Enhancer (R2SEnhancer) refines the local structural features using the stored simulated features and outputs the grasp scores s g and widths s w shaped as M × 48, where 48 denotes the grasp candidates of the grasp points. Loss Design. The grasp detector is trained with the following loss function:EQUATIONwhere L o , L h , L v , L s , L w are used to supervise the learning of object points, graspable points, grasp views, grasp scores and grasp widths respectively. I * o , I * h , s * v , s * g , s * w is the ground truth of object point mask, graspness heatmap, grasp view scores , grasp scores and grasp widths. L o adopts binary classification loss, while others use regression loss. Due to the simplification of our grasp annotations, some grasp poses may lack supervision signals. Therefore, when calculating the loss, we ignore any predicted grasp poses that lack supervision signals.The overall process of dataset construction is shown in Figure S6 . We start by selecting 256 daily household objects from the Google Scanned Objects dataset [26] and the GraspNet-1Billion training dataset [5] . Then, we generate scenes in blenderproc [25] and simultaneously label the grasp poses of the objects. After that, we project object-level grasp annotations into the scenes and detect the grasp annotations that result in collisions. Finally, we adopt our proposed grasp annotation simplification method to remove ineffective grasp poses. The remaining grasps serve as scene-level annotations. To sum up, our R2Sim dataset comprises 500 scenes with 76,800 RGB-D images. Each scene contains approximately 14.4 million grasp annotations and each frame in every scene is also annotated with object segmentation maps, 6-DoF poses of objects and camera, graspness heatmap and view graspness. In the graspness heatmap, brighter areas indicate a higher likelihood of successful grasps. Similarly, higher values of view graspness also represent a greater probability of successful grasps. Some examples of RGB, depth map, segmentation map and graspness map are shown in Figure S7 . We use a sampling-evaluation approach to annotate the grasp poses of objects. Grasp poses are determined by downsampling high-quality mesh models to ensure that the grasp points are evenly distributed in the voxel space. For each grasp points, 300 approach directions are sampled uniformly on a spherical space. Grasp candidates of each approach directions are explored on a grid defined by 4 gripper depths and 12 rotation angles. To sum up, there are 48 grasp candidates along each approach direction and 14,400 grasp candidates on each grasp point. The gripper width is adjusted as necessary to prevent empty grasps or collisions. We adopt analytic computation method as [5? ] to grade the sampled grasp poses. The grasp scores range from 0 to 1, with higher scores indicating a greater likelihood of successful grasps.A.3.2 Details of scene level grasp annotation.Using the object poses, the grasp pose in object coordinate system is projected onto the world coordinate system. We detect collisions for the projected grasp poses in the scene and set the scores of those that collide to zero. Assuming there are N grasp points projected into the scene, we calculate the success rate of grasp candidates at each point, resulting in N graspness values. Using the camera's intrinsic parameters, we convert the depth map into a single-view point cloud. We then use the K-NN algorithm to match the grasp points in the scene, assigning each point in the point cloud the corresponding graspness value. This value is then back-projected into the image to create the graspness heatmap. Similarly, we calculate the success rate of grasp candidates along each approach direction and using this as the view-graspness. We also make further simplification of grasp pose annotation to improve training efficiency.We automated the construction of cluttered desktop scenes using Blenderproc [25] . We first construct a simple indoor scene with a table placed at the center. Textures for the table, floor, and walls are chosen randomly from specific material categories provided on AmbientCG. The lighting setup of the scene is randomized as well, with practical adjustments to intensity and variations in light color to improve visual clarity and accuracy. Then, we choose a variable number of objects ranging from 7 to 10 from our object pool and place on the table. To create sufficiently cluttered scenes, we place the objects 1.5 meters above the table and allow them to fall naturally onto the surface. Finally, we set 128 camera poses to capture RGB-D images from multi views, where the poses are randomly sampled on the upper hemisphere, with a radius of 1.1 meters centered on the objects' region. Based 


Decision: The paper presents a novel framework, R2SGrasp, aimed at improving 6-DoF grasp detection by addressing the gap between simulation and real-world data. R2SGrasp employs a real-to-sim method comprising of R2SRepairer to mitigate camera noise in real depth maps and R2SEnhancer to enhance real features with simulated geometric primitives. The paper also proposes a large-scale simulated dataset, R2Sim, containing 64000 RGB-D images and 14.4 million grasp annotations. 

Reviewers favorably viewed the proposed method and the experimental results. However, the reviewers also raised some concerns: applicability of proposed approach to other more contact rich tasks, comparisons to other related works, some concerns about claims made in the paper, and the lack of many important training details.

The author response addressed many reviewer concerns. `TS41`, `xCby`, `SR4p` reviewed the author response and settled on accept leaning ratings. The AC looked through the paper, the reviews, and the author response. While the residual concern from `7Q2J` (gap in dynamics) is well placed the AC agrees with `TS41` that it may be beyond the scope of the current paper. The authors are encouraged to include the author-reviewer discussion when revising the paper, particularly the points about additional experiments, appropriate framing of claims, and adding a discussion of the limitations.

Reviews: review: title: title: Paper Review for Real-to-Sim Grasp: Rethinking the Gap between Simulation and Real World in Grasp Detection Download PDF
paper_summary: paper_summary: The paper presents a novel framework, R2SGrasp, aimed at improving 6-DoF grasp detection by addressing the gap between simulation and real-world data. Unlike traditional sim-to-real approaches, which adapt simulated data to real-world noise, R2SGrasp employs a real-to-sim method. Specifically, this paper uses R2SRepairer to mitigate camera noise in real depth maps (data level) and R2SEnhancer to enhance real features with simulated geometric primitives (feature level). In addition, this paper proposes a large-scale simulated dataset, R2Sim, containing 64000 RGB-D images and 14.4 million grasp annotations. Experimental results demonstrate that R2SGrasp outperforms other baseline methods that are trained on real-world data and sim-to-real methods.
main_review: main_review: **Strengths and weaknesses**

**Strengths**
1. This paper proposes a real-to-sim approach, which is different from common sim-to-real ones, that bridges the perception gaps between sim and real and enjoys better generalization. 
2. The contribution of the R2Sim dataset, with extensive annotations and diverse scenarios, provides a robust training ground for the community.

**Weaknesses**
1. The gap between sim and real is usually two-fold, dynamics and perception. It seems that this paper only discusses and addresses the perception gap. The framework's performance is contingent on the physical plausibility of the simulated data, and inaccuracies in the simulation could impact the real-world applicability. From my own experience, when deploying a grasping policy from sim to real, a lot of failures are actually due to the dynamics mismatch between sim and real, such as simulation artifacts, dynamics friction etc. Failure to mention and address the dynamics mismatch somewhat weakens the paper. 
2. There is no limitation section.
questions: questions: 1. What are the computational requirements for training and deploying R2SGrasp on a real robot? How does the performance compare to other methods in terms of processing time and resource usage?

2. How robust is the proposed method in real-time applications with dynamic environments? Some discussion around this will be appreciated. 

3. It would be interesting to see the common failure modes of the proposed methods, such as unreachable joint position, gripper collision, bad repaired depth, etc. Some discussions and analysis will make the paper stronger.
limitations: limitations: 1
review_summary: review_summary: I recommend weak reject of this paper based on the aforementioned weaknesses. I am happy to raise my score if the authors provide clarifications on the questions I raised and address my concerns in the review.
title: title: A novel denoising approach to closing the sim-to-real gap for grasp detection
paper_summary: paper_summary: Learning to denoise raw sensor data to match the "clean" inputs seen during simulation training can enable effective and non-conservative sim-to-real transfer.
main_review: main_review: **Summary:** The authors consider the problem of transferring grasp policies from simulation to reality. Typically, this is addressed by adding noise to the simulation data that policies are trained over (i.e., domain randomization), but the authors point out this approach is heuristic, and actually reduces the performance of the resulting policy. In contrast, they propose an approach where they train a policy over noiseless data in simulation, and online, pass the noisy sensor data through a trained pipeline to denoise it before passing it to the policy. The authors support their claims with good-quality hardware experiments, demonstrating both the performance drop induced by domain randomization, and successful sim-to-real transfer using their learned denoiser.

**Strengths:**

- Very interesting perspective + idea: instead of noising simulation data, denoise real data. Trains models to denoise real-world image data (using digital twins constructed in Blender) along with grasp feature detector that can rely on 3D primitives accessible in simulation.
- Strong results: show their method outperforms methods trained only on real data via access to simulation, and demonstrate compelling sim-to-real performance.
- Thorough real-world experiments with enough data to make a compelling sim-to-real case.
- Good comparisons against existing methods for grasp detection (GraspNet1B) and a more traditional domain randomization-based approach. Hardware experiments show a clear improvement from applying the proposed method. 
- Mostly clearly written + a simple, but compelling story.

**Weaknesses:**

- Missing an explicit discussion of paper limitations (required by CoRL guidelines).
- Hardware experiments are also somewhat limited in scope (36 objects, which is good, but grasped only once per method) — more data + object diversity could better support the authors’ claims.
- Parts of the methods section need editing for clarity. In particular, the actual training process for the depth reconstruction is unclear — the authors say simply they generate a “digital twin dataset” and learn a residual between the simulated point clouds (treated as ground truth) and real ones. The “digital twin dataset” seems like a core contribution, and more explanation is needed (either in the paper or the appendix) to better understand the core insights behind this process.
- Unclear if the problem of grasp detection is still compelling -- reported success rates are similar to or below those reported by DexNet and other techniques from 10+ years ago. Applying this approach to other, more contact-rich problems, if the benefits of real-to-sim were still present, would provide a more compelling and exciting demonstration of the core idea.

**Summary after Rebuttal Phase:** Overall, I think this paper contains some nice, novel ideas re: real-to-sim and thinks seriously about getting grasp generation to work well with real perception stacks in hardware. I want to commend the authors on performing a DexNet baseline (something I haven't seen in years of reviewing 6DoF grasp papers) -- while they don't significantly outperform it, they *do* outperform it, and clearly beat other 6DoF baselines. Between the improved performance + new baseline + nice algorithmic contributions I will bump my score to a strong accept.
questions: questions: - What are some core limitations of the proposed approach?
- How do you collect the "digital twin" data needed to train the proposed denoiser? 
- If the residual labels are themselves noisy, does this create problems for the method? 
- How much calibration data is needed?
- Do you think the results would hold if only working with RGB images? Can you hypothesize re: how important depth is to the core setup?
- Why is grasp detection a "hard" problem? Is your method (or the other grasp detection methods you considered) able to synthesize grasps for objects, or perform better, than a system like DexNet [1], which reported 95% success rates in large-scale, real-world trials?

[1] Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics, Mahler et al., RSS '17. 

n.b.: My apologies if the last question comes across as pointed -- I would not push to reject your paper over something like this, I am just curious why this subfield remains quite active after an impressive result like DexNet + mid-2010s methods are not compared to or cited within this community.
limitations: limitations: 1
review_summary: review_summary: The particular approach to real-to-sim here (denoising hardware data to match simulation data, to prevent dist. shift) is quite interesting. While the paper is not perfect, I think this algorithmic improvement is of interest to the broader CoRL community.
title: title: Grasp Predictor Method that is Trained in Sim.
paper_summary: paper_summary: A method for predicting 6-DoF grasps of objects through learning a depth map denoiser to predict the residual depth maps. The method also trains a feature enhancer to improve the quality of features to predict the 6-DoF grasps but that has marginal effect.
main_review: main_review: - The method uses Rs2Repairer to improve the quality of depth map and R2SEnhancer to improve the features. The enhancer model basically learns a memory bank during training and at inference time it encodes the local features and enhances the features through cross attention between local features and the memory bank. Majority of benefits come from the Rs2Repairer and the enhancer has marginal effect.

- It seems like Rs2Repairer requires real data + reconstructed meshes are required to train the repairer. Please clarify what is the training data used for Rs2Repairer.

- Missing Comparison: The paper claims that it is the state-of-the-art method for grasp prediction that uses simulation data. Most of the other methods are related to graspnet 1-billion. I think a comparison with M2T2 [Yuan et al, CoRL 2023] as that method is also only trained with synthetic data and transfers to the real world makes a lot of sense. It would strengthen the paper.

- Overclaiming concern: The paper claims that it is the first method that only uses sim data and beating real world methods. However there is a long history of methods using point clouds or 2.5 depth images (e.g. DexNet series) that even led to starting robotic bin picking companies. 

- What are the limitations of the method? Where does it fail?
questions: questions: See Above.
limitations: limitations: 1
review_summary: review_summary: I improved my rating after rebuttal.
title: title: Real-to-sim: a new idea for bridging the gap between real-to-sim generalizations for purely vision based models
paper_summary: paper_summary: In this work, authors proposed a new way to train 6DoF grasp detection models. Real-world annotated grasp dataset is usually very limited in size, as it is costly to label such dataset, and does not scale well. Traditionally, research works have tried to address this issue either via training on noisy simulated data, or via reducing noise in real world data via extrapolating/averaging or smoothing. Authors show that training on noisy data has adverse effect on model performance (Figure 1). Instead, authors propose a new idea of Real-to-Sim (R2S) grasp framework, where they train on noise-less simulated data, and during deployement/inference, enrich the real-world data with (a) R2S-Repairer: reducing the noise and positional drfit, and (b) R2S-Enhancer: enhance the real-world features via matching them with a dataset of simulated features. Via extensive ablation studies, comparison with baselines, and real-world robot demonstrations, authors show that their method performs close to / superior than models trained only with simulated data or even real data.
main_review: main_review: This paper presented a new idea of bridging the gap between real-to-sim generalizations for purely vision based models. The writing and explanation of ideas were very clear. Sufficient experiments including comparison with baselines, ablation studies and real-world robot hardware demonstration video were all very helpful.

R2SRepairer and R2SEnhancer are the key contributions of this paper. The explanation of R2SEnhancer was clear, however, R2SRepairer training dataset and validation procedure needs more details. Following are some of the questions, that are not clear to me:

1. On lines 134-135, the authors mentioned: “The generation of twin datasets are not difficult since it does not need to construct grasp annotations, which is easy to be adapted to different depth cameras”. However, no further information was provided on how this “twin” dataset was actually generated. From the best of my knowledge, the generation process would look something like this: authors will set up a static scene in real-world and capture its RGB-D images from multi-view calibrated cameras. Then, somehow combine these multi-view images to get noise-less depth maps. I understand that this is easier than getting pose and grasp labels for all the objects in the scene. However, I am not sure, how multi-view images are combined? Also, I can imagine that multi-view images will help reduce the noise, but not sure how it will help with the positional drift?

2. In experiment section, on line 188-190, the authors provided some details about dataset for R2SRepairer: “use twin datasets based on GraspNet189 1Billion [5] to train our R2SRepairer. The grasping performance is validated using the test set 190 from GraspNet-1Billion”. Does this mean that R2SRepairer was trained on 100 real-world train scenes from the said dataset? How were the ground-truth depth-maps obtained? Also, what was the performance of this module on test set?


Minor Typo on L112,  “Expect the two modules” should be “Except the two modules”.
questions: questions: Please provide more details of training and testing dataset for R2SRepairer. Specifically, how this dataset was collected/obtained? What was the train and test set size? What was the performance on test set?
limitations: limitations: 1
review_summary: review_summary: Authors present a new approach for training grasp detection models in sim, and transforming real-world to be close to sim. The approach seems to work well. More details on datasets generation for R2SRepairer would be very helpful.