Real-to-Sim Grasp: Rethinking the Gap between Simulation and Real World in Grasp Detection
Authors: Jia-Feng Cai
Zibo Chen
Xiao-Ming Wu
Jian-Jian Jiang
Yi-Lin Wei
Wei-Shi Zheng
Venue: CoRL 2024 (2024)
Keywords: Grasp pose detection
simulated datasets
sim-to-real
TL;DR: A sim-to-real mehod for 6-DoF grasp poses detection

Abstract: For 6-DoF grasp detection, simulated data is expandable to train more powerful model, but it faces the challenge of the large gap between simulation and real world. Previous works bridge this gap with a sim-to-real way. However, this way explicitly or implicitly forces the simulated data to adapt to the noisy real data when training grasp detectors, where the positional drift and structural distortion within the camera noise will harm the grasp learning. In this work, we propose a Real-to-Sim framework for 6-DoF Grasp detection, named R2SGrasp, with the key insight of bridging this gap in a real-to-sim way, which directly bypasses the camera noise in grasp detector training through an inference-time real-to-sim adaption. To achieve this real-to-sim adaptation, our R2SGrasp designs the Real-to-Sim Data Repairer (R2SRepairer) to mitigate the camera noise of real depth maps in data-level, and the Real-to-Sim Feature Enhancer (R2SEnhancer) to enhance real features with precise simulated geometric primitives in feature-level. To endow our framework with the generalization ability, we construct a large-scale simulated dataset cost-efficiently to train our grasp detector, which includes 64,000 RGB-D images with 14.4 million grasp annotations. Sufficient experiments show that R2SGrasp is powerful and our real-to-sim perspective is effective. The real-world experiments further show great generalization ability of R2SGrasp. Project page is available on https://isee-laboratory.github.io/R2SGrasp.

Real-to-Sim Grasp: Rethinking the Gap between Simulation and Real World in Grasp Detection
Jia-Feng Cai Zibo Chen 1 Xiao-Ming Wu Jian-Jian Jiang', Yi-Lin Wei!, Wei-Shi Zheng 1,21
1 School of Computer Science and Engineering, Sun Yat-sen University, China
2 Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China fcaijf23, chenzb8, wuxm65, jiangij35, weiylin5 (aayyiin5@@mail @mail2 wtw.ii.....uu.....mmoooom sysu..yuu.uu eduauueddu..o wazheng@ieee.co org
Abstract: For 6-DoF grasp detection, simulated data is expandable to train more powerful model, but faces the challenge of the large gap between simulation and real world. Previous works bridge this gap with sim-to-real way'. However, this way explicitly or implicitly forces the simulated data to adapt to the noisy real data when training grasp detectors, where the positional drift and structural distortion within the camera noise will harm the grasp learning. In this work, propose a Real-to-Sim framework for 6-DoF Grasp detection, named R2SGrasp, with the key insight of bridging this gap in real-to-sim way?, which directly bypasses the camera noise in grasp detector training through an inference-time real-to-sim adaption. To achieve this real-to-sim adaptation, our R2SGrasp designs the Real- to-Sim Data Repairer (R2SRepairer) mitigate the camera noise of real depth maps in data-level, and the Real-to-Sim Feature Enhancer (R2SEnhancer)) en- hance real features with precise simulated geometrie primitives in feature-level. To endow our framework with the generalization ability, we construct large- scale simulated dataset cost cost-efficiently to train our grasp detector, which includes 64,000 RGB-D images with 14.4 million grasp annotations. Sufficient experi- ments show that R2SGrasp is powerful and our real-to-sim perspective is effective. The real- worldworld experiments further show great generalization ability of R2SGrasp. Project page is available on https://see- witpryigiiiib..SSiri......
Keywords: Grasp pose detection, simulated datasets, sim-to-real
Introduction
Grasping objects in unstructured environment is fundamental for robots designed to accomplish various complex tasks[1, 2, 3]. Previous works [4, 5, 6, 7, 8, 9, 10] utilize real-world data to achieve impressive performance. However, collecting large-scale datasets in real world is still challenging, which limits the improvement of the grasp and generalization abilities. To address this problem, simulated data provides feasible alternative, but the grasp performance trained in simulation is hindered by the large gap between simulation and real world when applying to real world scenarios.
There exist a few methods attempting to bridge this gap, which use domain randomization [6, 11, 12] or domain adaptation [13, 14, 15, 16. 17] achieve it. The former adopts various randomized con- ditions make the simulated data closer reality, while the latter narrows the gap by aligning feature distribution between two domains. However, they both include sim-to-real way that adapts the simulated distribution to the real-worte one, which explicitly or implicitly introduces the camera noise in real data when training the grasp detector. This noise appears as positional drift and struc- tural distortion as shown in Figure 1 (a), disrupting the training process of the grasping detector that
Corresponding author:
"sim-to-real way" refers to making simulated data features more similar to real-world data features. ""real-tossim way" involves making real-world data or features more similar to simulated data or features. Our real-to-sim process bypasses the camera noise in grasp detector training and obtains great performance.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

ccurate mesh points
Points from depth map
70
Distorte structure
Drift posiiion
GF PRICISION
60
50
21.94
10
0
w/o noise
noise
(a) Real-word point clouds with mesh
Upper bound of grasp performance
Figure 1: Illustrations of problems related to the gap between simulation and real world. Figure (a) shows mixed point clouds, including single-view point clouds of the scene and point clouds sampled from accurate object meshes. There are positional drift and structural distortion in real real-world single- view point clouds which are caused by camera noise in real data. Figure (b) depicts that the camera noise disrupts the training of the grasp detector, as the average precision of the grasp detector trained in point clouds with real-world noise is lower than that trained in noiseless point clouds.
highly sensitive to variance in 3D space. To demonstrate the negative effect of camera noise, we compare the upper bound of grasp performance of the grasp detector trained noiseless data and on noisy data depicted in Figure (b). The performance of grasp detector trained in data with real-world noise is lower than that of trained in noiseless data, which suggests that the camera noise introduced by the sim-to-real way will disturb the learning of grasping skills.
To this end, we propose novel Real-to-Sim framework for 6-DoF Grasp detection, named R2SGrasp, to bridge the gap between simulation and real world real-to-sim way. This way adapts real-world distribution to the simulated one in inference phase to leverage the pre- cise grasping skills learned from the simulation, which bypasses the camera noise in detector training and ensures robust performance. Specifically, a grasp detector is trained with noise- less simulation data to attain accurate grasping capability. During inference, we achieve real-to-sim adaptation by using a Real-to-Sin Repairer (R2SReparier) to preprocess the depth map so as to mitigate positional drift and structural deformation, along with Real-to-Sim Feature Enhancer (R2SEnhancer)) enhance real-world local features with simulated structural features. Benefiting from our novel modules, R2SGrasp implements the real-to-sim way in data and feature levels to achieve higher performance.
Moreover, benefiting from our real-to-sim perspective, our grasp detector only requires training in simulated data, which endows our framework with the expandable generalization ability by scaling up the simulated data cost-effectively Therefore, we build a large-scale simulated dataset, named R2Sim, including 256 daily objects, 500 cluttered scenes and 64,000 RGB-D images with total 14.4 million grasp annotations. To improve the efficiency of training such a large-scale simulated data, we propose simplified annotation strategy, resulting in a reduction on training time by 73%.
We conduct extensive experiments on the GraspNet- Billion dataset [5] to verify the effectiveness of our R2SGrasp. R2SGrasp achieves superior performance in real world only with simulated data, surpassing methods of using annotated real-world data. Moreover, we further evaluate R2SGrasp through many real- wolld grasping experiments, verifying the great generalization of our framework.
2 Related Work
Sim-to-real transfer in grasp detection. The gap between simulation and reality a significant problem in robotics grasping. For those
grasping methods, most works utilize advanced image processing techniques, such as adversarial training [13, 14], GAN-based methods [17, 18) or teacher-stundent models [15] to bridge the gap between simulated and real RGB images or features. We focus the transfer of point cloud- based grasping methods from simulation to reality. Fang et al. [6] adds Gaussian noise to the point cloud for training. Zheng et al. [16] aligns the features

---

Inferencephess
R25T niannar
Depti
Cross attention
projettoe
ratractor
R2S Repairex
Menrory bark
Roal world point cloud
Sim. feeture
Greep pois
RGB
Training of R22Hugaair
Training of graep detector with R25FEhaancer
R2SEnbsmner
MLP
Deppi
R2S Repairer
Festure
Cross attention
Updao
Residual
pointcloud
...
RGB
Represenbboo space
Mersey back
Grasp poses
Figure 2: Overview of R2SGrasp framework. In inference phase, the Real-to-Sim Data Repairer (R2SRepairer) repairs depth map from RGB-D input, then feature extractor extracts local fea- tures from the single-view point cloud which is transformed from the repaired depth map. Then Real-to-Sim Feature Enhancer (R2SEnhancer) enhances the real features using the stored simulated structural features and finally predicts the grasp poses. In training phase, we train the R2SRepairer on twin datasets and train the grasp detector with R2SEnhancer on our R2Sim dataset.
between the real and simulated domain through adversarial training. The above sim-to-real methods in point clouds enable the grasping model to handle the noise at the data and feature levels, but the introduction noise during training will reduce the upper bound of the performance of grasp detector. In contrast, our method based on point clouds makes the real data and features closer to simulation ones.
Grasp datasets in simulation. Simulation datasets [18, 19, 20) are the most commonly used data for robot learning. Previous works [21, 22, 23, 24] propose the datasets that define grasp poses within the continuous grasp sample space, which have insufficient annotations, challenging to cover the entire sample space. Recently, [5] uses discrete annotations fully cover the entire sample sample so to learn the robustness of grasp abilities. Following that, decouple the grasp pose parameters and discretize them at fixed intervals within the parameter space. Our annotations rich enough to cover the entire discrete parameter space. Due to the demand for large-scale simulation training. we also propose simplified annotation method to enhance training efficiency. Compared to previous simulated datasets, our dataset employs a discrete representation of grasp poses, and provide more dense grasp annotations that cover the entire sample space, which enables our model to achieve great generalization ability.
3 R2SGrasp: A Real-to-Sim Framework for 6-DoF Grasp detection
3.1 Task Definition
We first describe the task of 6-DoF grasp detection. Given the single-view RGB-D image Irghd E H xW x4, the goal of grasp detector to predict a set of accurate grasp poses in cluttered scenes.
The 6-DoF grasp pose can be denoted as G [R. T, w), where R SO(() the rotation matrix, T R33 | is the translation and is the gripper width. We follow [5] to decouple R into approaching vector and in-plane rotation A, T into grasp point p and approaching distance d from grasp points shown in Figure 2. In this work, we aims to bridge the gap between simulation and real world in 6-DoF grasp detection, where we will train our model in simulation and test it in real world data for employment.
to grasp origin
3.2 Why should we use real-fo-sim adaptation?
Previous works employ the sim-to-real way to bridge the gap between simulation and real world. However, they explicitly or implicitly introduce the camera noise in real data when training the grasp detector. The positional drift and structural distortion caused by camera noise disrupt the training

---

process of the grasp detector, limiting the grasp performance, as depicted in the right hand side of inference phase Figure (c).
To solve the above problem, we propose a novel real-to-sim perspective that bridges the gap between simulation and real world in real-to-sim way. This way is able to train a robust grasp detector noiscless simulated data to avoid the interference of noise in grasping skills learning, thereby achieving stronger grasping capabilities. Although this grasp detector performs exceptionally well on noiseless simulated data, its performance degrades on real data due to the presence of noise. To apply the precise grasping ability learned from simulation to the real world, we consider adapt- ing the real world data and feature to the simulated ones. Specifically, we introduce the R2SGrasp framework, including Real-to-Sim Data Repairer (R2SRepairer) and Real-to-Sin Feature En- hancer (R2SEnhancer) to achieve real-to-sim adaptation data and feature level respectively. The overview of R2SGrasp is shown in Figure 2. Except the two modules, framework also consist of grasp detector to generate the aggregated features of the grasp points and MLP to output the grasp widths and grasp scores for each grasp candidate along the approach direction, following [8]. We adopt the same loss function as [8] to train our grasp detector. More details about our grasp detector and loss function can be seen in Appendix A.2. How the two modules adapt the real world data and feature to the simulated ones will be detailed below.
AVVERA CIE PRECISION
80
60
baseline
23.61
20
0-10mm
10-20mm
20-40mm
40-80mm
10-150mm
Before repair repir
(a)
(b)
Figure 3:
Illustrations on the impact of camera noise. (a) shows the point cloud and noise map, where different colors in the noise map represent different noise amplitude ranges, with amplitude measured in millimeters. (b) shows the performance difference before and after depth map repair using ground truth, and the green line presents the real-world training performance.
3.3 Real-to-Sim Data Repairer
Impact of camera noise. We find that the gap between simulated and real domain primarily stems from camera noise in real data which appears the positional drift and struetural deformation in point clouds. As shown in Figure 3 (a), there is severe noise in depth map captured from the real world. To explore the impact of this noise, we conduct verified experiment that simply replacing the severe noise regions in the real depth map with accurate depth values to repair the structure. As shown in Figure 3 (b), the average precision increases by 23.61 %, even surpassing the baseline [8] whose model trained on real dataset. The results suggest that if we can mitigate this noise and repair the structure, the grasp performance can be improved significantly.
Mitigating positional drift and structural deformation. We propose the real-to-sim data repairer (R2SRepairer) to mitigate positional drift and structural deformation by reducing the noise ampli- tude using an encoder-decoder architecture. It takes RGB-D images Trght E H HW x4 as input and outputs residual map I, E H The repaired depth map La can be obtained Eq. (I):
fa(i.)) = La(i.) 136
(2)
To obtain supervision data I; we use the object models and object poses from GraspNet-1 Billion [5] efficiently generate the simulated data paired with the real ones in Blenderproc [25]. By setting the camera position in the simulation environment to match the real camera pose, we can render simulated depth maps that are identical to the real depth maps from GraspNet- CraspNet-BBillionn Ie

---

is calculated as Eq.(2), where la and I, are paired simulated and real-world depth maps. The R2SRepairer is trained with Smooth L.1 loss.
3.4 Real-to-Sim Feature Enhancer
To further bridge the gap between the simulation and real world, R2SEnhancer conducts the real- to-sim adaptation at feature level by using the precise structural features enhance the real-world local features based feature similarity. As shown in Figure 2, memory bank in R2SEnhancer cumulates diverse geometric structure information through clustering during training with simulated data. During inference. this simulated structure information is used to enhance the features of real structures, thereby improving the perception of our grasp detector in real geometric structures.
Building the memory bank. Memory bank M [k,}) stores K simulated structure features kj during training. At first, initialize M by sampling from the normal distribution. Given the local features Fi = NW of each batch extracted in simulated data when training, we calculate the distance between local features and stored features based on cosine similarity, and assign each local feature the closest stored feature. We then update the stored features with the average of all assigned feature clusters using momentum update (3);
kj aki + (1 aaff:
(3)
where f is the average feature of the cluster corresponding to kj. and a E [0, 1] is the momentum update parameter. After that, we only keep the centers of clutters for the next-batch training.
Real-world feature enhancement. After training with extensive simulated data, the memory bank stores diverse set of structural features that represent common geometric structures. We retrieve similar simulated structural features enhance the real features through cross-attention mechanism, where the local features FT {f extracted from real-world point clouds serve the queries, and the simulated structural features stored in the memory bank serve as the keys and values. The enhanced real features are obtained using (4)
Fj- (MMW"""" /. FY.
(4)
where P denotes the Softmax function, W W W RCDD are learnable encoding matrices for query, key and value in cross pross-attention mechanism.
4 R2Sim Dataset
Motivation of the R2Sim Dataset. Thanks to our real-to-sim perspective, our grasp detector only needs simulated data for training. This endows our framework the expandable generalization abil- ity by scaling up the simulated data cost-effectively. Towards this end, we construct a large-scale dataset named R2Sim in Blenderproc [25], including 256 daily household objects selected from GSO [26] and GraspNet-1Billion [5], 500 cluttered desktop scenes, 64,000 RGB-D images taken from different views and approximately 14.4 million grasp annotations. Each frame is also anno- tated with object segmentation map, object poses, camera pose, graspness heatmap [8] and so More information about our R2Sim is available in Appendix A.3.
Scene generation. We provide an automated process for generating scene data with various ran- domization techniques to enhance the diversity, such as rich background textures, diverse lighting conditions, cluttered table scenes, and various random camera views. This process automatically generates RGB-D images, segmentation masks, 6D object poses, and camera poses, providing rich training data for the grasp detector. More details are available in the Appendix A.3.3.
Grasp pose annotation for efficient training. Given the object meshes, we use a discrete dense annotation method [5] for object grasp pose labeling, where the grasp points are densely and uni- formly distributed on the object surface, with each grasp point having 300 approach directions and 48 grasp candidates per direction. Scene grasp pose annotations can be obtained by projecting the grasp annotations of objects according to their 6-DoF poses. As the data scale expands, the dense
5

---

annotations significantly increase training burden, including prolonged data loading time and exces- sive memory consumption. To improve training efficiency, we simplify the grasp pose annotations by reducing the number of negative samples. Specifically, we first filter out grasp points that do not have successful grasp candidates. Then, for the retained grasp points, we select the top 60 approach- ing vectors from the original predefined 300 vectors based on the proportion of successful grasp candidates. Benefiting from simplification, we filter out many invalid grasp annotations, which improves the training efficiency and is more suitable for large-scale simulation data training. More details and further experimental validation are illustrated in Appendix A.3.4.
Experimental Results
5.1 Experimental setup and details
Datasets. We use R2Sim dataset to train the grasp detector and use twin datasets based on GraspNet- 1Billion [5] to train our R2SRepairer. The grasping performance validated using the test set from GraspNet- BBallint-BBillion. 100 scenes allocated for training and 90 for testing. The test set further divided into three subsets: seen, similar, and novel, where each category contains 30 scenes.
Billion. GraspNet- PraspNet--Billionn IBillion is a real-world dataset comprising 190 scenes, with
Metrics. We employ AP,, and AP as the evaluation metric for grasp performance, the metrics introduced by [5]. AP, represents the average precision of the top 50 grasp poses in terms of grasp scores under different friction coefficient μs. AP denotes the mean of AP,-
Implementation Details. The backbone of R2SRepairer implemented using a UNet [27] archi- tecture with ResNet34 [28] encoder, outputting feature vectors with 32 channels. The feature extractor in our grasp detector adopts the cascaded graspness model in GSNet [8] with cylinder grouping operation. The cross-attention mechanism consists of four heads, with model dimension of 256. The storage capacity of the memory bank K 120, with the momentum update parameter a = 0.999. Additionally, R2SRepairer trained with a learning rate of 0.001, using the Adam W optimizer and a batch size of for 30 epochs. And the grasp detector trained with learning rate of 0.001, using the Adam optimizer and batch size of for 30 epochs.
5.2 Experiments on GraspNet- -Biliin
Comparison with sim-to-real methods. We first compare our method with current sim-to-real approaches as shown in Table (the methods without using labeled real data). Source-only trains the grasp detector in simulted data without any sim-to-reat re-ll techniques. ADD-Noise [6] adds Gaussian noise to the simulated point cloud. Global-DA (29) and Local-DA [29] utilize adversarial domain adaptation methods to align the distributions of the simulated and real domains at global and local feature levels respectively. Note that the above methods utilize the same grasp detector [8] ours for fair comparison. It evident that method significantly surpasses others by 9.19/18.64 64,44, 3.65/7.62, 5.07/10. 14 AP at least in the seen, similar, and novel categories respectively. Additionally, the results of previous methods have little improvement, even decline on grasp performance. We think the reason is that these works explicitly or implicitly introduce the camera noise in real data when training the grasp detector, which disturbs the learning of grasping skills.
Comparison with methods trained in real world. To further illustrate the advantages of R2SGrasp, we also compare our method with others [5, 9, 8, 7, 4] trained real-world datasets, as shown in Table 1 (the methods using labeled real data). The performance of our method is close to that of models trained on real data, and it even surpasses current SOTA methods in most metrics. Specifically, compared to GSNet [8), our method outperforms it by 4.93, 7.08, and 5.73 AP in the seen, similar, and novel categories on Kinect data. The results demonstrate that our R2SGrasp can leverage the precise grasping skills learned from the simulation in the real world.
Ablation Study. We conduct ablation study to validate the effectiveness of each component in method as shown in Table 2. Line 1 is the control group, which is trained on real-world data. When

---

Usig
nal
Methods
AP
Alax
Seen
Simear
APr.
Novel
AB
(irapeeshuuslle]]
27.5099.8
15951911
14331661
3OSSIT
11:5012:2
5.9075566
TransGrasp99
3981/35.5
47.54/91.00
953223136
29,32/29 21
348099
25 19.19/4119
13881141
12.11/44
7.675,84
GSNett8)
68.7061.15 19
76.25/11.46
61,0856.04
S5.2697 39
65 1104556.0
45.770.43
2198/1901
2993/23.73
14,0510000
H0G017
59.3690.26
$1,2068.90
22 1718844
(1RF-Cimppd)
.85. 19
-75.37
-89.22
-27.0 80
-13.53
Source only
5888546.77 72
06.66155.67
94.2638.92
$5,2965888
648156441
46,63739,22
21 2513000
26.34/15.55
10,8977399
ADD Notse(s)
55.5247.45 48
67.24/6
45.57/00111
$5.1 05
64. 325811
45 15.4039.01
21 241441444
26.88/18.67
10,51/7.35
Global DA[29
47.16942.22
57.31/5089
58.22/36.22 22
151800000 65
S4
78
35 55.47/22550
17 2:22222227
22.39/15.57
8.6246.1 18
Local DA
53.21485
63.90/4
0723
88663
07252.1
46 1644366.01
25.55/17.00
10.9077.5
Ours
64.7186.11
76.00/77.89
00099
56.9854477
66.2163300
52.78/49.08
26.31/24.44
32.70/3081
15,611412
Table 1: Performance comparison on real-world data captured by Realsense/Kinect.
RGB
After Regairr Before Repair
0-10 mm
10-20 mm
20-40 mm
40-80 mm
80-1 50mm
150 mm
Figure 4: Comparison of camera noise before and after R2SRepairer. Different colors the noise map represent different noise amplitude ranges, with amplitude measured in millimeters
training solely on R2Sim, there remains noticeable performance gap compared the control group. After integrating R2SRepairer (see Line 3), the grasp performance increases significantly and surpasses the grasp detector trained in real-world data. This performance increase steps from mitigating the camera noise, which can be seen in Figure 4. When further adding R2SEnhancer (see Line 5), we notice an increase in AP across the seen, similar, and novel categories by 3.67, 1.63, and 1.49 respectively. These results indicate that our real-to-sim approach bridges the gap between the simulation and real world, where the R2SRepairer and R2SEnhancer modules achieve real-to-sim adaptation at the data and feature levels respectively.
8
R2SRequirer
R2SFahancer
AP
AP..
APp
AP
Similar
AP,.
APp.4
AP
APr.
Novel
APs.
61.19
71.46
56.04
47.39
56.78
40.45
19.01
23.75
10.60
62.45
74.67
53.53
63.58
29.02
47.77
55.92
11.45
47.96
56.02
42.12
15.76
19.95
8.60
66.12
77.59
59.17
54.47
63.90
49.08
24.74
30.81
14.12
Table 2: Ablation study on two components. The table shows the results on data captured by Kinect. 'R' and 'S' mean the real-world training data and simulated training data respectively.
Advantages of expandable simulated data. To verify the advantages of expandable simulated data, we train R2SGrasp on different number of scenes selected from R2Sim and test on real-world dataset as shown in Fig- ure As the number of simulated scenes increases from 100 500, the top-1, top-10, and top-50 grasp poses in all test scenarios increase by 6.24, 5.51, and 5.14 AP re- spectively. The results highlight the benefits of simulated datasets, as we can easily use open-souree object meshes in simulated environment to scale up the simulated data for training, thereby continuously improving the perfor- mance of the grasp detector.
$0.0
Average Predision
$7.5
$5.0
$2.5
50.0
47.5
45.0
top-10
Top 50
42.5
200
Scene size
Figure 5: number of scenes. Top-N represents the N grasp poses with the highest scores.
Experiments with different
Memory bank size in R2SEnhancer. Table 3 illustrates the effect of memory bank size adjustment on grasping performance. As the value of K increases, the grasp performance improves at the
7

---

beginning, while reducing after K reaches 120. We believe that large value of K means more structural features are stored in memory bank, but too large K leads to information redundancy. We experimentally find that the optimal value K is 120.
K
similar
novel
Object
GSNet (real)
ADD-Noise [6] (sim)
Ours (sim)
Anemmpe
SR
Altempp
SR
Attempt
SR
150
47.93
65.32
54.34
24.12
2.1..1922.23311
48.44
66.12
54.47
24.74
8,14418,,0,3333,,6
100%
8
87.3%
100%
10,15,17 28,124
10
8016
12
66.7%
8
100%
90
48.22
65.79
53.97
24.90
3,4.6.1.21.26.37
60
47.08
64.04
53.20
24.00
3.9.13.20.22.2.399
87.5%
7
100%
100%
2,7,12, 16.12,16,22..00 22,23,55,34
10
80%
10
80%
8
100%
30
45.78
63.63
50.60
23.11
Table 3: Experiment on memory Table 4: Results of grasping experiment in real world. "real" and "sim" mean that training in real-world data and simulated data respectively.
bank size adjustment.
RlvelSenae D455 ('we-fizge fingl grigerr
10
11
18
Failed grasp
Sucresfull Fu
Stirrage bos
20
28
29
30
(a) System setup
(b) Daily obj elj
(c) Failed grasp of GSNet
(d) Successful grasp of R2SGrasp
Figure 6: Description of real-world experiments. (a) shows the experimental setup and environment. (b) displays the objects used in the experiments. (c) and (d) show the failed grasp of GSNet and the successful grasp of R2SGrasp on the same object in the same scenario. Zoom in for better view.
5.3 Real Grasping Experiments
To further verify the effectiveness and generalization ability of R2SGrasp, we conduct real-word experiments on the Franka Emika Panda robotic arm with an Intel RealSense D455 camera and two-finger gripper as illustrated in Figure 6. The experiments are performed six cluttered scenes, each containing 6-8 randomly placed daily objects. We execute the best grasp pose predicted by grasp detectors for each attempt until all the objects are taken away. The grasping capability is measured by the success rate (SR), defined as the proportion of successful grasps to the total number of grasp attempts. We compare the performance of the grasp detectors [8] trained on GraspNet- 1Billion dataset [5]. R2Sim dataset with added noise [6] and R2Sim dataset with our method, as than
shown in Table 4. We find that the grasp detector trained real data has lower success grasp detectors trained in simulated data, due the insufficient amount of real data, which includes only 40 objects, limiting the generalization ability. Using large-scale simulation data improves the grasping success rate, and the success rate can further increase with R2SGrasp. This indicates that our method can effectively leverage grasping skills learned from simulation data in the real world.
Conclusion & Limitations
Conclusion. In this work, we introduce the R2SGrasp framework from a real-to-sim perspective to bridge the gap between simulation and real world in 6-DoF grasp detection. our framework, we train a robust grasp detector in simulated data, and adapt real world to simulation both data and feature levels utilize the skill learned from simulation. Moreover, large-scale simulated dataset is constructed cost-effectively to enhance the generalization of methods. By large-scale training and real-to-sim adaptation, the real world generalization ability actually gains great improvement.
Limitations. Grasping flat objects or complex shaped objects suboptimal, and it is a significant challenge in the current 6-DoF grasping community. We think that our real-to-sim idea can help address this issue by generating data in simulated environments that include these types of objects, allowing the network to learn how grasp them. In the future, also plan to collect these objects and expand the dataset size further tackle these challenges.

---

Acknowledgments
This work was supported partially by NSFC(92470202, U21A20471), Guangdong NSF Project (No. 202381515040025.
References
[1] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems, 2023.
[2] Y. Fan, X. Li, K. Zhang, C. Qian, F. Zhou, T. Li, and Z. Huang. Learning robust skills for tightly coordinated arms in contact-rich tasks. IEEE Robotics and Automation Letters, 2024.
[3] B. Zhou, H. Zhou, T. Liang, Q. Yu, S. Zhao, Y. Zeng, J. Lv, S. Luo, Q. Wang, X. Yu, H. Chen, C. Lu, and L. Shao. Clothesnet: An information rich 3d garment model repository with simu- lated clothes environment. In IEFE/CVF International Conference on Computer Vision, 2023.
[4] Z. Chen, Liu, S. Xie, and W.-S. Zheng. Grasp region exploration for 7-dof robotic grasping in cluttered scenes. JEEE/RSJ International Conference on Intelligent Robots and Systems, 2023.
[5] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Graspnet- Ibillito A large-scale benchmark for general object grasping. In IEEE/CVF conference computer vision and pattern recognition, 2020.
[6] H.-S. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y. Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 2023.
[7] S. Chen, W. Tang, P. Xie, W. Yang. and G. Wang. Efficient heatmap-guite guinddd 6-dof grasp detection in cluttered scenes. IEEE Robotics and Automation Letters, 2023.
[8] C. Wang, H.-S. Fang, M. Gou, H. Fang. J. Gao, and C. Lu. Graspness discovery in clutters for fast and accurate grasp detection. In IEEE/CVF International Conference on Computer Vision, 2021.
[9] Z. Liu, Chen, Xie, and W.-S. Zheng. Transgrasp: multi-scale hierarchical point trans- former for 7-dof grasp detection. In IEEE International Conference Robotics and Automa- lion, 2022.
[10] X.-M. Wu, J.-F. Cai, J.-J. Jiang, D. Zheng, Y.-L. Wei, and W.-S. Zheng. An economic frame- work for 6-dof grasp detection. arXiv preprint 2rX1:2400.008666, 2024.
[11) Q Dai, J. Zhang, Q. Li, T. Wu, H. Dong, Z. Liu, P. Tan, and H. Wang. Domain randomization- enhanced depth simulation and restoration for perceiving and grasping specular and transparent objects. In European Conference on Computer Vision, 2022.
[12] J. Huber, Hélénon, H. Watrelot, F. B. Amar, and S. Doncieux. Domain randomiza- tion for sim2real transfer of automatically generated grasping datasets. arXiv preprint arXiv:2310.04517 2023.
[13] K. Fang, Bai, S. Hinterstoisser, S. Savarese, and M. Kalakrishnan. Multi-task domain adap- tation for deep learning of instance grasping from simulation. In IEEE International Confer- ence Roboties and Automation, 2018.
[14] H. Ma, R. Qin, B. Gao, D. Huang, al. Sim-to grasp detection with global-to-local rgb-d adaptation, arXiv preprint arXiv:2403.11511 2024.

---

[15] K. Chen, R. Cao, S. James, Y. Li, Y.-H. Liu, P. Abbeel, and Q. Dou. Sim-to-real 6d object pose estimation via iterative self-training for robotic bin picking. In European Conference on Computer Vision, 2022.
[16] L. Zheng, W. Ma, Y. Cai, T. Lu, and S. Wang. Gpdan: Grasp pose domain adaptation network for sim-to-real 6-dof object grasping. IEEE Robotics and Automation Letters, 2023.
[17] K. Rao, Harris, A. Irpan, S. Levine, Ibarz, and M. Khansari. Rl-cyclegan: Reinforcement learning aware simulation-to-real. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[18] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, Pastor, K. Konolige, al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. IEEE international conference on robotics and automation, 2018.
[19] C. Eppner, A. Mousavian, and D. Fox. Acronym: A large-scale grasp dataset based on simu- lation. In IEEE International Conference on Robotics and Automation, 2021.
[20] M. Gilles, Y. Chen, T. R. Winter, E. Z. Zeng, and A. Wong. Metagraspnet: A large-scale bench- mark dataset for scene-aware ambidextrous bin picking via chysics-based metaverse synthesis. In IEEE International Conference on Automation Science and Engineering, 2022.
[21] X. Yan, J. Hsu, M. Khansari, Y. Bai, A. Pathak, A. Gupta, J. Davidson, and H. Lee. Learning 6-dof grasping interaction via deep geometry-awart 3d representations. In IEEE International Conference on Robotics and Automation, 2018.
[22] D. Kappler, J. Bohg, and Schaal. Leveraging big data for grasp planning. In IEEE interna- tional conference on robotics and automation, 2015.
[23] D. Morrison, P. Corke, and J. Leitner. Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. IEEE Robotics and Automation Letters, 2020.
[24] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In JEEE/CVF international conference on computer vision, 2019.
[25] M. Denninger, D. Winkelbauer, M. Sundermeyer, W. Boerdijk, M. Knauer, K. H. Strobl, M. Humt, and R. Triebel. Blenderproc2: A procedural pipeline for photorealistic rendering. Journal of Open Source Software, 2023.
[26] L. Downs, A. Francis, N. Koenig, Kinman, Hickman, K. Reymann, B. McHugh, and V. Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items.
In IEEE International Conference Robotics and Automation, 2022.
[27] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer- intervention, 2015.
[28] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition, 2016.
[29] M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 2018.
[30] L. Yang, Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao. Depth anything v2. arXiv:2406.09414, 2024.
[31] C. Choy, J. Gwak, and S. Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
10

---

A Appendix
A.1 More experiments
Further demonstration of the ad- vantages of real-to-sim way. Our R2SRepairer can mitigate camera noise which is beneficial for the grasp detector. To further validate our real- to-sim perspective, we also conduct experiment that adding R2SRepairer the grasp detector [8] trained on real-world data. As shown in Table S1, after adding R2SRepairer (see Line 2), the grasp performance shows modest improvement, but it still lags significantly be- hind our method that is trained on simulated data. This demonstrates that a large amount of simulated data can train more robust grasp detector, and by utilizing our real-to-sim method, this capability can be effectively applied to real-world data.
Method
All
Seen
Similar
Novel
GSNet [8] wo R2SRepairer
42.33
61.19
47.39
19.01
GSNet [8] w R2SRepairer
43.05
61.39
48.05
19.72
Ours
48.44
66.12
54.47
24.74
Table S1: Average precision comparison on real-world data captured by Kinect.
RGB
Alber Desper Reloin Repai
40
RGD
After
0-10mm
10-20mm
20-40mm
40-80mc
80-150mm
150mm
Figure S1: Comparison of camera noise before and after R2SRepairer. Different colors in the noise map represent different noise amplitude ranges, with amplitude measured in millimeters
Quantitative analysis of R2SRepairer. We use the depth
Prediction
Input
Output
RMSS(mm)
Depth Anything V2 [30]
RGB
Depth value
2016
maps from GraspNet-1Billion [5] test set to test R2SRepairer, which consists of 22950 samples. The eval-
Ours
RGB
Depth value
220
Ours
RGBD
Depth value
8.55
Ours
RGBD
Depth residual value
7.91
Table S2: Comparison of depth repair Performance.
uation metric is the root-mean-squan -quare error (RMSE) of the predicted depth map, measured in millimeter. The results showed in Table S2
indicate that if the network only deals with RGB images, the depth value prediction is not accurate, which reduces the grasp performance. Moreover, predicting the residual values more effective than predicting the depth values.
Qualitative analysis of R2SRepairer. We futher demonstrate the effectiveness of R2SRepairer by presenting camera noise before and after refinement. As shown in Figure S1, it is evident that

---

Before Repair
After Repair
CHEEZZY
Before Repair
Skat Hoiee
After Repair
WEtER
W000
Figure S2: Visualization of single-view point cloud before and after repair. Zoom in better view.
the correct noise image significantly reduce noise compared to the initial noise image. To further demonstrate the performance of R2SRepairer, we compared the single-view point cloud before and after camera noise repair. As shown in Figure S2, after noise repair, the positional drift and structural deformation of the point cloud are mitigated, bridging the gap between real and simulated data.
The impact of noise in Residual labels. R2SRepairer is robust the noise in residual labels. As shown in Table S3, add Gaussian noise with mean of 0 and differ- ent variance (std) to the residual labels. The overall AP value increases after adding Gaussian noise, with a more significant increase when std=10. We believe that intro- ducing appropriate noise can enhance the robustness of model learning.
Std
All
Seen
Similar
Novel
48.44
66.
54.47
24.74
48.84
67.29
54.53
24.69
10
49.67
67.81
55.93
25.25
Table S3: Experiment on noise adjust- ment in residual labels.
Qualitative analysis of structural features in memory bank. Our Real-to-Sim Feature Enhancer (R2SEnhancer) uses the precise structural features stored in the memory bank to enhance the real- world features. To visually demonstrate the semantic information of the stored structural features, we visualize the structures represented by these features. First, we calculate the cosine distance between the features of each graspable point and the stored features in the memory bank, assigning each graspable point to the nearest stored structural feature. Then, obtain points for each stored structural feature. Based on cosine similarity, select three distinct features from the memory bank and visualize their corresponding sets of graspable points. As depicted in
of graspable
12

---

Blue
Cyan
Yellow
Figure S3: Semantic information of the stored structural features. "Blue", "Cyan", "Yellow" repre- sent the structures corresponding to the three selected structural features.
Sean
Similar
Novel
Figure S4: Top-30 grasp poses predicted by R2SGrasp on the test set of GraspNet- 1Billinn. The red gripper indicates successful grasp pose, while the blue gripper indicates failed grasp pose.
Figure S3, there are obvious differences the structure represented by the three features. The first feature represents sharp object structures, shown in the first row of Figure S3, which commonly found on toy legs and heads. The second feature represents planar object structures, as depicted in the second row, primarily seen square boxes. The third feature represents curved object structures, as illustrated in the last row, appearing on various curved surfaces, with bottles being the most prominent.
Qualitative analysis of grasping performance. To demonstrate that our R2SGrasp can adapt to real data, R2SGrasp to predict grasp poses the GraspNet-1Billins test set and visualize the results, as shown in Figure S4. In seen and similar scenes, R2SGrasp predicts grasping poses
13

---

with success rate close to 100%. In novel scenes, there are some failed cases, which occur due to collisions or grasping at empty locations.
A.2
Implementation details of grasp detector.
Select NA
MI.P pooling
MX
R2S Eahanoer
M 60%
clouts
Cylinder grouing
Local fostures
widths
Figure S5: Grasp detector in details
Architecture design of grasp detector. The grasp detector in details is shown in Figure S5, We first randomly sample N points from the single-view point clouds generated from depth map, and then use a point cloud backbone to extract point-wise features with C1 dimension. The point cloud backbone adopt a Unet [27] architecture with ResNet14 [28] encoder built upon the Minkowski Engine [31]. Followed by a MLP layer, we predict the object point mask and graspness heatmap J, select the graspable points along with their corresponding point-wise features with the shape of M (C1 3), where M is the number of graspable points and 3 denotes the cartesian coordinates of the points. We also select the grasp view of the graspable points from the predefined 300 approaching vectors based on the grasp view scores & which is also predicted by MLP layer. Then, we perform cylinder grouping operation along the grasp view for each grasp point to aggregate the features of neighboring points, followed by the MI.P and max pooling operations to extract local structural features with C2 dimension. Finally, our Real-to-Sim Feature Enhancer (R2SEnhancer) refines the local structural features using the stored simulated features and outputs the grasp scores 89 and widths sw shaped as M 48, where 48 denotes the grasp candidates of the grasp points. For our network, we set N = 20000, M = 1024, C1 512, C2 = 256, G = 16.
Loss Design. The grasp detector trained with the following loss function:
Lg - Loo(o,,, Ln [2] + A2 s,) + Ag (g) + da (Sws si))
where Lo. Ln, Lu, L,, Lu are used to supervise the learning of object points, graspable points, grasp views, grasp scores and grasp widths respectively. I Jh 8 6g s is the ground truth of object point mask, graspness heatmap, grasp view scores grasp scores and grasp widths. L, adopts binary classification loss, while others use regression loss. Due the simplification of our grasp annotations, some grasp poses may lack supervision signals. Therefore, when calculating the loss, ignore any predicted grasp poses that lack supervision signals.
A.3
R2Sim dataset details
The overall process of dataset construction is shown in Figure S6. We start by selecting 256 daily household objects from the Google Scanned Objects dataset [26] and the P train- ing dataset [5]. Then, we generate scenes in blenderproc [25] and simultaneously label the grasp poses of the objects. After that, project object-level grasp annotations into the scenes and de- tect the grasp annotations that result in collisions. Finally, adopt proposed grasp annotation simplification method to remove ineffective grasp poses. The remaining grasps serve as scene-level annotations. To sum up. our R2Sim dataset comprises 500 scenes with 76,800 RGB-D images. Each scene contains approximately 14.4 million grasp annotations and each frame in every scene is also annotated with object segmentation maps, 6-DoF poses of objects and camera, graspness heatmap and view graspness. In the graspness heatmap, brighter areas indicate higher likelihood of suc- cessful grasps. Similarly, higher values of view graspness also represent greater probability of successful grasps. Some examples of RGB, depth map, segmentation map and graspness map shown in Figure S7.
14

---

Selecting objects from open-source object datasets.
Object-level grasp pose label
Scene level grasp pose label
10000
Render desktop scenes to obtain RGB, depth, and segmentation simulation
Detect collisions and Simplify
annotations
Point
Titrr
outer
surfoce
Vector fileer
surfhoo
tnner
Figure S6: Overview of data generation pipline.
A.3.1 Details of object level grasp annotation.
approach to annotate the grasp poses of objects. Grasp poses are determined by downsampling high-quality mesh models to ensure that the grasp points are evenly distributed in the voxel space. For each grasp points, 300 approach directions are sampled uniformly on spherical space. Grasp candidates of each approach directions are explored on a grid defined by gripper depths and 12 rotation angles. To sum up, there 48 grasp candidates along each approach direction and 14,400 grasp candidates each grasp point. The gripper width is adjusted as necessary to prevent empty grasps or collisions. We adopt analytic computation method as [5? to grade the sampled grasp poses. The grasp scores range from 0 to 1, with higher scores indicating a greater likelihood of successful grasps.
We use a sampling-evaluatin
A.3.2 Details of scene level grasp annotation.
Using the object poses, the grasp pose in object coordinate system is projected onto the world coor- dinate system. We detect collisions for the projected grasp poses in the scene and set the scores of those that collide to zero. Assuming there are grasp points projected into the scene, we calculate the success rate of grasp candidates at each point, resulting in N graspness values. Using the cam- era's intrinsic parameters, we convert the depth map into a single-view point cloud. We then use the K-NN algorithm to match the grasp points in the scene, assigning each point in the point cloud the corresponding graspness value. This value is then back-projected into the image create the graspness heatmap. Similarly, we calculate the success rate of grasp candidates along each approach direction and using this the view-graspness. We also make further simplification of grasp pose annotation to improve training efficiency.
A.3.3 Scene generation.
We automated the construction of cluttered desktop scenes using Blenderproc [25]. We first con- struct a simple indoor scene with table placed the center. Textures for the table, floor, and walls are chosen randomly from specific material categories provided on AmbientGG. The lighting setup of the scene randomized as well, with practical adjustments to intensity and variations in light color to improve visual clarity and accuracy. Then, we choose a variable number objects ranging from to 10 from our object pool and place on the table. To create sufficiently cluttered scenes, we place the objects 1.5 meters above the table and allow them fall naturally onto the surface. Finally, set 128 camera poses to capture RGB-D images from multi views, where the poses randomly sampled on the upper hemisphere, with a radius of 1.1 meters centered on the objects' region. Based
15

---

RGB
Depti
Mack
Hostmep
RGB
Depth
Mask
Hoelmap
Figure S7: Display of RGB, depth map, segmentation mask and graspness heatmap in our simulated dataset.
on the above setup, obtain RGB-D images, object segmentation maps, object poses, and camera poses from different angles efficiently.
A.3.4 Analysis of simplified annotation.
To demonstrate the advantage of sim- plified annotation, we compare the sim- plified and non-simplified annotations across multiple metrics. The average precision measures the performance of grasp detector trained 300 scenes selected from R2Sim dataset. Except for the average accuracy, all other met- rics are evaluated on the entire R2Sim dataset. As shown in Table S4, following the simplification of annotations, there are a notable in- crease of 2.26 AP. We believe that this improvement is due the simplified annotations alleviating the imbalance between positive and negative samples present in the original annotations, where positive samples made up less than 2% of the total according statistics. Moreover, after simpli- fying the annotations, the program's 's runtime, memory usage, and GPU usage decrease by 73.36%. 84.51 ,,, and 54.66%, respectively, and the storage usage for the entire dataset annotation decrease by 79.29%. This simplified annotation method crucial for constructing a large-scale dataset, as reduces the burden of data storage and neural network training.
Metrics
Simplified
Non-Simplified
Average Precision($)
35.94
33.68
Run Time(epoch/h)
5.64
21.17
Memory Usage(GB)
7.50
48.41
GPU Memory Usa
4.43
9.77
Storage Usage(GIB)
15
72.44
Table S4: Compare metires between simplified and non- simplified annotations.
16


Reviews:
[{"review_id": "ifFxuHYfJo", "reviewer": "Reviewer_7Q2J", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel framework, R2SGrasp, aimed at improving 6-DoF grasp detection by addressing the gap between simulation and real-world data. Unlike traditional sim-to-real approaches, which adapt simulated data to real-world noise, R2SGrasp employs a real-to-sim method. Specifically, this paper uses R2SRepairer to mitigate camera noise in real depth maps (data level) and R2SEnhancer to enhance real features with simulated geometric primitives (feature level). In addition, this paper proposes a large-scale simulated dataset, R2Sim, containing 64000 RGB-D images and 14.4 million grasp annotations. Experimental results demonstrate that R2SGrasp outperforms other baseline methods that are trained on real-world data and sim-to-real methods.", "summary_of_recommendation": "I recommend weak reject of this paper based on the aforementioned weaknesses. I am happy to raise my score if the authors provide clarifications on the questions I raised and address my concerns in the review.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. What are the computational requirements for training and deploying R2SGrasp on a real robot? How does the performance compare to other methods in terms of processing time and resource usage?\n\n2. How robust is the proposed method in real-time applications with dynamic environments? Some discussion around this will be appreciated. \n\n3. It would be interesting to see the common failure modes of the proposed methods, such as unreachable joint position, gripper collision, bad repaired depth, etc. Some discussions and analysis will make the paper stronger.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8sGr75tXyf", "reviewer": "Reviewer_TS41", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Learning to denoise raw sensor data to match the \"clean\" inputs seen during simulation training can enable effective and non-conservative sim-to-real transfer.", "summary_of_recommendation": "The particular approach to real-to-sim here (denoising hardware data to match simulation data, to prevent dist. shift) is quite interesting. While the paper is not perfect, I think this algorithmic improvement is of interest to the broader CoRL community.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "- What are some core limitations of the proposed approach?\n- How do you collect the \"digital twin\" data needed to train the proposed denoiser? \n- If the residual labels are themselves noisy, does this create problems for the method? \n- How much calibration data is needed?\n- Do you think the results would hold if only working with RGB images? Can you hypothesize re: how important depth is to the core setup?\n- Why is grasp detection a \"hard\" problem? Is your method (or the other grasp detection methods you considered) able to synthesize grasps for objects, or perform better, than a system like DexNet [1], which reported 95% success rates in large-scale, real-world trials?\n\n[1] Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics, Mahler et al., RSS '17. \n\nn.b.: My apologies if the last question comes across as pointed -- I would not push to reject your paper over something like this, I am just curious why this subfield remains quite active after an impressive result like DexNet + mid-2010s methods are not compared to or cited within this community.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l4fSoxekEE", "reviewer": "Reviewer_SR4p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "A method for predicting 6-DoF grasps of objects through learning a depth map denoiser to predict the residual depth maps. The method also trains a feature enhancer to improve the quality of features to predict the 6-DoF grasps but that has marginal effect.", "summary_of_recommendation": "I improved my rating after rebuttal.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "See Above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ECkdVujRdF", "reviewer": "Reviewer_xCby", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this work, authors proposed a new way to train 6DoF grasp detection models. Real-world annotated grasp dataset is usually very limited in size, as it is costly to label such dataset, and does not scale well. Traditionally, research works have tried to address this issue either via training on noisy simulated data, or via reducing noise in real world data via extrapolating/averaging or smoothing. Authors show that training on noisy data has adverse effect on model performance (Figure 1). Instead, authors propose a new idea of Real-to-Sim (R2S) grasp framework, where they train on noise-less simulated data, and during deployement/inference, enrich the real-world data with (a) R2S-Repairer: reducing the noise and positional drfit, and (b) R2S-Enhancer: enhance the real-world features via matching them with a dataset of simulated features. Via extensive ablation studies, comparison with baselines, and real-world robot demonstrations, authors show that their method performs close to / superior than models trained only with simulated data or even real data.", "summary_of_recommendation": "Authors present a new approach for training grasp detection models in sim, and transforming real-world to be close to sim. The approach seems to work well. More details on datasets generation for R2SRepairer would be very helpful.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Please provide more details of training and testing dataset for R2SRepairer. Specifically, how this dataset was collected/obtained? What was the train and test set size? What was the performance on test set?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel framework, R2SGrasp, aimed at improving 6-DoF grasp detection by addressing the gap between simulation and real-world data. R2SGrasp employs a real-to-sim method comprising of R2SRepairer to mitigate camera noise in real depth maps and R2SEnhancer to enhance real features with simulated geometric primitives. The paper also proposes a large-scale simulated dataset, R2Sim, containing 64000 RGB-D images and 14.4 million grasp annotations. 

Reviewers favorably viewed the proposed method and the experimental results. However, the reviewers also raised some concerns: applicability of proposed approach to other more contact rich tasks, comparisons to other related works, some concerns about claims made in the paper, and the lack of many important training details.

The author response addressed many reviewer concerns. `TS41`, `xCby`, `SR4p` reviewed the author response and settled on accept leaning ratings. The AC looked through the paper, the reviews, and the author response. While the residual concern from `7Q2J` (gap in dynamics) is well placed the AC agrees with `TS41` that it may be beyond the scope of the current paper. The authors are encouraged to include the author-reviewer discussion when revising the paper, particularly the points about additional experiments, appropriate framing of claims, and adding a discussion of the limitations.
Decision: Accept