Towards Scale Balanced 6-DoF Grasp Detection in Cluttered Scenes
Authors: Haoxiang Ma
Di Huang
Venue: CoRL 2022 Poster (2022)
Keywords: Grasp Detection
Point-cloud Representation
Scale Balance

Abstract: In this paper, we focus on the problem of feature learning in the presence of scale imbalance for 6-DoF grasp detection and propose a novel approach to especially address the difficulty in dealing with small-scale samples. A Multi-scale Cylinder Grouping (MsCG) module is presented to enhance local geometry representation by combining multi-scale cylinder features and global context. Moreover, a Scale Balanced Learning (SBL) loss and an Object Balanced Sampling (OBS) strategy are designed, where SBL enlarges the gradients of the samples whose scales are in low frequency by apriori weights while OBS captures more points on small-scale objects with the help of an auxiliary segmentation network. They alleviate the influence of the uneven distribution of grasp scales in training and inference respectively. In addition, Noisy-clean Mix (NcM) data augmentation is introduced to facilitate training, aiming to bridge the domain gap between synthetic and raw scenes in an efficient way by generating more data which mix them into single ones at instance-level. Extensive experiments are conducted on the GraspNet-1Billion benchmark and competitive results are reached with significant gains on small-scale cases. Besides, the performance of real-world grasping highlights its generalization ability.

Towards Scale Balanced 6-DoF Grasp Detection in Cluttered Scenes
Haoxiang Ma, Di Huang"
State Key Laboratory Software Development Enviroment School of Computer Science and Engineering Beihang University, Beijing, 10019, China
)uuang)uuua.. (mahaoxiang822, dhuang @buaa edu. cn
Abstract: In this paper, we focus on the problem of feature learning in the pres- ence of scale imbalance for 6-DoF grasp detection and propose novel approach especially address the difficulty in dealing with small-scale samples. A Multi- scale Cylinder Grouping (MsCG) module is presented to enhance local geom- etry representation by combining multi-scale cylinder features and global con- text. Moreover, a Scale Balanced Learning (SBL) loss and an Object Balanced Sampling (OBS) strategy are designed, where SBL enlarges the gradients of the samples whose scales are in low frequency by apriori weights while OBS cap- tures more points on small-scale objects with the help of auxiliary segmen- tation network. They alleviate the influence of the uneven distribution of grasp scales in training and inference respectively. In addition, Noisy-clean Mix (NcM) data augmentation is introduced to facilitate training, aiming to bridge the domain gap between synthetic and raw scenes in an efficient way by generating more data which mix them into single ones at instance-level. Extensive experiments are conducted on the GraspNet- Biillinn benchmark and competitive results are reached with significant gains on small-scale cases. Besides, the performance of real-world grasping highlights its generalization ability. Our code is available https://githut: com/ mahaoxiang822/Scal.... http://aalllaaaaaeed--deeeeeeeee Balanced-Grasp
Keywords: grasp detection, point-cloud representation, scale balance
1 Introduction
Grasping is a fundamental task in robotic manipulation, and grasp detection aims to generate rich gripper configurations objects for stable lift. It is expected deal with objects varying in shape, size. appearance, material, pose etc. Traditional methods usually work with 3D models of objects and policies manually designed [1, 2, 3]. However, these methods highly rely the accuracy of pose estimation and are not applicable to unseen objects, limiting their use in real-world scenarios. With the innovation of depth sensors and the development of deep learning, data-driven methods have been greatly advanced [4, 5, 6], offering the advantage in generalization. More recently, several studies have extended 6-DoF grasp detection a more difficult case, investigating diverse grasping poses for objects in clutter [7, 8, 9]. Although such attempts deliver promising results, they incline large- and medium-scale objects or parts. There exist quite a number of hard examples. As shown in Fig. 1 (a), there are many qualified grasps detected on the large red box in (1), but very few for the bolt in (2) and the fork in (3). For the bottle in (4), there exist several valid grasps on the body while none is on the red head.
Indeed, grasping small objects (e.g. the bolt in (2) and the fork in (3) in Fig. 1 (a)) or small parts of large- and medium-scale objects (e.g. the red head of the bottle in (4) in Fig. (a)) is still low precision and recall. Here, formally define the scale of grasp the minimal width between two fingers of the gripper, which is determined by the pose of the grasp and the shape of the region. To illustrate the correlation between the grasp scale and quality, we visualize the scale distribution of the objects and their corresponding grasp performance by the baseline method on the GraspNet-
Corresponding author.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

IBillion benchmark [8]. As depicted in Fig. 1 (b), current grasp detection method performs well on the objects which seldom have small-scale ground truth grasps. On the contrary, for the objects with a large proportion of small-scale ground truth grasps, the baseline method tends to fail.
10
08
05
04
86
40
(4)
00
Trctiee conficer: wrall - large
Force-clnsure grop
Object ID
Failed graop
Collisioa grasp
(a)
(h)
Figure 1: (a) Grasping objects of varying scales by GraspNet [8]. (b) Grasping Average Precision (AP) and Percentage of Small-scale Grasps (PSG) for different objects.
is really challenging to improve the results of such hard cases towards acale-balanced grasp de- tection. Firstly, for a local region, the ambiguity of its geometry increases when its size becomes smaller since information conveyed reduces. [8, 10, 11] extract local geometry features through spe- cific modules, but due to fixed receptive fields, they not competent distinguish similar but different shapes. Secondly, the scale distribution of grasps for daily objects in cluttered scenes is im- balanced. Large- and medium-scale grasps dominate and thus suppress the learning on small-scale ones. To the best of our knowledge, this issue has not been investigated in 6-DoF grasp detection before. Thirdly, the noise of mainstream depth sensors (e.g. Intel RealSense and MS Kinect) is serious, which degrades shape description, in particular for small regions. [12, 13] provide tentative solutions. Unfortunately, the former applies point completion, mainly working on seen objects be- cause completion of missing regions of unknown objects is confusing without prior knowledge [14], while the latter conducts denoising, which requires rendering raw data by adding noise of differ- ent patterns to clean point-clouds for training models, struggling in generalizing to the complicated real-world noise from RGB-D sensors.
This paper proposes a novel approach to 6-DoF grasp detection, which addresses the difficulties aforementioned in unified framework. Specifically, Multi-scale Cylinder Grouping (MsCG) module is introduced, where multi-scale cylinder features are combined to global feature with additional context embedded for more comprehensive description of local shapes. Furthermore, Scale Balanced Learning (SBL) loss and an Object Balanced Sampling (OBS) strategy are designed for the training and inference phases respectively. SBL enlarges the gradients of the samples whose scales are in low frequency (i.e. small) by apriori weights while OBS captures more points on small- scale objects with the help of an auxiliary segmentation network. They both alleviate the influence of the uneven distribution of grasp scales. Additionally, data augmentation method, namely Noisy- clean Mix (NcM), is presented to facilitate training. It aims to bridge the domain gap between synthetic and raw scenes in efficient way, by generating more data which mix them into single scenes at instance-level.
In summary, our contributions are follows: (1) we introduce an MsCG module to enhance the representation of local shapes for grasp detection; (2) we design an SBL training loss and an OBS inference strategy to mitigate the imbalanced scale distribution of grasps in cluttered scenes; (3) we present NeM data augmentation by blending point-clouds with and without noise at instance-level to improve model training; and (4) we carry out extensive simulation and real- world experiments and reach competitive results with significant gains on small-scale grasps and a strong generalization ability.

---

2 Related Work
Grasping is long-standing task in robotic manipulation. Recently there has been a trend on 6-DoF grasp detection in cluttered scenes. [7] proposes pioneering framework with point-cloud input, which first samples grasp candidates and then scores them with a Convolutional Neural Network (CNN)) [9] employs a point-based neural network score grasp candidates for improvements, Due to the fact that grasp candidate sampling is time-consuming and prone to objects (or parts) of thin shapes, number of studies [15, 16, 8, 10, 17] handle this task an end-to-end manner. [15] troduces a variational auto-encoder to generate grasp poses from point-clouds of single objects. To directly regress grasping clutter, [16] builds single-shot grasp proposal network. [8] generates diverse grasps for objects in clutter by constructing a discrete grasp pose space and predicting the gripper configurations progressively. For more accurate and robust grasps, [17] defines graspness as the quality metric to determine where to grasp and [10] focuses capturing local features from points inside grasp regions. Although the methods above have made large progress, they generally work well on objects (or parts) of large- and medium-scales and have difficulty in tackling small- scale ones, leading to the imbalanced grasping detection performance.
As the overwhelming majority of the studies on 6-DoF grasp detection take point-clouds as input and such data often contain noise depth sensors, there exists another line ameliorate scene represen- tation. [18, 19] bring additional clues to complement description where [18] uses multiple frames to map raw input to Truncated Signed Distance Function (TSDF) smooth noise and [19] integrates texture maps deliver multi-modal features. Besides, [20] introduces implicit representations to model scenes and employs 3D reconstruction as an auxiliary task to refine data. Such solutions ei- ther require more information in inference or highly correlate to the specific model, suggesting the necessity plug and play alternative.
Object Balanced Sampling
N X
Sampling
M x
Only in Inference
Nolse-Clear Mix
Feature
Interpolation
Only Training
Segmentation Network
Grasp Candidate Propusal
X N
M X
Approach
Hend
Graspable
Agpmoach Direction
Point Encoder
Grasp Regression
0.1
Multi-scale Cylinder Grouping
Operation
0.5
Head
0.8
Plane Rotation
Width
Seore
Figure 2: Pipeline of the proposed scale balanced grasp detection approach.
3
Methodology
The pipeline of the scale balanced grasp detection approach is demonstrated in Fig. 2. During training, we first conduct Noisy-clean Mix (NcM) augmentation, which synthesizes clean point- clouds from CAD models and mixes them with the raw noisy ones instance-level. The new point- cloud E J x3 is employed as input, where N is the number of points. After that, M candidate points are sampled by Farthest Point Sampling (FPS) from the input and encoded as seed features IR MxC by a transformer-ban based point encoder [21]. We predict whether these candidates can be grasped and the approach directions of positive candidates using an approach head. To regress other gripper configurations, Multi-scale Cylinder Grouping (MsCG) is built to extract local features for positive candidates from original point-clouds. An operation head is employed to predict the plane rotation, width and score of the grasp. During inference, Object Balanced Sampling (OBS) is used to explore more positive samples different scales, where 3D instance segmentation network integrated to predict the 3D masks E (0, 1) NxK (K is the number of instances) and new positions
3

---

E IR MM are evenly sampled in each mask. The seed features of these new samples are interpolated from the initial ones in scene and used for successive grasp configuration prediction.
3.1 Multi-scale Cylinder Grouping
Considering that grasps correspond to detailed local geometries, it is rather rough to choose fixed Receptive Field (RF) to represent the grasp region. As Fig. (a) displays, a single RF tends to incur ambiguity for the small-scale grasp in position A and miss crucial cues for the large-scale one in position B. To enhance geometric representations for shapes in different sizes, we propose the Multi-scale Cylinder Grouping (MsCG) module, which captures local features of position with varying RFs and combines them to deliver a comprehensive description. Specifically, we set up several cylinders with different radii and crop related points. Radii are evenly distributed within the maximum width of the gripper. Multi-layer Perceptrons (MLPs) and max max- -pooling are employed introduce the to encode the points from different cylinders. To further improve the semanties, corresponding seed features as guidance for local feature fusion at different scales, and a gate module is applied to screen information in them. Fig. 3 (b) shows the structure of the MsCG module. With the help of this module, our network achieves more accurate local geometric representations for grasps at different scales.
Mutti-sctle Cylincers
Cencanieratee
Tocal
MLP
Fodiucs
Mulii scade
Seel Featuee
Local Festures
(a)
(b)
Figure 3: (a) Grasp regions at different scales for local feature extraction. (b) Structure of the Multi- scale Cylinder Grouping module.
3.2 Grasp Scale Balancing
Firstly, we give an analysis on the scale distribution of grasps in cluttered scenes on the training set of the CeospNet-BBillion benchmark [8]. 1,024 points are sampled by FPS in each scene and for each sample, we search for the grasp with the highest score in the SO(3) space, whose gripper width is considered as the scale of the sample. Through this, for all scenes in the training set, we draw the grasp scale distribution, as shown in Fig. 4. It is observed that medium- and large-scale grasps are much more than small-scale ones. This imbalance issue degrades grasp detection in two aspects. First, in training, the model is prone to medium- and large-scale grasps whose percentage is high and thus suppress the learning of small-scale grasps. Second, in inference, few small-scale grasps can be sampled by current evenly sampling techniques in the 3D space such as FPS and voxel sampling. Simply enlarging the number of samples can indeed case this problem, but the memory cost and run-time both sharply increase, making it impractical.
To address the imbalance issue in training, we design Scale Balanced Learning (SBL) loss to weight the errors of the samples at different scales inspired by Cost-Sensitive Learning [22], where the weights are calculated based on the frequencies of the classes of samples. Different from the imbalance in the discrete classification problem, that on grasping scales is continuous and we thus divide the max width of the gripper into T bins. For sample i in the scene, its weight W; is calculated by:
C;
W 1 log
(1)
Cmax
where C is the number of grasps of the same scale with sample i and Cmar is the maximum of the T scales. For negative samples where no successful grasp is in its SO(3) space, we assign W,, = 1. With the loss in [8], we add the weight W to the approach and operation heads:
Loss W[Loss Approach + aLoss Rotation)
Approooh
(2)
where a is trade-off hlyer--paameete Please refer to the supplementary material for more details.

---

To handle the imbalance issue in inference, we propose Object Balanced Sampling (OBS) strat- egy, which adjusts the number of samples on different objects so that sufficient samples are obtained on them. In this case, we first train a 3D instance segmentation network in advance, Lee modified version of [23], only with point-clouds as input to generate masks for objects. We then uniformly carry out FPS each foreground mask and each object has M K samples, where M and K are previ- ously defined as the number of total samples and the number of instances respectively. The features of new sampled positions are linearly interpolated by three nearest neighbors from the original seed features in scene. With the OBS strategy, we are able to explore more grasp candidates on small- scale objects, benefiting scale grasp detection.
30000
35000
Mag Kocils la Socte
COCLCC'
23905
190006
Close Scate
Diject Mukils
Small (8-4am))
Median 14- Toma)
Large (1:88am)
Nevg
Figure 4: Grasp scale distribution PeaspNet-BBillion.. IBillion.
Figure 5: Pipeline of NeM data augmentation.
GraspNet-
3.3 Noisy-clean Mix
The noise of point-clouds affects the accuracy of grasping detection, especially for small-scale cases. On the one hand, the noise destroys the geometry structure the grasp, which makes it hard to detect. The attempts on point-cloud completion denoising partially solve this problem during in- ference but numerous unseen objects and complicated noise patterns in the real-world prevent their further use. On the other hand, the noise degrades the training of grasp detection where the network struggles in learning the mapping from ambiguous shapes to grasp labels. Compared to larger ge- ometries, the distortions due to noise on small ones are more serious, decreasing the contribution in training.
To overcome this downside, propose Noisy-clean Mix (NcM) data augmentation method. We introduce the clean version of the input point-clouds to increase the number of valid grasp samples. However, Sim-to-Real raises another challenge when only using synthetic clean point-clouds during training. Therefore, we mix noisy and clean data into single scenes, which reduces the domain gap between training and inference. Fig. 5 shows the process of NcM. With 3D models and poses of objects, we synthesize corresponding clean scenes by composing them to full point-clouds. We use the original point-clouds captured by the depth sensor as noisy scenes. For the missing parts in noisy scenes caused by camera poses and occlusions, we also remove the points in the same region of the clean point-clouds keep the consistency. The mixed input is constructed by randomly replacing the objects in noisy scenes with the clean counterparts, leading to more valid samples to facilitate the learning of grasps.
Experiments
4.1 Benchmark, Baseline and Metric
We conduct all the simulation experiments on GraspNet- CraspNee-IBillion [8], which includes 190 cluttered scenes captured in the real-world by Realsense/Kinect cameras in 256 views. For each point on the object surface, grasps annotated densely in its SO(3) space by force closure estimation. It has nearly billion annotations, offering rich grasps to explore the scale distribution in scene.
The baseline method is enhanced version of GraspNet-baseline [8], where transformer-based point encoder and new graspable loss with higher threshold are applied for better performance. For more details about the baseline, please refer to the supplementary material.
5

---

To evaluate the grasps detected in cluttered scenes, the precision of the top-k ranked grasps is used. As in [8]. AP, is computed to represent the average Precision @k for k ranging from to 50 with friction jr, and AP is obtained by the average of AP,,, where ls varies from 0.2 to 1.2.
The original metric does not directly reflect the quality of grasps at different scales because the top ranked grasps mostly distribute on objects (or parts) with simple and large geometries. We thus give new metric to better evaluate the scale-aware grasping quality in scene. The max width of the gripper ( (10em in benchmark) is divided to three intervals, where widths in 0cm-4cm, 4cm-7cm, and 7cm 10em are small-scale, medium-scan scale and large-scale. With such width masks, we evaluate APs, APAr M and A for grasps in small-, medium- and large-scale respectively.
4.2 Ablation Study
First, we validate the effectiveness of the proposed MsCG module, SBL loss, OBS strategy, and NeM augmentation method, and the results are listed in Table 1. All the studies are conducted on the real-world scenes captured by Intel RealSense D435, We evaluate s AP, and APL, and cal- culate the mean to reflect the grasp quality at all the scales. According to Table 1, MsCG achieves significant improvements in all the metrics, which shows the importance of enhanced shape repre- sentations. For SBL, the performance of small-scale grasps is boosted by increasing the weights for such samples in training, comparable that on medium- and large-scale grasps. NeM augmenta- tion brings additional gains for almost all the settings, which demonstrates its necessity in reducing ambiguities during training. For fair validation of the advantage of OBS, we keep the same number of samples (1,024 in our experiment) and take Foreground Sampling (FS) as comparison, where we do FPS in foreground point-clouds. Compared FS, OBS delivers an improvement of 2.37% for small-scale grasps in the seen set with stable results of medium- and large-scale scale grasps. We also note that OBS does not show an improvement on small-scale grasps in the novel set (but still comparable), and it is mainly caused by the inferior performance of the segmentation network for unseen objects.
Seen
Sümilar
Novel
Model
APs
AP
AP
Mean
APs
AP
APL
Mean
AP3
AP
APt
Mean
Buseline
9.44
45.99
54.13
36.52
5.15
35.54
47.82
29.50
4,91
15.26
19.83
13.33
Ours
13:47
48.12
61.81
41.13
6.23
37.90
53.89
32.67
7.60
17.04
23.10
15.91
wfo MsCG
9.14
42.65
52.75
34.85
3,53
32.82
49.07
28.47
4.00
14.56
21.08
13.21
SBL
10.50
47.20
63.26
40.32
4.98
37.02
54.52
32.17
5.97
18.03
23.12
15.71
NcM
61.78
31.81
7.06
16.38
23.27
15.57
FS
15.92
52.73
64.56
44.40
36.15
9.42
18.64
24.23
17.43
Ours OBS
18.29
$2.60
64.34
45.08
10.03
42.77
57.09
36.63
9.29
18:74
24.36
17.46
Table I: Ablation study of the proposed modules on scenes captured by Intel RealSense D435.
4.3
Analysis on Different Scales
We discuss how the proposed approach influences the number and success rate of grasps different scales. For each object in scene, we select the top-10 ranked grasps for statistics and set the friction J 0.8 for grasp successful checking. Results on seen objects are shown in Fig.
For small-scale samples, MsCG, SBL, and OBS all increase the number and success rate of detected grasps. Note that NcM augmentation reduces the number of small-scale grasps but improves the success rate. By introducing more valid small-scale samples in learning, our network avoids to generate grasps in the region where the shape is seriously corrupted by noise. For medium- and large-scale samples, MsCG still shows significant improvement in terms of the success rate.
Ous=Gill
w/o (b) Medium-scale grasping
wid NcM
MsCG
(a) Small-scale grasping
(c) Large-scale grasping
Figure 6: The number and success rate of grasps at different scales on seen objects.

---

4.4 Visualization
We visualize some grasps generated by our approach on the objects that the baseline does not per- form well. The results are shown in Fig. 7. Gripper poses in red are good grasps while those in purple and blue are collision and bad ones respectively. Objects in (1-3) are small-scale and our approach detects more successful grasps on them. The scissors in (4) and the bottle in (5) consist of several geometric parts in different sizes. Benefiting from the ability to detect grasps at various scales, our approach delivers diverse grasps for individual parts, helpful to the downstream manipu- lation task.
Objects
Baseline
AP
13:48 15.84
11.10 12.96
20.56 24.45
7.24 11.10
25.23 26.28
(3)
Pisteecceffeeer ueall
Porce-cloetre group
Failed group
Collision group
Figure 7: Visualization of detected grasps on hard objects.
4.5
Comparison with the State-of-the-art
We make fair comparison with the state-of thet--rrtt counterparts the data acquired by the Re- alSense camera, including [7, 6, 24, 9, 8, 19, 25, 17) and the results are displayed in Table 2.
Seen
Similar
Novel
Model
AP
A.'
APo.4
AP
APo.s
APo.4
AP
APa.s
APo.4
GG-CNN [6]
15.48
21.84
10.25
13.26
18.37
4.62
5.52
5.93
1.86
chu et al. [24]
15.97
23.66
10.80
15.41
20.21
7.06
7.64
8.69
2.52
GPD [7]
22.87
28.53
12.84
21.33
27.83
9.64
8.24
8.89
2.67
PointnetGPD [9]
25.96
33.01
15.37
22.68
29.15
10.76
9.23
9.89
2.74
GraslNet-basellne [8]
27.56
33.43
16.95
26.11
34.18
14.23
10.55
11.25
3.98
gou et al. [19]
27.98
33.47
17.75
27.23
36.34
15.60
12.25
12.45
5.62
li et al. [25]
36.55
47.22
19.24
28.36
36.11
10.85
14.01
16.56
4.82
GSNet [17]
65.70
76.25
61.08
53.75
65.04
45.97
23.98
29.93
14.05
GSNet [17] CD
67.12
78.46
60.90
54.81
66.72
46.17
24.31
30.52
14.23
Ours
58.95
68.18
54.88
52.97
63.24
46.99
22.63
28.53
12.00
Ours + CD
63.83
74.25
58.66
58.46
70.05
51.32
24.63
31.05
12.85
Table 2: Comparison on Graspnet- 1 Billion (RealSense). CD denotes Collision Detection.
Although we focus scale balanced grasp detection in particular tackling small-scale cases, the pro- posed approach achieves an AP of 63.83% on seen objects, 58.46% on similar objects and 24.63% novel objects, which outperforms most counterparts. GSNet [17] shows better performance seen objects mainly because they design an efficient cascaded graspness model to detect grasps in the position which has more successful grasps in its SO(3) space. However, GSNet tends incur negative impact on the diversity of grasps to some extent since grasps in low graspness positions are not taken into consideration.
4.6 Real-world Evaluation
To validate the generalization ability of the proposed approach, we conduct additional real-world experiments. As shown in Figure 8 (a), the grasping system is built on a 7-DoF Agile Diana-7 robot
7

---

arm 30 objects are randomly chosen from the YCB dataset [26], including 10 objects only with grasps of small scales (GSS) (Fig. 8 (b)) and 20 objects with grasps of diverse scales (GDS) (Fig. (c)).
(a) Physical system setting
(b) Objects only with GSS
(c) Ohjects with GDS
Figure Robot and object settings in the real-world grasping experiments.
We compare our model the baseline in settings: isolated object grasping and cluttered object grasping. For isolated object grasping, only the GSS objects are considered and each object is placed three different poses. Success Rate (SR) used as the metrie. As shown Table 3, our model delivers a large improvement, which demonstrates its effectiveness in dealing with grasps of small- scales. For cluttered object grasping, 5--6 objects from both GSS and GDS compose a scene and we make the robot remove them all with maximum number of operations at 8. SR and Scene Completion Rate (SCR) are employed as the metrics. According Table 3, our model performs better in both SR and SCR, also indicating its superiority (as the baseline often fails on the GSS objects).
Isolated
Cluttered
Model
SR (%)
SR (%)
SCR (%)
Baseline
56.67
68.42
87.10
Ours
80.00
88.24
96.77
Table 3:
Results of the real-world grasping experiments.
(a) Complicated object
(b) Flat object
Figure Failure cases.
5 Limitations
Despite the advantages, our approach still has two limitations. First, as in Fig. 9 (a), our model fails on very complicated object, which is composed of some small geometrical components. Second, for small-scale flat objects laying in the plane, few grasps are detected, as in Fig. (b) shows. We speculate that the noise is the major reason in the two cases, which requires more discussion high-pree pihh-prcciion sensor selection and efficient data refinement.
Conclusion
In this paper, we work towards scale balanced 6-DoF grasp detection. An MsCG module is proposed to capture comprehensive local description for grasps different scales. To mitigate the imbalanced scale distribution of grasp labels, we design an SBL loss to adjust the weights of grasp samples in training and OBS strategy to sample more points small-scale objects in inference. We also present NeM data augmentation to reduce the influence of sensor noise. Extensive experiments show the advantages of our approach in grasp detection, especially for small-scale samples which not well handled in the previous work.

---

Acknowledgments
This work is partly supported by the National Natural Science Foundation of China (No. 6202201 1), the Research Program of State Key Laboratory of Software Development Environment (SKLSDE- 2021ZX-04) and the Fundamental Research Funds for the Central Universities.
References
[1] I.-M. Chen and J. W. Burdick. Finding antipodal point grasps on irregularly shaped objects. IEEE Transactions on Robotics and Automation, 9(4):507-512, 1993.
[2] N. Pollard. Closure and quality equivalence for efficient synthesis of grasps from examples. The International Journal of Robotics Research, 23(6):595-611 613, 2004.
[3] M. A. Roa and R. Suárez. Computation of independent contact regions for grasping 3-d objects. JEEE Transactions on Robotics, 25(4):839 850, 2009.
[4] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. In Robotics: Science and Systems, 2017.
[5] A. Depierre, E. Dellandréa, and L. Chen. Jacquard: A large scale dataset for robotic grasp detection. In JEEE/RSJ International Conference on Intelligent Robots and Systems, 2018.
[6] D. Morrison, P. Corke, and J. Leitner. Closing the loop for robotic grasping: real-time, generative grasp synthesis approach. In Robotics: Science and Systems, 2018.
[7] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research, (6(13-14)11555-143, 2017.
[8] H. Fang, Wang, Gou, and C. Lu. Graspnet-Ibillion A large-scale benchmark for general object grasping. In IEEE/CVF Conference on Computer Vision and Pallern Recognition, 2020.
[9] H. Liang, X. Ma, S. Li, M. Görner, S. Tang, B. Fang, F. Sun, and J. Zhang. Pointnetgpd: Detecting grasp configurations from point sets. In International Conference on Robotics and Automation, 2019.
[10] B. Zhao, H. Zhang. X. Lan, H. Wang, Z. Tian, and N. Zheng. Regnet: Region-based grasp network for end-to-end grasp detection in point clouds. In IEEE International Conference on Robotics and Automation, 2021.
[11] W. Wei, Luo, F. Li, G. Xu, Zhong, W. Li, and Wang. Gpr: Grasp pose refinement network for cluttered scenes. In IEEE International Conference on Robotics and Automation, 2021.
[12] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert. Pen: Point completion network. In International Conference on 3D Vision, 2018.
[13] M.-J. Rakotosaona, V. La Barbera, P. Guerrero, N. J. Mitra, and M. Ovsjanikov. Pointcleannet: Learning to denoise and remove outliers from dense point clouds. Computer Graphics Forum, 39(1):18-2203, 2020.
[14] J. Choe, B. Joung. F. Rameau, J. Park, and I. S. Kweon. Deep point cloud reconstruction. In International Conference on Learning Representations, 2022.
[15] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In IEEE/CVF International Conference on Computer Vision, 2019.
[16] Y. Qin, R. Chen, H. Zhu, M. Song, J. Xu, and H. Su. S4g: Amodal single-view single-shot se(3) grasp detection in cluttered scenes. In Conference on robot learning. 2020.
[17] C. Wang, H. Fang, M. Gou, H. Fang. J. Gao, and C. Lu. Graspness discovery in clutters for fast and accurate grasp detection. In IEEF/CVE International Conference on Computer Vision, 2021.

---

[18] M. Breyer, J. J. Chung, L. Ott, S. Roland. and N. Juan. Volumetric grasping network: Real-time 6 dof grasp detection clutter. In Conference on Robot Learning. 2020.
[19] M. Gou, H. Fang, Z. Zhu, S. Xu, C. Wang, and C. Lu. Rgb matters: Learning 7-dof grasp poses on monocular rgbd images. In IEEE International Conference on Robotics and Automation, 2021.
[20] Z. Jiang Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. In Roboties: Science and Systems (RSS), 2021.
[21] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun. Point transformer. In IEEE/CVF Interna- tional Conference on Computer Vision, 2021.
[22] C. Elkan. The foundations of cost-sensitive learning. International Joint Conference on Artificial Intelligence, 2001.
[23] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments. IEEE Transactions on Robotics, 37(5):1343-1333 2021.
[24] F.J. Chu, R. Xu, and P. A. Vela. Real-world multiobject, multigrasp detection. JEEE Robotics and Automation Letters, 3(4):3355-3362 2018.
[25] Y. Li, T. Kong. Chu, Li, Wang, and L. Li. Simultaneous semantic and collision learning for 6-dof grasp pose estimation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2021.
[26] B. Calli, Singh, Bruce, Walsman, Konolige, Srinivasa, Abbeel, and A. M. Dol- lar. Yale-cmu- Yale-cmu-berkeley dataset for robotic manipulation research. The International Journal of Robotics Research, 36(3):261-268, 2017.
10


Reviews:
[{"review_id": "nCA7-UbEO05", "reviewer": "Reviewer_iqJc", "summary": "This paper considers the problem of 6DoF robotic grasp detection from point clouds. Small-scale grasp detection is identified as a particular challenge, not having been a focus in prior work. The authors propose a collection of strategies (MsCG, SBL, OBS) geared toward making improvements for the small-scale grasp detection case that is shown to be beneficial on experiments conducted with the GraspNet 1 Billion benchmark.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe authors address an important problem in the field of robotics and provide a good motivation for the problem they solve; Figure 1 is a good visual example that captures the problem well. The ideas are well conveyed and overall the paper is easy to follow.\n\n\n\nWeaknesses\n\nThe main weakness for me is that no real robot experiments are performed. While results on the GraspNet 1 Billion benchmark showcase the performance gains of the introduced modules, they are not a substitute for real robot experiments. This is missing and would otherwise make the paper a strong submission.\n\nSecondly, I am not entirely convinced by the contribution of the noisy-clean mix augmentations. The results in Table 1 and Figure 5 reveal that its inclusion does not have a significant impact. We also do not see the performance when the network is trained on purely synthetic scenes, which I think is essential to justify the mixed training data. It would be really interesting to see an ablation on the performance from 100% synthetic (clean) to 100% raw (noisy) to find out what the optimal ratio is.\n\nA minor weakness is that there is no ablation on the number of cylinder radii for the MsCG. Would it help to have more? What are the trade-offs?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper contains interesting ideas that address a highly relevant in robotics, hence I rate a weak accept given the merit in and contribution of the methodology. Unfortunately, there are key experiments missing (real robot, ablations) that restrict me from giving a strong accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Some aspects of the paper are unclear. These are discusses below.\n\n- When is the segmentation network trained? Prior to the main network (point encoder, approach heads and operation head)? If so, the authors could make this more clear to avoid any confusion as I don't see it made explicit in the manuscript. Also, is there a good reason why the same point encoder cannot be used for both the segmentation network as well as the grasp candidate proposal module?\n\n- The loss in equation (2) is not clear without the definition of the terms and sub-losses. While it is a modification of the loss in [8], I suggest to make the definition in equation (2) more clear to make the manuscript self-contained.\n\n- The sentence on line 147-158 that explains how features of new samples are computed is a bit vague. I don't follow how the new feature is the linear interpolation of the three nearest neighbours. Is this the mean of the nearest three? Is there a weight applied to the neighbours based on their distance to the new point?\n\n- The claim on lines 49-51 that point completion and denoising methods are unsuitable is not supported by experiments either conducted in this manuscript or in related work. It would be good to provide quantitative evidence to back up the claim and better justify the design choices of the proposed method.\n\n- In Figure 5, which set of objects are the experiments performed on? Seen, similar, novel or all?\n\n- In Table 2, what does \"+ CD\" mean?\n\n- Typos\n  -- Line 131: \"not practical\" -> \"impractical\".\n  -- Line 154: \"complicate\" -> \"complicated\"."}}, {"review_id": "1d0lpUnfHRD", "reviewer": "Reviewer_dLNy", "summary": "This paper suggests that most grasps datasets are biased towards large objects than small objects, and as a result grasp planners compute more grasps on large objects/object-parts and struggle to compute grasps on small objects. To address this, the paper proposes the MsCG method to process local features of a point cloud in varying scales, the SBL loss to fix the imbalance using weights during training, the OBS sampling strategy to fix the imbalance during test time, and a data augmentation procedure that mixes raw data with synthetic data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The problem of data imbalance for 6-dof grasping is an interesting problem, and is well demonstrated in Fig 1 (b).  \n\n2. The paper presents a large amount of experiments in simulation with multiple ablations and the results are quite interesting.\n\n3. The MsCG method is interesting and seems to work well empirically.\n\nWeaknesses:\n1. The second contribution the paper makes includes weights that scale the training loss according to the proportion of the sample’s class in the dataset, however this seems like a trivial way to address data imbalance and I’d appreciate additional information that supports the novelty of these weights, for example contrasting it with the commonly used method.\n\n2. The third contribution suggests a data augmentation procedure that mixes raw and synthetic data to increase the number of the number of training examples and address the sim2real gap at once. The paper would benefit from additional experiments that compare the performance of NcM with other common data augmentation techniques, for example training on synthetic data and fine-tuning on raw data. \n\n3. The results in tables 1 and 2 seem impressive, however it is not clear what their scale represents, and it might be clearer if the results are normalized in a way that makes the scale more intuitive (% for example).\n\n4. The method includes a step in which “M candidates are sampled…”; What are these candidates? Are these points from the point cloud or grasps, and if grasps how are they represented? Where does the farthest point sampling happen in Fig 2? The input to the point encoder seems to be the entire point cloud, but the output is sampled with dimensions MxC.\n\n5. Was this method evaluated on a physical system? It is not clear that it was, and if not it is strongly recommended to test it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper addresses an interesting problem and I’m recommending a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The weaknesses section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "-MbWS24Ytv", "reviewer": "Reviewer_XwLV", "summary": "This paper focuses on solving scale imbalance problems in 6-DOF grasp detection tasks. A framework that consists of several modules is proposed. The main contributions include:\n\n1. A Multi-scale Cylinder Grouping (MsCG) module that allows the network to extract features from different scales.\n2. A Scale Balanced Learning loss (SBL) to weigh the imbalanced distribution of object sizes in the dataset. \n3. A Object balance Sampling (OBS) strategy to deal with the imbalance during inference.\n\nThe proposed method is evaluated on the GraspNet-1Billion dataset, and the ablation study shows the effectiveness of each module in the network. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The proposed method can detect grasps for objects on different scales and is shown to perform better on unseen objects compared to GSNet and other baselines.\n* The multi-scale cylinder grouping is an interesting strategy that helps the network to extract features on different scales. It is similar to the image pyramid approach used in the Visual SLAM community. \n* The experimental results are thorough and convincing. The ablation study shows the effectiveness of each module. However, some clarification is needed.\n\n**Weaknesses** (Also see Issues)\n* The Noisy-clean Mix (NcM) seems to be very similar to the common noise augmentation approach in machine learning. I would appreciate it if the authors could emphasize the difference between the two.\n* The paper is generally well-written and easy to follow. However, some of the writings are ambiguous and need to be clarified. (See issues.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a grasp detection framework that addresses the scale-imbalance issue. The results show the proposed method achieves better average precision on unseen objects and the ablation studies of each module are consistent with the claims. There exists some issues that need to be addressed. However, the paper is overall convincing, and thus I recommend a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* What is the method that denotes \"baseline\" in the evaluation section? It is unclear what method is being compared. It'll be great if it can be explicitly cited.\n* In Figure 5, the proposed method without NcM performs better or equally to the full method. Any idea why this is inconsistent with the results in the ablation study? \n* What is the \"CD\" in table 2? I don't see it being mentioned in the paper. \n\nWritings: \n* In Figures 1 & 6, how are the good grasp and bad grasp defined?\n* In Figure 1 (b), AP is not introduced before. It was confusing for me until I saw the experimental section. I recommend the authors use Average Precision directly instead of an abbreviation. \n* In Figure 1 (b), I understand the authors are trying to show with the percentage of small grasps increases, the AP drops accordingly. However, it is also not very clear at the first glimpse. I suggest the authors change the \"percentage\" to maybe \"the percentage of small grasps.\" in the future.\n\nNit-Picking:\n* I recommend the authors also add color legends in Figure 6. It'll help readers to understand the figure more smoothly. "}}]

Author rebuttal:

Decision: Accept (poster)