OrbitGrasp: SE(3)-Equivariant Grasp Learning
Authors: Boce Hu
Xupeng Zhu
Dian Wang
Zihao Dong
Haojie Huang
Chenghao Wang
Robin Walters
Robert Platt
Venue: CoRL 2024 (2024)
Keywords: Grasp Detection
Equivariance
Symmetry
Grasp Learning
TL;DR: We propose an SE(3)-equivariant grasp learning algorithm based on spherical harmonics.

Abstract: While grasp detection is an important part of any robotic manipulation pipeline, reliable and accurate grasp detection in $\\mathrm{SE}(3)$ remains a research challenge. Many robotics applications in unstructured environments such as the home or warehouse would benefit a lot from better grasp performance. This paper proposes a novel framework for detecting $\mathrm{SE}(3)$ grasp poses based on point cloud input. Our main contribution is to propose an $\mathrm{SE}(3)$-equivariant model that maps each point in the cloud to a continuous grasp quality function over the 2-sphere $S^2$ using a spherical harmonic basis. Compared with reasoning about a finite set of samples, this formulation improves the accuracy and efficiency of our model when a large number of samples would otherwise be needed. In order to accomplish this, we propose a novel variation on EquiFormerV2 that leverages a UNet-style backbone to enlarge the number of points the model can handle. Our resulting method, which we name OrbitGrasp, significantly outperforms baselines in both simulation and physical experiments.

OrbitGrasp: SE (E)33-EEuiva (3)-Eqiivriian Grasp Learning
Boce Hu', Xupeng Zhu"!, Dian Wang"1, Zihao Dong"!, Haojie Huang"!,
Chenghao Wang",, Robin Walters Robert Platt Nortteesternn University Boston Dynamics Al Institute https://orbitgrrasp citpasspp.gttuuutte github.io/
Abstract:
While grasp detection is important part of any robotic manipula- tion pipeline, reliable and accurate grasp detection in SE(3) remains research challenge. Many robotics applications in unstructured environments such the home or warchouse would benefit a lot from better grasp performance. This paper proposes a novel framework for detecting SE(3) grasp poses based on point cloud input. Our main contribution is to propose an each point in the cloud to continuous grasp quality function over the 2-sphere S using spherical harmonic basis. Compared with reasoning about finite of samples, this formulation improves the accuracy and efficiency of our model when a large number of samples would otherwise be needed. In order to accomplish this, we propose a novel variation EquiFormerV2 that leverages a UNet-style encoder-decoder architecture to enlarge the number of points the model can han- dle. Our resulting method, which we name OrbitGrasp, significantly outperforms baselines in both simulation and physical experiments.
model that maps
Keywords: Grasp Detection, Equivariance, Symmetry, Grasp Learning
Introduction
The ability to detect and evaluate good grasp poses in manipulation scene is critical part of robotic manipulation. Despite extensive recent work in the area, c.g. [1. 2, 3, 4, 5, 6, 7, 8, 9], grasp detection still not accurate and reliable enough for many practical ap- plications. A key challenge here is ef- fective reasoning over hand poses in SO(3), i.e. the three dimensions of orientation spanned by SE(3). Simply representing orientations in (3(3) can be a challenge due to classic problems like gimble lock [10] and discontinu- ity [11]. More importantly, it would be very helpful to be able to infer grasp quality over contin- uous range of orientations in SO(3) something which is major challenge for conventional grasp methods. Finally, since SE(3) grasping problems often equivariant (i.e. symmetric) in SO(3), is often desirable to encode symmetry assumptions into the model.
up
np
We infer an orbit grasps (yellow ellipse) defined relative to the surface normal (red arrow) at the contact point (pink doi). Since our model is equivariant over SO(3). the optimal pose (represented by the solid gripper) the orbit rotates consistently with the scene (left and right show rotation by 90 degrees).
Figure 1.
This paper addresses these challenges by leveraging recent innovations in equivariant point mod- els [12, 13, 14, 15]. Specifically, we use a model that maps each point in a point cloud to a grasp
Equal contribution.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

quality function over the 2-sphere S². For each point in the cloud, this function encodes grasp qual- ity for the space of possible hand approach directions toward that point. An important question here is how to represent this quality function over 5². Following recent work [1, 16, 8], our approach is to leverage the spherical harmonic basis. Specifically, at each point in the cloud, the neural network outputs vector of Fourier coefficients to the spherical harmonic basis functions, thereby defining a grasp quality function over hand approach directions. With this leamed per-point grasp distribution, we can evaluate the quality of large number of potential grasp poses quickly and easily, thereby more easily locating a grasp appropriate for a given downstream manipulation task.
To summarize, this paper introduces several key contributions. First, we propose a novel method of using spherical harmonics to reason about an orbit (i.e. a S' manifold embedded in 5²) of grasp approach directions defined relative the surface normal at each contact point in the point cloud, shown in Figure 1. Second, we enhance the recently developed (E(()--quivarrin model, EquiFormerV2 [17], by incorporating U-Net backbone and thereby enabling the model to ac- commodate a larger number of points in the point cloud. Finally, we evaluate our method against multiple established baselines [5, 18, 8, 9]. using the benchmark tasks proposed in [5]. The results indicate that our approach, which we name OrbitGrasp, convincingly outperforms the baselines in both single-view and multi-view settings, in both simulation and physical experiments.
2 Related Work
SE(3) Grasp Detection: Current 6-DoF grasping tasks in cluttered tabletop scenarios primarily use a volumetric-basee or point cloud-based scene representation as input and output one or more optimal grasp poses [7, 3, 19, 2]. Volumetric Grasping Network (VGN) [5] and Grasp detection via Implicit Geometry and Affordance ( (IGA) [18] use 3D convolutional models to reason about 3D truncated signed distance function (TSDF). However, these methods suffer from high memory consumption and resolution limitations. In contrast, point clouds can provide higher resolution but are more difficult to reason about due to the lack of structure. One approach to adding structure is to leverage equivariant point cloud models [20, 21J. For example, EdgeGrasp [8] introduced a method based on a vector neuron model. CAPGrasp [22] is another equivariant model that samples grasp candidates while constraining the gripper approach direction to be within a certain angular distance from the surface normal at the associated point. Other recent grasping methods leverage surface reconstruction of the object to be grasped, including NeuGraspNer [23] and ICGNet [9].
Equivariance in Robot Learning: Using symmetry, i.c. equivariance, in robot learning tasks has recently been used to improve sample efficiency and generalization. [24, 25, 26] introduce equiv- ariant neural network models in the form of steerable convolutional layers [27, 28] to SE(2) manip- ulation tasks. [29] applies similar models to transporter network [30], making it bi-equivariant and significantly improving sample efficiency. EDF [31] introduces an equivariant energy model that produces continuous distribution over pose. Fourtran [32] extends [29] from SE(2) to SE(3) using Fourier representation of rotations. [33] develops novel, dense, interpretable representation for relative object placement tasks. RiEMann [34] presents the first near real-time -eqiivvrava robot manipulation framework for point cloud inputs that addresses the problem of slow inference over the SE(3) SE(3)-manifold manifold. Most relevant to this paper are [1] who obtain SE(2) equivariance using steerable convolutional layers and [8] who use vector neurons to obtain (E equivariance.
OrbitGrasp: The method proposed in this paper, OrbitGrasp, is distinct from the work described above in a couple of important ways. First, prior work in grasp detection is generally sample-based [35, 7, 6]. i.e., they evaluate grasp quality for a (large) finite set of samples or discrete voxel/pixel lo- cations. In contrast, OrbitGrasp evaluates grasp quality for a continuous range of approach directions by inferring the parameters of spherical harmonic basis functions over S This is simultancously more computationally efficient and more precise than sample- methods. Another distinction between OrbitGrasp and prior work is that the representation of rotation in terms of high degrees of spherical harmonics encodes continuous equivariant constraints more accurately and, therefore, models the grasp function better.

---

(Dh x)
Reconstmation
Grau Qualit Org
Qualtty Osa Onia
(n's.s)
Sytenical Homootes
Dar Black
Hew Bank
(EEg) p S
EqualomerV2 Block
Driit Pose Sempler
Sampler
Quality Recorstructon
query orbits on S
Quality Fanction
|p S.
Figure 2. OrbitGrasp takes point cloud B, (a neighborhood around center point ci) as input and outputs grasp quality function fp: S + for each point p in B,. The model produces Fourier coefficients for each p (represented as different channels in the network output), which are used to reconstruct fp based on spherical harmonics, in Equation 1. The Orbit Pose Sampler generates multiple poses for each perpendicular to the surface normal np and queries corresponding fp(-) evaluate these grasp qualities along the orbit. The grasp with the highest quality then selected, thereby producing the optimal grasp pose a as shown on right.
3 Background
We introduce concepts from group theory and -3(33--quivarrian neural networks, which are key to our method. For additional background, see, c.g., [36].
Representations of SE(3): Our work focuses on the special Euclidean group SE(3) = SO(3) x R3 of 3D rotations and translations. representation of group G is a homomorphism p: > GL(V) mapping each group element g an invertible linear operator the vector space V. The representation (p, V) is an irreducible representation, irrep, if no non-trivial subspace W C V is closed under the action of G. The irreps of 5O(3) are classified by positive integers / E Z20 with the irrep Vi of type / having dimension 2/ + 1. For example, / = gives the standard rotation action on R33 The group action on Vi is given by the Wigner D-matrices D': SO(3) - GL(Vi) of size (2l+ x 1)). Traditionally, the Wigner D-matrices are parameterized by Euler angles (a, 3, 7) and indexed symmetrically about 0 as D m (a, B, 7) where -1 m, m' S
Spherical Harmonics: The real spherical harmonics (SH) are functions on the sphere Y,"; S + R indexed by / E Z20 and -1 < m < 1. As set, they define an orthonormal basis of the Hilbert space of square-integrable functions on the sphere (5 = (/: 52 + IR. Ssa 1/12 < oo). That is, given f: 52 + I. we can write f(6, Φ) Ei= FT" (0. where Fir are the Fourier coefficients of f. For a finite truncation
L. the total number of Fourier coefficients is (L+1)² The mapping FT: + {F(') is known the spherical Fourier transform. Approximating f by storing the F?nh up to degree < L gives an efficient way to encode the spherical signal f. The spherical harmonics are compatible with the SO(3) action on the sphere, making them suitable for use in
networks. The rotation of a spherical signal f can be computed in Fourier space using Wigner D-matrices. Let g E SO(3). Define f' =
f to be the rotated spherical signal defined (g ()( = f(g-1 u) for u E S². Denote the Fourier coefficients of f' by FT(f') ( Then in terms of the Fourier coefficients of f we have F D') where Fi denotes the vector Fi )
EquiFormerV22
EquiFormerV2 [17] is a Eqquiv-aauv Graph Neural Network (GNN) that addresses the high computational costs and poor generalization of higher-degree irreps in EquiFormer [37]. Unfortunately, since EquiFormer and EquiFormerV2 were developed for applica- tions in computational biology, they have a hard time scaling point cloud data, which generally contain more points than molecules do atoms. To address this limitation, paper proposes variation of EquiFormerV2 that uses U-Net architecture increase the capacity of the model. Compared to [38], our new model can reason about higher-degree spherical harmonic functions.
4 Method
Problem Statement: Given point cloud P C Rd captured by multiple depth cameras, objective is to identify set of good grasp poses. Specifically, we want to estimate a function
3

---

T: (P.a) + [0, 1] which maps the point cloud P and the hand pose E SE(3) to the probability of successful grasp. Notice that we expect I to be invariant to translations and rotations E SE(3), i.e. we expect (g g successful does not change when both the scene and the grasp pose transform in concert.
g- a) = [(P,a) a). This reflects an assumption that the probability that a grasp
Summary of Approach: Figure 2 illustrates
model and approach. First, we downsample and crop the point cloud into a small number of point neighborhoods BI.... By_ Then, we evaluate our model for each B, For each point p E B, the model predicts the grasp quality over the space of possible hand approach directions, represented as function fp: 52 + R. We densely sample this function over an orbit of approach directions orthogonal to the object surface normal at p and obtain single grasp that maximizes quality over the orbit. The result is single-hand orientation in SO(3) each point in the cloud, corresponding produces dense sampling of grasps that be filtered further for grasping other downstream tasks. Figure/ 7 in the Appendix provides a detailed, zoomed-in visualization. This approach has several important advantages. First, since our model outputs a continuous distribution over 5 it generalizes better over orientation than would a sample-based approach. Second, since our model is eqquivaraa, it incorporates problem symmetry as an inductive bias into the model. Finally, the approach only considers grasps that make contact parallel to the object surface normal, thereby incorporating an additional geometric prior.
good grasp that point. This
Sampling k Center Points:
We prepare the point cloud for processing by the prediction model as follows. First, we downsample the input point cloud P to tractable number of points P (around 4k to 6k). Rather than passing the entire point cloud through our model (which would be expensive), we define smaller point clouds BI...., By C P which we evaluate over separately (k is usually around 10). The B, are defined as neighborhoods of & center points Ci...., Cj from P selected as follows. If a segmentation of the point cloud into objects is available, then these k center points could be the 3D positions of the centers of individual object masks. Otherwise, these centers could be random samples obtained using farthest point sampling (FPS). In our experiments, we generate center points using object masks during training to enhance the model's 's object-centric awareness, and FPS during inference to demonstrate generalization and performance without segmentation. A detailed descrip- tion of our mask-based sample generation is provided in Appendix A. We then construct each B, as a neighborhood around e as follows. First denote the set of points contained within radius rı ball centered c by B, B(c, r1) {p P ||p - call S ri} for a parameterized radius ri. Since the grasp quality at a point p depends on the geometry of sufficiently large neighborhood of p to avoid boundary effect, we then set B, = . N(c,, m) to be the m nearest neighbors of c;, where m is chosen large enough such that B, C B; and B; is larger by fair margin. The larger point cloud B; is passed as input to the model, but the output grasp quality is only inferred over the smaller point cloud B, This assures inference is only performed on points with sufficient geometric context.
Representing Grasp Quality Over S2: We model the grasp quality function r: (P,a) ~ [0, 1] with neural network D B; + {pp: : S R p E B.} that takes each of the neighborhoods B1, .,,..., Bg in P as input. For each B;, the model outputs spherical function fp: 52 - IR for each E B,, The function fp represents the grasp quality over all approach directions in S2 at the point p. The function fp is represented by Fourier coefficients of the spherical harmonics,
f0( 1=0 m=
FT, (0,
where (0, σ) denotes the spherical harmonic of degree and order m evaluated at the point on S defined in spherical coordinates (8, Φ). Concretely, the model outputs the set of Fourier coefficients (FI,)) ) for at each p E Bq.
Implementing I' as an Equivariant Neural Network: We implement using a modified version of EquiFormerV2 [17], a fully SE(3) -quiiariqu,, GNN-based network designed for handling node- based data. In the original EquiFormerV2 [17], all nodes are fully connected. However, in order to handle the larger number of points in our point clouds, we developed a UNet-style version of the model, which has greater capacity (inspired by [38]). Figure 2 illustrates the model architec-

---

ture. Given the point cloud B,, successive blocks in our model sequentially downsample the points using FPS. As the number of points decreases, the radius of the edge in the graph increases, al- lowing distant points to communicate, expanding the receptive field per block, and capturing global features. To recover the features of each point in the original point cloud after downsampling, we upsample point features back by inverting the edges: during downsampling, we record the source and target edges in each block, and during upsampling, features gathered during downsampling are effectively propagated back to the original points. In addition, we employ skip connections between each downsample block and its corresponding up- sample block to aggregate features and prevent degradation. Details of the architecture are in Ap- pendix B. This model satisfies the desired SE(3) equivariance constraint described in Section 4, TyyB.) = (B;)) = {g- fp: S + IR p E g.:). For g E SO(3), if the input point cloud rotated gB, the equivariance constraints ensure the output Fourier coefficients are also transformed D' (g).).. Evaluating a transformed grasp g thus gives the same quality of a successful grasp 0g u) fplg 1 g
swap these edges. This ensures that
g. u) fp(u).
Inferring Grasp Pose: Since our neural network model only infers grasp quality over 5², we must obtain the remaining orientation DoF somehow. Following [8], we accomplish this by constraining one of the two gripper fingers to make contact such that the object surface normal the contact point is parallel to the gripper closing direction (see Figure 3). Specifically, for point p E B, C B; in region B; with object surface normal Pp, we constrain the hand y-axis (gripper closing direction) to be parallel to ny. Therefore, valid hand orienta- tions form a submanifold in SO(3) homeomorphic to a 1-sphere S which we call the orbit at p
np
Figure Green and blue denote the y. z directions of the hand, and mp is the normal vector at (red). Black is the orbit of the
O {R np, r3] SO(3)}
(2)
where 71, Rpp r3 are the columns of the 3-by-3 rotation matrix R. Valid orientations determined by the 2-axis of the gripper (the approach direction.
approach direction of the hand) which may be any unit vector perpendicular to ng. We may thus specify valid grasps by their approach vector r's E Op {ra S² n, r1 0} since -np x r3. The details of how we evaluate the fp can be found in Appendix C.
In the above discussion, notice that we have constrained two DoF' of orientation, which could suggest that our model fy should infer only one DoF of orientation, not two. This raises the question: Why is fp defined over 5 (the spherical harmonics) rather than 5 (the cireular harmonies)? The answer is interesting and important. Defining the output of fp over S would require choosing gauge. i.c. mapping of 5 to the tangent plane at each point. Assuming S were parameterized by an angle 0, we would need to select a vector that defined 0 = O. Notice that it would be hard to make this selection in a consistent way across the point cloud. In contrast, by defining the output of fp over 5², we avoid this problem because we can express this function a single global coordinate system for all points in the cloud. One way to think about this is that we have gained consistency in representation by adding extra dimension, i.e. going from S to $2.
5 Experiments
Simulation Environment: We evaluated model in simulation using PyBullet [39]. Our setting is the same as that used in [18, 5, 8, 9]. The workspace size is 30 em cube and contains varying numbers of objects randomly placed within it. The object dataset includes 303 training and 40 test objects from [40, 41, 42, 43]. A Franka-Emin Panda floating gripper is used to grasp objects in the workspace. We evaluated two camera configurations. The first a single-view setting where a camera is positioned randomly on a spherical region around the workspace. The second a three- camera multi-view setting. Following [5], we evaluate two different grasping tasks: a Pile setting where objects are randomly dropped into the workspace and Packed setting where objects are placed upright in random poses. For more details, Appendix D.
5

---

Setting
Method
Packed
Pile
(%)
(S)
(%)
(S)
GIGA [18]
89.9 1.7
87.6
2.0
76.3 2.4
80.9 4.1
GIGA-HR [18]
91.4 1.5
88.5
86.5
80.8 1.9
Single- view
EdgeGrrasp [8]
92.5 0.9
94.3 1.1
91.5
1.3
92.5 1.3
GradgeGrasp [8]
92.0
ICGNet [9]
97.7 + 0.9
97.5 - 0.3
92.0 + 2.6
94.1 + 1.4
OrbijGrasp (3M)
98.4 0.5
98.8 1 0.3
96.3 1 0.3
97.7 1: 0.7
OrbitGrrssp (6M)
98.3 0.7
98.8 0.6
96.7 2 1.1
97.9 0.5
VGN
45.6 0.7
Multi-
VNFdgeGrasp [8]
97. + 1.3
96.1 + 0.5
95. + 1.0
95.5 + 1.5
view
OrbitGrrasp (3M)
98.6
0.3
99.1
98.6 0.7
98.5 0.5
OrbitGrasp(6MM
99.0 0.6
99.2 0.3
98.5 0.6
98.2 0.6
Table 1. We compared the OrbitGrasp with various baselines in terms of grasp success rate (GSR) and declutter rate (DR), the same metrics as used [8]. For the single-view setting, we tested pretrained models from [8] and obtained results for GIGA, GIGA-HR -HR, and ICGNet directly from [9]. For the multi-view setting, we retrained VNEdgeGrasp and tested the pretrained VGN model for comparison. OrbitGrasp (3M) is trained on the 3M dataset, which is similar in size to that used to train ICGNet and EdgeGrasp. OrbitGrasp (6M) is trained on the full 6M dataset and reaches slightly higher level of performance. The best results marked with bold and the second best results are underlined.
We generate data in simulation by loading random number of objects into the workspace and then generating point clouds in both single-view (one depth camera) and multi-view settings (three depth cameras spaced evenly around the workspace). We add Gaussian noise sampled from (0, 0.001) to the point cloud to make the model more robust to real world sensor noise. After creating the scenes, we must generate training data for the grasp model. First, object masks obtained using the Segment Anything Model (SAM) [44] (see Appendix A). We use SAM to generate object masks for training only, and for inference, SAM is not required. For each masked object, we randomly select a set of candidate grasp points and evaluate 36 candidate grasp orientations per point that satisfy Equation 2, i.e. where the gripper axis is orthogonal to the object surface normal at contact. In total, approximately 6M grasp poses (approximately 2.5M positive and 3.5M negative) are generated for each camera setting: 4M from 2,500 Pile scenes and 2M from 800 Packed scenes. The data is split with 90% for training and 10% for validation.
Generating Training Data:
Training Details: Since our training data is segmented into objects, we use the center of every mask as the center point 6 and obtain the local point neighborhood B, During training, we balance the positive and negative grasp labels in each B, The model trained using the Adam optimizer(45), starting with learning rate of le-4 and using a cosine annealing scheduler [46]. We apply binary cross-entropy loss for each grasp pose and a dropout rate [47] of 0.1 to prevent overfitting. Our network is trained for 15 epochs, taking approximately 25 hours 22k point clouds. Each SGD step takes 0.25 seconds with batch size of 1. The model is trained NVIDIA RTX 4090 GPU.
5.1 Comparison With Baseline Methods in Simulation
Baselines: We compare our method with several strong basclines. In the single random view setting, we compare it with volumetric-based methods: GIGA and GIGA-HR (high resolution) [18]. Since GIGA was trained under a fixed view, we compare it with the results from [9], which retrains it under a random view setting. We also compare our method with EdgeGrasp and VNEdgeGrasp [8], two point cloud-based methods, where VNEdgeGrasp SE.(3)-invariant. Finally, we compare our method with ICGNet [9], an approach focused on partial observation to object-centric grasping that reconstructs the full 3D shape of objects before finding grasps. In the multi-view setting, compare method with VGN [5], which takes TSDF as input and generates single grasp pose per voxel, as well as VNEdgeGrasp [8] trained in the multi-view setting.
We report the comparison result in Table under two different settings, single-view and multi-view, and two different tasks, Packed and Pile. For a fair comparison with the base- lines, we also trained our model on similar amount of data (3M labeled grasp poses uni- formly randomly downsampled from the original 6M dataset.), consistent with EdgeGrasp [8] and ICGNet [9]. Additional performance details under different amounts of data pro-
Results:

---

(b)
(c)
Figure 4. Real world Experiment Setting, (a) Robot platform setup. (b) Upper: Packed object set with 10 objects. Bottom: Packed scene (c) Upper: Pile object set with 25 objects. Bottom: Pile scene
vided the Appendix E. Performance is evaluated based on two metries: (1) Grasp Suc-
cess Rate (GSR) num of successful grasps/ /num of total grasps, and (2) Declutter Rate (DR)
num of grasped abjects/nunm of total objects. For each task, we conducted five iterations of testing, with each iteration containing 100 rounds (5 objects per per round). A round ends when either all objects cleared two consecutive grasp failures occur.
The results in Table indicate that our method outperforms all baselines across both settings and tasks in terms of both GSR and DR, in both the 3M and 6M training sets. The high GSR indicates that our model can predict accurate grasp quality. On the other hand, the high DR signifies that our model infers accurate grasp poses that do not move objects outside of the workspace. Notably, our model performs well even though point centers are selected differently training time (centered on object segments) and test time (selected using FPS), i.e., described in Section 4. This demonstrates that the grasp quality function I' output by our method exhibits strong robustness to different point cloud geometries.
5.2
Physical Experiments
Setting
Method
Packed
Pilc
GSR (%)
DR (%)
GSR (%)
DR (5)
Single-
VNEdgeGrasp [8]
88, 08.9(99/100) 108)
96 (960(99/)00)
88 (8.2(97/110)
07.0(97/100
view
OrbitGrasp
92.5(98/ (106)
98.0(98/100)
93.2(97/ 104)
98.0[98/100
Multi-
VNEdgeGrasp [8]
90.1 /111)
100.0(100/100)
92.1(93/100
97.0(97/100
view
OrbitGrrspp
95.2(100/105)
100.0( 000.0(100/100)
94.3(99/100
99.0(99/ (100)
Table 2. We compared the results of OrbitGrasp with VNEdgeGrasp [8] using the same metries in the simu- lation experiments. Our experiment settings for single-view and multi-view are shown Figure Sometimes, a single trial resulted grasping two objects simultancously.
To assess our method's real-world performance, conduct physical experiments involving two tasks under two camera settings, replicating those in the simulation. We directly transfer the trained model from the simulation to the real-world setting to evaluate the performance gap between them.
Setup: As shown in Figure (a), UR5 robot arm is equipped with a Robotiq Gripper. We mount and calibrate 4 RealSense D455 cameras on the table. In the multi-view setting, low-mounted cameras capture the scene for full observation. In the single view setting, we use the top-mounted camera. After obtaining the point cloud, we crop it to the workspace, filter outliers, and calculate the surface normals at each point. We do not crop the table from our point clouds. Objects are placed on the table as follows. In the Packed setting, we draw 5 objects uniformly at random from the set of 10 objects shown at the top of Figure (b) and place them in upright orientations with a randomly selected planar position and orientation. In the Pile setting, we draw 5 objects uniformly random from the of 25 objects shown at the top of Figure 4 (c) and dump them onto tabletop out of box. We conduct 10 rounds of experiments for the Packed scene and 4 rounds for the Pile scene for each view setting. We configure method as follows. We use k = 10 center points selected
7

---

Model
#Params
Packed
Pile
(%)
(5)
(8)
(%)
PoiniNet++ [48]
14M
78.1
2.9
78.8 : 1.9
60.5 4: 1.8
54.1 :: 2.7
OrhiiGrasp (1
1)
4M
94.2
1.2
96.3
1.3
93.9 1 0.5
97.0
1: 0.7
OrbitGrasp (1 2, 2)
8M
97.4
0.5
96.5
0.6
96.3 = 1.0
97.6
= 0.7
OrbitGrasp (1
96.7
Table 3. Comparison results between non non-equivariant and equivariant networks with varying degrees of SH.
using height-thresholded FPS to form the neighborhoods Bi,..., Bg. From the orbit of each point, we evenly sample 36 grasp poses and evaluate them based on the grasp quality function from the network. After filtering out unreachable and low-quality poses (i.e. grasp poses with a quality < 0.95), select the pose with the highest Z value.
Results: We baseline our method against VNEdgeGrasp [8]. The results are shown in Table 2. the single view Packed task, the GSR of our model outperformed VNEdgeGrasp by an average of 3.6% and by 5% in the single view Pile task. In the multi-view setting, our model's GSR outperformed by 5.1% on the Packed task and by 2.2% on the Pile task. We analyze the failure modes we encountered in Appendix F.
5.3 Ablation Study
In our ablation study, we measure the significance of network equivariance by comparing to Point- Net++ [48], a well-known non-equivariant model under a single random camera setting, keeping all settings identical except for the network structure. For fairness, we set up PointNet++ so that has the same number of parameters as our method. We also measure the influence of different degrees of spherical harmonics, where higher degrees correspond more basis functions and larger numbers of Fourier coefficients. The results, shown in Table 3, indicate that PointNet++ performs poorly (GSR of 60.5% versus 96.7% for the Pile task), presumably due to the lack of equivariance in SO(3) space. Interestingly, even with spherical harmonic basis functions of degree 1, our net- work performs well (GSR of 93.9% for the Pile task), result we attribute to the capabilities of our UNet-based EquiFormerV2 V2, which effectively captures both local and global information. As the degree increases, the performance improvement from additional parameters diminishes. Although the performance of ours (I = 3, m = 2) is slightly better (GSR of 96.7% for the Pile task) than (1=2, 2), it requires nearly double the parameters. This trade-off should be considered when computing power is limited. We also amalyze the impact of larger neighborhoods B; on performance in simulation, along with the effect of different training data generation methods (Mask versus FPS) in the real world, as detailed in Appendix G.
6 Conclusion and Limitations
In this paper, we propose OrbitGrasp, grasp detection method that exploits SE(3) -qquivvraie to achieve state-of-ther -ate--ofthhe-att performance. Our model learns a continuous grasp function over 52 for each point the point cloud, Using geometric prior, we constrain potential grasp poses around the surface normal direction of each candidate contact point, forming an orbit. By evaluating densely sampled poses along the orbit with the learned grasp function, multiple good grasp poses can be identified. Simulation experiments demonstrate that our model outperforms several strong baselines across different grasping tasks and settings. Physical experiments show high grasp success rates and good generalization across diverse objects.
Our work has several limitations. First, as discussed in Appendix E, the inference time relatively long because Py PyTorch's lack of optimization for the geometric operations used in EquiFormerV2 and the need evaluate multiple point neighborhoods, Bı,..., Bg. One potential approach is to incorporate gauge equivariance [49] and predict directly in the tangent space [50], which would lower the computational complexity and dimensionality of the problem and speed up inference. Another limitation is the absence of a direct mechanism for constraining grasps to specific objects or object parts. Future work may address this challenge by integrating methods for conditioning on language or object crops.

---

Acknowledgments
This work is supported in part by NSF 1750649, NSF 2107256, NSF 2134178, NSF 2314182, NSF 2409351, and NASA 80NSSC19K1474. Dian Wang is supported in part by the JPMorgan Chase PhD fellowship. The authors would like thank Owen Howell and David Klee for their valuable discussions and advice on designing the SE(3)-equivariant network and preparing the paper manuscript.
References
[1] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and R. Platt. Sample efficient grasp learning using equivariant models. arXiv preprint arXiv:2202.0944440 2022.
[2] H.-S. Fang. C. Wang. H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y. Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 2023.
[3] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2901-2910, 2019.
[4] M. Sundermeyer, A. Mousavian, R. Triebel, and D) Fox. Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13438 13444. IEEE, 2021.
[5] M. Breyer. J. J. Chung. L. Ott, R. Siegwart, and J. Nieto. Volumetric grasping network: Real- time 6 dof grasp detection in clutter. In Conference on Robot Learning, pages 1602-1611. PMLR, 2021.
[6] U. Asif. J. Tang, and S. Harrer. Graspnet: An efficient convolutional neural network for real- time grasp detection for low-powered devices. In IJCAI, volume 7, pages 4875-4882 2.18.
[7] A. Ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research, 36(13
2017.
[8] H. Huang, D. Wang, X. Zhu, R. Walters, and R. Platt. Edge grasp network: A graph-based se (3)-invariant approach to grasp detection. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3882-3888, IEEE, 2023.
[9] R. Zurbrügg, Y. Liu, F. Engelmann, S. Kumar, M. Hutter, V. Patil, and F. Yu. Icgnet: A unified approach for instance-centrie grasping. arXiv preprint arXiv:2401.09939 2024.
[10] E. B. Dam, M. Koch, and M. Lillholm. Quaternions, interpolation and animation, volume 2. Citeseer, 1998.
[11] Y. Zhou, C. Barnes, Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. CoRR, abs/1811.00003,, 2018. URL http:///arriv.orr/bbb//// 07035.
[12] M. Jia, Wang, G. Su, D. Klee, X. Zhu, R. Walters, and R. Platt. Seil: Simulation-augmented equivariant imitation learning. In 2023 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 1845-1851. IEEE, 2023.
[13] A. Simeonov, Y. Du, Y.-C Lin, A. R. Garcia, P. Kaelbling, T. Lozano-Pérez, and P. Agrawal. Se equiqvvaat relational rearrangement with neural descriptor fields. In Conference Robot Learning, pages 835-846. PMLR, 2023.

---

[14] A. Simeonov, Y. Du, A. Tagliasacchi. J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 6400. IEEE, 2022.
[15] H. H. Nguyen, A. Baisero, Klee, D. Wang, Platt, and Amato. Equivariant reinforce- ment learning under partial observability. In Conference on Robot Learning, pages 3309-3320. PMLR, 2023.
[16] D. Wang, M. Jia, X. Zhu, R. Walters, and R. Platt. On-robot learning with equivariant models. arXiv preprini 1xXx::2003.04222 2022.
[17] Y.-L. Liao, B. Wood, A. Das, and T. Smidt. Equiformer22: Improved equivariant transformer for scaling to hegrer-degree representations. arXiv preprint arXiv:2306. 20059 2023.
[18] Z. Jiang, Y. Zhu, M. Svetlik, Fang, and Y. Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. arXiv preprint arXiv:2104.01542) 2021.
[19] H.-S. Fang, C. Wang, M. Gou, and C. Lu. Graspnet-Ibillion: A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11444-11453, 2020.
[20] N. Thomas, Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint 15t2:18802.8211 2018.
[21] C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas. Vector rons: A general framework for (3)-cquivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22200-1220,, 2021.
[22] Z. Weng, H. Lu, J. Lundell, and D. Kragic. Capgrasp: An R x SO(2)-Equivariant continuous approach-constrained generative grasp sampler. IEEE Robotics and Automation Letters, 9(4): 3641-3647, 2024. doi: 10i:10.1119//.. 10.11099AA.2222433535.
[23] S. Jauhri, I. Lunawat, and G. Chalvatzaki. Learning any-view 6dof robotic grasping in cluttered scenes via neural surface rendering. arXiv preprint arXiv:2306.07392, 2023.
[24] D. Wang, R. Walters, X. Zhu, and R. Platt. Equivariant learning in spatial action spaces. In Conference Robot Learning, pages 1713-1723. PMLR, 2022.
[25] D. Wang, R. Walters, and R. Platt. S0(2)-cquivariant reinforcement learning, arXiv preprint arXiv:2203.04439, 2022.
[26] D. Wang, Y. Park, N. Sortur, L. L. Wong, R. Walters, and Platt. The surprising effective- ness of equivariant models in domains with latent symmetry. In The Eleventh International Conference Learning Representations, 2023.
[27] T. Cohen and M. Welling, Group equivariant convolutional networks. In International confer- ence on machine learning, pages 2990-2999. PMLR, 2016.
[28) T. S. Cohen and M. Welling. Steerable cons. arXiv preprint arXiv:1612.084995 2016.
[29] H. Huang, D. Wang, R. Walters, and R. Platt. Equivariant transporter network. arXiv preprint :.Xi::222.00400 2022.
[30] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747. PMLR, 2021.
10

---

[3]] H. Ryu, H.-i. Lee, J.-H. Lee, and J. Choi. Equivariant descriptor fields: Se -3)-equivariann energy-based models for end-to-end visual robotic manipulation learning. arXiv preprint arXiv:2206.0832/, 2022.
[32] H. Huang. O. Howell, X. Zhu, D. Wang, R. Walters, and R. Platt. Fourier transporter: Bi- equivariant robotic manipulation 3d. arXiv preprint arXiv:2401.12466 2024.
[33] B. Eisner, Y. Yang, T. Davchev, M. Vecerik, J. Scholz, and D. Held. Deep se (3)-equivariant geometric reasoning for precise placement tasks. arXiv preprint arXiv:2404.13478, 2024.
[34] C. Gao, Z. Xue, S. Deng, T. Liang, S. Yang, L. Shao, and H. Xu. Riemann: Near real- time se (3)-cquivariant robot manipulation without point cloud segmentation. arXiv preprint arXiv:2403.194600 2024.
[35] H. Liang. X. Ma. S. Li, M. Görner, Tang, B. Fang, F. Sun, and Zhang. Pointnetgpd: Detecting grasp configurations from point sets. In 2019 International Conference on Robotics and Automation (ICRA), pages 3629 6629-3635. IEEE, 2019.
[36] A. W. Knapp and A. W. Knapp. Lie groups beyond an introduction, volume 140. Springer, 1996.
[37] Y.I... Liao and T. Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. arXiv preprint vrXiv:2206./199. 2022.
[38] H. Ryu, J. Kim, J. Chang. S. Ahn, Seo, T. Kim, J. Choi, and Horowitz. Diffusion-edfss Bi-equivariant denoising generative modeling on (3) for visual robotic manipulation. arXiv preprint arXiv:2300. 02685, 2023.
[39] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[40] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[41] A. Singh, J. Sha, K. S. Narayan, T. Achim, and Abbeel. Bigbird: A large-scale 3d database of object instances. In 2014 IEEE international conference robotics and automation (ICRA), pages 509-516. IEEE, 2014.
[42] A. Kasper, Z. Xue, and R. Dillmann. The kit object models database: An object model database for object recognition, localization and manipulation in service robotics. The International Journal of Robotics Research, 31(8):927-99 2012.
[43] D. Kappler, Bohg, and Schaal. Leveraging big data for grasp planning. 2015 IEEE international conference on robotics and automation (ICRA), pages 4304-4311. IEEE, 2015.
[44] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo. P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643. 2023.
[45] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 100 2017.
[46] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:
2016.
[47] G. E. Hinton, N. Srivastava, A. Krizhevsky, Sutskever, and R. R. Salakhutdinov. Im- proving neural networks by preventing co-adaptation of feature detectors. arXiv preprint 100.00100, 2012.

---

[48] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
[49] M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
[50] De Haan, M. Weiler, T. Cohen, and M. Welling. Gauge equivariant mesh enns: Anisotropic convolutions on geometric graphs. International Conference on Learning Representations, 2020.
[51] S. Passaro and C. L. Zitnick. Reducing so (3) convolutions to so (2) for efficient equivariant gnns, In International Conference on Machine Learning, pages 27420-27438. PMLR, 2023.
12

---

A Mask-Based Sample Generation
To obtain object masks on point cloud, all RGB-D images are first cropped to focus on the workspace area. The Segment Anything Model (SAM) is then applied each RGB image to acquire 2D segmentation masks of all objects. Given the pixel-wise correspondence between the RGB image, masks, and depth map, we can map these 2D masks onto the point cloud. Note that each point P: may belong to multi- ple masks, such as P. M, and Ps E Mg. where M, and M, are different masks. By combining the point clouds and mask informa- mask-centric
Figure 5. The mask-centric point cloud representation. Each mask rendered distinct color. Points that belong to multiple masks are rendered with only one color.
tion from multiple cameras, point cloud representation is reconstructed. Af- ter preprocessing and filtering, we finally obtain the refined mask-centric point cloud scene, illustrated in Figure 5.
In our implementation, use the centers of the individual object masks center points construct B, for generating grasp poses during training data collection. This method provides more balanced object-level point sampling compared to the FPS-based method, as it assigns center point to each object (mask), regardless of size. In contrast, FPS tends to allocate more points to larger objects and fewer to smaller ones, which leads to an uneven distribution of grasp points. Additionally, avoids generating many unstable poses on the edges and corners of objects. Although FPS still used construct B, during inference, we found that our model effectively reduces the probability of grasping the edges or corners of objects.
B Model Architecture
The detailed architecture of EquiForrmrr22, mentioned in Implementing! I as an Equivariant Neu- ral Network in Section 4, is illustrated in Figure 6. Compared to the original structure, we have made several key modifications. The original SO(3) embedding has been replaced with a single SO(3)- equivariant linear layer that directly takes the point coordinates and normal directions as input. This embedding is then used in subsequent blocks. After the first equivariant graph attention block, we apply FPS to downsample the point cloud. For each downsampled point, we use KNN to find its neighbors and build edges between them. During upsampling, we reverse these edges from each downsampling block by swapping the source and destination of these edges. This allows us grad- ually transfer information from the downsampled points to those points in the upsampling blocks. Meanwhile, this process aggregates information from distant points and gradually extends this in- formation to local points. The theoretical foundation and mathematical proof of EquiFormerV2 can be found in [17, 37, 51].
Figure visualizes the process of evaluating grasp poses along the orbit of point p the point cloud. Starting with the point's normal vector N, and the of Fourier Coefficients {F7m output by the network, the spherical harmonics basis functions are multiplied with these coefficients to reconstruct the spherical signal on S², according to Equation 1. This signal serves the grasp quality function, i.c., fp: 52 + R. The grasp sampler then uses the normal vector np to generate a set of approach vectors {ra E Op- = r3 E 5t n, r3 = 0). representing possible grasp poses. These vectors are then used to query fp to yield the quality of each grasp pose.
13

---

Input Point Cloud
SO(3)
X2
X;
Tij
71
Embedding
Concutenate
Lincar
LayerNorm
WignerD
X
Equivariant Graph Attention
Graph Attention Layer
(c) SO(3) Embedding
N
X
X
aij
Vij
LayerNorm
Gatingg Lincar
Foed Forward
Network
WignerD
Linear
Aggregate
SZActivation
LayerNorm
Lincar
Linear
Output Layer
(a) EquiFormerV2
(b) Equivariant
(d) Feed Forward Network
Architecture
Graph Attention
Figure 6. Overview EquiFormerV architecture. (a) shows overall structure of EquiFormerV2, while (b), (c), and (d) illustrate the submodules of (a). Multiple EquiFormerV2 blocks, incorporating FPS and KNN layers for connectivity, are stacked to form our UNet-style architecture.
Quality Grasp Over Orhit
SH
fy:
query
Orbit Pose Sampler
Normal of the point P
Sampled poses O
Figure 7. Visualization of grasp poses sampling and evaluation. To enhance clarity, this procedure is il- lustrated with single point p in the point cloud. The Fourier coefficients of p, denoted [F7r ), and the spherical harmonics basis functions SH. are used to reconstruct the grasp quality function fy: : S -> R on the 2-sphere S² described in Equation 1. The Orbit Pose Sampler generates potential grasp poses, which then evaluated using f-). The cirele at top right, viewed along normal direction of point p, shows the grasp quality each sampled pose, with redder colors indicating higher quality.
14

---

C Additional Details in Inferring Grasp Pose
The spherical signal fps output by the model, evaluates grasp orientations via the approach vector r3- Notice that the unit vector r3 a direction that can be described by spherical coordinates (0, o). While fp may hypothetically be evaluated any u E S in practice. we evaluate only at O, during training and inference. Using Equation I, fp is evaluated over O, and the pose that maximizes fp is selected through dense sampling of Op. To prevent the object from rotating when one finger makes contact before the other, we apply offset along the gripper's closing direction to center the object between the fingers.
D Simulation Additional Details
We provide several figures (Figure 8, 9) to give more information about the simulation environment and grasp pose evaluation process. Figure 8(a) shows a pile scene with objects randomly dropped onto the table while Figure 8(b) indicates packed scene with objects placed upright in random poses. Figure (c) presents the downsampled point cloud, the orbital sampled grasp poses, and the optimal pose within the entire scene, respectively.
(a)
(b)
Figure 8, (a) and (b) illustrate examples of "pile" and "packed" scenes, respectively.
b
(c)
Figure 9. (a) downsampled point cloud. (b) sampled grasp poses at points, pose the highest grasp quality in each of the 10 B, where a more intense green color indicates higher quality. For simplicity, only 18 of the 36 sampled poses point are displayed. (c) The best grasp poses selected from all B B
15

---

Setting
Data
Model
Packed
Pile
GSR (%)
DR (%)
GSR (8)
DR (8)
==
94.2
96.3 1.3
97.0 0.7
6M
= 2.m
97.4 + 0.5
98.5 + 0.6
96.3 =
97.6 0.7
1 3,m
98.3 = 0.7
98.8 0.6
96.7 1.1
97.9 1 0.5
Single- View
4:
96.7
94.7
97.1 0.4
3M
98.1 = 0.6
9..
96.6
98.0 0.5
3,m
98.4 = 0.5
98.8 0.3
96.3 0.3
97.7 0.7
+:
96.0
94.2
96.6
IM
96.9 = 0.5
0.8
93.8
96.2 1.2
3,m
97.3 1 0.7
18.7.10.4
94.1 1 0.6
97.0 1. 0.3
1.m
98.3 = 0.7
98.4
97.7
97.9 0.7
98.2
99.0
0.3
98.5
98.2 0.6
Multi- View
1.m
98.1 0.6
08.6::0.2
97.8 0.5
97.5 0.8
2,m
98.6
0.3
97.9
97.9
98.6 0.3
0.5
98.6
98.5 0.5
1,m
97.7 1: 0.7
0.8
97.3 = 0.3
97.8 1: 0.6
2,m
98.7
99.1
98.2
99.0 0.7
9..
98.4
98.0 + 0.4
Table 4. Comparison of results across different amounts training data, camera settings, and model degrees.
E Efficiency and Inference Time Analysis
Training Data Efficiency. While we 6M labeled grasp poses (both positive and negative) to train model to ensure sufficient data for learning and to prevent data limitations from being bottleneck, our method also performs well with less training data. To demonstrate this, we trained our model with 3M data points (matching the amount used in EdgeGrasp [8] and ICGNet (9]) and with IM data points (a significantly smaller dataset). Table 4 indicates that our model consistently outperforms the baselines across all scenarios. Even with just IM data points, our method still performs strongly and surpasses other methods, as seen Table 1. These findings emphasize the efficiency and robustness of our method.
Inference Time. Rapid inference of grasping poses is critical for grasp detection methods. Al- though our approach requires dense sampling of grasp poses for each point in the point cloud, this process is efficient because only requires a single forward pass through the model per region B;- Specifically, one forward pass yields a vector of spherical harmonic coefficients at each point in B;. To find the optimal grasp, identify the argmax of the grasp quality function by sampling values S and taking the maximum. This sampling is performed efficiently through matrix multiplication with spherical basis functions, as described in Equation which eliminates the need for additional forward passes.
To validate our approach efficiency, we report the inference, pre-processing, and post-processing times under a single-view setting in Table 5. For 1, object-centric grasping employed, where segmentation model is used to locate the target object and crop the point cloud for prediction. This is similar to conditional grasping, as it focuses on specific target. For A: = 10, 10 regions B are generated. This configuration helps our method to understand the entire scene and generate potential grasp poses for all points, which suitable for scenarios with multiple objects or when the target is not predefined. The quantitative results suggest that our method can respond within 0.25 seconds when grasping target object and evaluate grasp poses for nearly the entire scene in seconds-fast enough for typical open-loop grasping tasks. While our inference time comparable to VN-Edgegrasp for & 1, it is slower for & 10. However, given that our method outperforms theirs in accuracy and can evaluate almost the entire scene, this difference in speed is acceptable.
16

---

Model
number of
Inference
Lalency
1
0.126s
0.133
1 3, =
0.856
0,678
k 1
0.122
0.1
2,
k 10
0.743
0.659
1 0.095
0.117 5
1 1,m==
0.455
0.654
VN-EdgoGrusp [8]
x
0.153
0.154
Table 5. The runtime for different degrees of SH and the baseline is measured on an RTX 4090 GPU with Intel 19-12900k CPU. 'Inference' refers to the model's running time, while 'Latency' includes both pre- processing and post-processing times.
F Physical Experiments Additional Details
Implementation Details. Although our design initially selects the grasp pose with the highest Z value after filtering, observed that this highest Z pose sometimes be unstable the physical robot, unlike in simulations. To address this, since our method generates series of grasp poses for each point, we also consider poses within a 3cm range below the highest one. If pose within this range offers the best grasp quality (i.e., surpasses the highest pose and all other candidates), we select instead. This approach effectively reduces failures caused by weak grasps and mitigates the sim-to-real gap.
Failure Mode Analysis. the single-view setting, we observe that the GSR of our method for the Packed scene is lower than that for the Pile scene, which inconsistent with the simulation results. The primary failure mode (5/8) involves white bottle that blends with the table mat color. This blending results in inaccurate depth and point cloud shape estimation by the camera, which prevents the gripper from being inserted deeply enough to provide sufficient friction. For the Pile task, the primary reason for failure is the thickness of the objects. Thin objects tend to merge with the table in the point cloud, distorting the shape of objects and leading to inaccurate normal estimation and unreasonable poses generated by our model. Additionally, smooth surfaces and specific shapes, like those of stones, lead insufficient friction and cause them to slip from the gripper. In the multi- view setting, the overall tendency of failure similar to the single- view setting. While multiple perspectives help reduce distortion, introducing more cameras also introduces calibration errors. These errrr,, in turn, transfer to noises that appear in the point cloud.
G Full Ablation Results
Larger Point Cloud as Input to Provide Sufficient Context. As mentioned Sampling k Center Points Section using larger point cloud B, (N,(,, m) instead of just the local point cloud B, = B(c,, r1) is crucial for eliminating boundary effects by providing more context for evaluating the grasp quality of each point in B,. To evaluate this, we compared the performance of these two input formats. The results, shown in Figure 10, indicate that using the larger point cloud input significantly improves performance, with a 40% reduction in validation loss and a 10% increase in prediction accuracy. These findings highlight the importance of larger point clouds for providing sufficient context.
Performance Comparison Between Mask-Based and FPS-Based Training Data. We compared the effects of mask-based versus FPS- based training data. Although both methods perform similarly in simulations, differences appear in real-world experiments. shown in Figure 11, with the same input, the grasp quality distribution from the mask-based trained network is more uniform and cen- tered around the object's 's center of mass (e.g.. the banana, hammer, and shoe). This indicates that the mask-based trained network can evaluate a wide range of grasp poses more effectively. In contrast,
17

---

Validation Loss for Singleeview Training
Prediction Accuracy Validation Set
0.50
0.86
0.45
0.82
LO55
neighborhoeds
0.40
0.78
0.35
0.30
no larger neighbohoods
11
Epoch
Epoch
(a). The validation loss trends.
(b). The predicted accuracy of all validation poses.
Figure Ablation Study Results with Larger Point Cloud Input. The prediction accuracy on validation set is calculated by evaluating all grasp poses, including both positive and negative poses, to determine the overall accuracy.
the FPS-based trained network tends to produce grasps biased toward the object's edges or specific small regions. These edge-focused poses are generally less stable than those around the center of mass. We interpret this result of FPS-based training lacking object-centric awareness, which causes the network to focus on specific areas and struggles assess grasps across the object. Mask- based training data, however, incorporates object-centric information. This enables the network to evaluate grasp poses more evenly across the entire object. Therefore, even when FPS is used for input during inference, the network trained with mask-based data maintains enough robustness to handle unseen geometric variations.
18

---

Original Point Cloud
With Masks
Without Masks
Figure 11. Grasp quality distribution for different training data generation strategies. The highest grasp quality all sampled poses at each point represents that point's quality. left column displays the original point cloud. The middle column shows predictions from the network trained with mask-based data. The right column shows predictions from the network trained with FPS- -PSed data.
19


Reviews:
[{"review_id": "26Gt6wWs93", "reviewer": "Reviewer_RMP5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed to learn the grasping quality through an SE(3)-equivariant model. Using fourier coefficients as the output, the learned grasping quality is a continuous function on the 2-sphere, from which the authors claim better generalization.", "summary_of_recommendation": "The limitation in efficiency could fundamentally limit the value of this work.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "See above. Please answer the questions in terms of efficiency and the surface-normal heuristics.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EVHbFWjr4Z", "reviewer": "Reviewer_k5Je", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper develops an SE(3) equivariant grasp synthesis network that takes in point cloud input and generates full 6DoF grasps. The approach learns a mapping from each point in the point cloud to a grasp quality function on the sphere centering at the point, where the function is encoded via spherical harmonics. At inference time, the point cloud is separated into small regions of point clouds, and each is inference separately. The authors compare the results against another SE(3) invariant", "summary_of_recommendation": "While there are some limitations, such as the need to split large point clouds and the focus on tabletop settings, these do not significantly reduce the paper's contributions. The work presents a new perspective on grasp synthesis and shows promise for learning better grasp representations. Thus I recommend accepting the paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. What are some challenges you run into when performing sim2real transfer? How large is each $B_i$? Does increasing or decreasing the number of sub-point clouds change the performance? \n2. Given that grasps are sampled on the point cloud (object surface) instead of inside the object, will this parametrization result in more collision and object rotation when closing the gripper?\n3. In the simulation, all the $B_i$ are generated via SAM: “In our experiments, we generated the center points using object masks for training and FPS for inference.” (line 155-156). Why isn’t SAM run in real?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZD5dF4lVJi", "reviewer": "Reviewer_Zbw2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The work proposes an $SE(3)$-equivariant model predicting grasp qualities, given a point-cloud representing an object. The grasp quality is modelled as a function over the sphere, decomposed in spherical harmonics.", "summary_of_recommendation": "I believe that the approach, despite interesting and original, suffers from the technical issues explained above. Therefore, I lean towards rejecting it in the current state.", "technical_quality": 1, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "I would like the authors to address the questions above regarding 1) the constraint and redundancy in the model, and 2) the normalization issue, especially as related to the issue in the optimization problem.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to CoRL 2024. The reviewers appreciated many aspects of the paper, including its clarity, use of formalisms, the novelty of the approach, method aspects (e.g., use of spherical harmonics), and the wealth of real-world and simulated demonstrations and comparisons to baselines.

Reviewers raised concerns about efficiency, heuristic and sampling limitations (including imposed constraints), and would like to see more experiments requiring full 6-DOF grasping.

However, the reviewers and ACs found the authors' thoughtful and thorough responses convincing. While there are lingering concerns (which we encourage the authors to address for the camera-ready version), several reviewers raised their final rating.
Decision: Accept