OrbitGrasp: SE(3)-Equivariant Grasp Learning

Authors: boce  hu
xupeng  zhu
dian  wang
zihao  dong
haojie  huang
chenghao  wang
robin  walters
robert  platt
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 04:37:34

Abstract: While grasp detection is an important part of any robotic manipulation pipeline, reliable and accurate grasp detection in $\\mathrm{SE}(3)$ remains a research challenge. Many robotics applications in unstructured environments such as the home or warehouse would benefit a lot from better grasp performance. This paper proposes a novel framework for detecting $\mathrm{SE}(3)$ grasp poses based on point cloud input. Our main contribution is to propose an $\mathrm{SE}(3)$-equivariant model that maps each point in the cloud to a continuous grasp quality function over the 2-sphere $S^2$ using a spherical harmonic basis. Compared with reasoning about a finite set of samples, this formulation improves the accuracy and efficiency of our model when a large number of samples would otherwise be needed. In order to accomplish this, we propose a novel variation on EquiFormerV2 that leverages a UNet-style backbone to enlarge the number of points the model can handle. Our resulting method, which we name OrbitGrasp, significantly outperforms baselines in both simulation and physical experiments.

Main paper: 
 introduction: 
 Figure 1 . We infer an orbit of grasps (yellow ellipse) defined relative to the surface normal (red arrow) at the contact point (pink dot). Since our model is equivariant over SO(3), the optimal pose (represented by the solid gripper) on the orbit rotates consistently with the scene (left and right show a rotation by 90 degrees).The ability to detect and evaluate good grasp poses in a manipulation scene is a critical part of robotic manipulation. Despite extensive recent work in the area, e.g. [1, 2, 3, 4, 5, 6, 7, 8, 9] , grasp detection is still not accurate and reliable enough for many practical applications. A key challenge here is effective reasoning over hand poses in SO(3), i.e. the three dimensions of orientation spanned by SE (3) . Simply representing orientations in SO(3) can be a challenge due to classic problems like gimble lock [10] and discontinuity [11] . More importantly, it would be very helpful to be able to infer grasp quality over a continuous range of orientations in SO(3) -something which is a major challenge for conventional grasp methods. Finally, since SE(3) grasping problems are often equivariant (i.e. symmetric) in SO (3) , it is often desirable to encode symmetry assumptions into the model. This paper addresses these challenges by leveraging recent innovations in equivariant point models [12, 13, 14, 15] . Specifically, we use a model that maps each point in a point cloud to a grasp ⋆ Equal contribution. quality function over the 2-sphere S 2 . For each point in the cloud, this function encodes grasp quality for the space of possible hand approach directions toward that point. An important question here is how to represent this quality function over S 2 . Following recent work [1, 16, 8] , our approach is to leverage the spherical harmonic basis. Specifically, at each point in the cloud, the neural network outputs a vector of Fourier coefficients to the spherical harmonic basis functions, thereby defining a grasp quality function over hand approach directions. With this learned per-point grasp distribution, we can evaluate the quality of a large number of potential grasp poses quickly and easily, thereby more easily locating a grasp appropriate for a given downstream manipulation task.To summarize, this paper introduces several key contributions. First, we propose a novel method of using spherical harmonics to reason about an orbit (i.e. a S 1 manifold embedded in S 2 ) of grasp approach directions defined relative to the surface normal at each contact point in the point cloud, as shown in Figure 1 . Second, we enhance the recently developed SE(3)-equivariant model, EquiFormerV2 [17] , by incorporating a U-Net backbone and thereby enabling the model to accommodate a larger number of points in the point cloud. Finally, we evaluate our method against multiple established baselines [5, 18, 8, 9] , using the benchmark tasks proposed in [5] . The results indicate that our approach, which we name OrbitGrasp, convincingly outperforms the baselines in both single-view and multi-view settings, in both simulation and physical experiments.2 Related Work SE(3) Grasp Detection: Current 6-DoF grasping tasks in cluttered tabletop scenarios primarily use a volumetric-based or point cloud-based scene representation as input and output one or more optimal grasp poses [7, 3, 19, 2] . Volumetric Grasping Network (VGN) [5] and Grasp detection via Implicit Geometry and Affordance ( GIGA) [18] use 3D convolutional models to reason about a 3D truncated signed distance function (TSDF). However, these methods suffer from high memory consumption and resolution limitations. In contrast, point clouds can provide higher resolution but are more difficult to reason about due to the lack of structure. One approach to adding structure is to leverage equivariant point cloud models [20, 21] . For example, EdgeGrasp [8] introduced a method based on a vector neuron model. CAPGrasp [22] is another equivariant model that samples grasp candidates while constraining the gripper approach direction to be within a certain angular distance from the surface normal at the associated point. Other recent grasping methods leverage surface reconstruction of the object to be grasped, including NeuGraspNet [23] and ICGNet [9] .Equivariance in Robot Learning: Using symmetry, i.e. equivariance, in robot learning tasks has recently been used to improve sample efficiency and generalization. [24, 25, 26] introduce equivariant neural network models in the form of steerable convolutional layers [27, 28] to SE(2) manipulation tasks. [29] applies similar models to transporter network [30] , making it bi-equivariant and significantly improving sample efficiency. EDF [31] introduces an equivariant energy model that produces a continuous distribution over pose. Fourtran [32] extends [29] from SE(2) to SE(3) using a Fourier representation of rotations. [33] develops a novel, dense, interpretable representation for relative object placement tasks. RiEMann [34] presents the first near real-time SE(3)-equivariant robot manipulation framework for point cloud inputs that addresses the problem of slow inference over the SE(3)-manifold. Most relevant to this paper are [1] who obtain SE(2) equivariance using steerable convolutional layers and [8] who use vector neurons to obtain SE(3) equivariance.The method proposed in this paper, OrbitGrasp, is distinct from the work described above in a couple of important ways. First, prior work in grasp detection is generally sample-based [35, 7, 6] , i.e., they evaluate grasp quality for a (large) finite set of samples or discrete voxel/pixel locations. In contrast, OrbitGrasp evaluates grasp quality for a continuous range of approach directions by inferring the parameters of spherical harmonic basis functions over S 2 . This is simultaneously more computationally efficient and more precise than sample-based methods. Another distinction between OrbitGrasp and prior work is that the representation of rotation in terms of high degrees of spherical harmonics encodes continuous equivariant constraints more accurately and, therefore, models the grasp function better. 1. The Orbit Pose Sampler generates multiple poses for each p perpendicular to the surface normal np and queries corresponding fp(•) to evaluate these grasp qualities along the orbit. The grasp with the highest quality is then selected, thereby producing the optimal grasp pose a * , as shown on the right. 
background: 
 We introduce concepts from group theory and SE(3)-equivariant neural networks, which are key to our method. For additional background, see, e.g., [36] .Our work focuses on the special Euclidean group SE(3) = SO(3) ⋉ R 3 of 3D rotations and translations. A representation of a group G is a homomorphism ρ : G → GL(V ) mapping each group element g to an invertible linear operator on the vector space V . The representation (ρ, V ) is an irreducible representation, or irrep, if no non-trivial subspace W ⊂ V is closed under the action of G. The irreps of SO(3) are classified by positive integers l ∈ Z ≥0 with the irrep V l of type l having dimension 2l + 1. For example, l = 1 gives the standard rotation action on R 3 . The group action on V l is given by the Wigner D-matrices D l : SO(3) → GL(V l ) of size (2l + 1) × (2l + 1). Traditionally, the Wigner D-matrices are parameterized by Euler angles (α, β, γ) and indexed symmetrically about 0 as D l m ′ m (α, β, γ) where -l ≤ m, m ′ ≤ l. Spherical Harmonics: The real spherical harmonics (SH) are functions on the sphere Y m l : S 2 → R indexed by l ∈ Z ≥0 and -l ≤ m ≤ l. As a set, they define an orthonormal basis of the Hilbert space of square-integrable functions on the sphereL 2 R (S 2 ) = {f : S 2 → R : S 2 |f | 2 < ∞}. That is, given f : S 2 → R, we can write f (θ, ϕ) = ∞ l=0 l m=-l F m l Y m l (θ, ϕ)where F m l are the Fourier coefficients of f . For a finite truncation at l ≤ L, the total number of Fourier coefficients is (L+1) 2 . The mapping FT : f → {F m l } is known as the spherical Fourier transform. Approximating f by storing the F m l up to degree l ≤ L gives an efficient way to encode the spherical signal f . The spherical harmonics are compatible with the SO(3) action on the sphere, making them suitable for use in SO(3)-equivariant networks. The rotation of a spherical signal f can be computed in Fourier space using Wigner D-matrices. Let g ∈ SO(3). Define f ′ = g • f to be the rotated spherical signal defined(g • f )(u) = f (g -1 • u) for u ∈ S 2 . Denote the Fourier coefficients of f ′ by FT(f ′ ) = {F m′ l }.Then in terms of the Fourier coefficients of f we have F ′ l = D l (g)F l where F l denotes the vectorF l = (F m l ) l m=-l . EquiFormerV2: EquiFormerV2 [17] is a SE(3)-Equivariant Graph Neural Network (GNN)that addresses the high computational costs and poor generalization of higher-degree irreps in EquiFormer [37] . Unfortunately, since EquiFormer and EquiFormerV2 were developed for applications in computational biology, they have a hard time scaling to point cloud data, which generally contain more points than molecules do atoms. To address this limitation, our paper proposes a variation of EquiFormerV2 that uses a U-Net architecture to increase the capacity of the model. Compared to [38] , our new model can reason about higher-degree spherical harmonic functions. 
method: 
 Problem Statement: Given a point cloud P ⊂ R 3 captured by one or multiple depth cameras, our objective is to identify a set of good grasp poses. Specifically, we want to estimate a function Γ : (P, a) → [0, 1] which maps the point cloud P and the hand pose a ∈ SE(3) to the probability of successful grasp. Notice that we expect Γ to be invariant to translations and rotations g ∈ SE(3), i.e. we expect Γ(g • P, g • a) = Γ(P, a). This reflects an assumption that the probability that a grasp is successful does not change when both the scene and the grasp pose transform in concert.Summary of Approach: Figure 2 illustrates our model and approach. First, we downsample and crop the point cloud into a small number of point neighborhoods B 1 , . . . , B k . Then, we evaluate our model for each B i . For each point p ∈ B i , the model predicts the grasp quality over the space of possible hand approach directions, represented as a function f p : S 2 → R. We densely sample this function over an orbit of approach directions orthogonal to the object surface normal at p and obtain a single grasp that maximizes quality over the orbit. The result is a single-hand orientation in SO(3) at each point in the cloud, corresponding to a good grasp at that point. This produces a dense sampling of grasps that can be filtered further for grasping or other downstream tasks. Figure 7 in the Appendix provides a detailed, zoomed-in visualization. This approach has several important advantages. First, since our model outputs a continuous distribution over S 2 , it generalizes better over orientation than would a sample-based approach. Second, since our model is SE(3)-equivariant, it incorporates problem symmetry as an inductive bias into the model. Finally, the approach only considers grasps that make contact parallel to the object surface normal, thereby incorporating an additional geometric prior.Sampling k Center Points: We prepare the point cloud for processing by the prediction model as follows. First, we downsample the input point cloud P to a tractable number of points P (around 4k to 6k). Rather than passing the entire point cloud through our model (which would be expensive), we define smaller point clouds B 1 , . . . , B k ⊂ P which we evaluate over separately (k is usually around 10). The B i are defined as neighborhoods of k center points c 1 , . . . , c k from P selected as follows. If a segmentation of the point cloud into objects is available, then these k center points could be the 3D positions of the centers of individual object masks. Otherwise, these centers could be random samples obtained using farthest point sampling (FPS). In our experiments, we generate center points using object masks during training to enhance the model's object-centric awareness, and FPS during inference to demonstrate generalization and performance without segmentation. A detailed description of our mask-based sample generation is provided in Appendix A. We then construct each B i as a neighborhood around c i as follows. First denote the set of points contained within radius r l ball centered at c i by B i = B(c i , r l ) = {p ∈ P | ∥p -c i ∥ ≤ r l } for a parameterized radius r l . Since the grasp quality at a point p depends on the geometry of a sufficiently large neighborhood of p to avoid boundary effect, we then set B i = N (c i , m) to be the m nearest neighbors of c i , where m is chosen large enough such that B i ⊂ B i and B i is larger by a fair margin. The larger point cloud B i is passed as input to the model, but the output grasp quality is only inferred over the smaller point cloud B i . This assures inference is only performed on points with sufficient geometric context.Representing Grasp Quality Over S 2 : We model the grasp quality function Γ : (P, a) → [0, 1] with a neural network Γ :B i → f p : S 2 → R | p ∈ B i that takes each of the k neighborhoods B 1 , . . . , B k in P as input.For each B i , the model outputs a spherical function f p : S 2 → R for each p ∈ B i . The function f p represents the grasp quality over all approach directions in S 2 at the point p. The function f p is represented by Fourier coefficients of the spherical harmonics,EQUATIONwhere Y m l (θ, ϕ) denotes the spherical harmonic of degree l and order m evaluated at the point on S 2 defined in spherical coordinates (θ, ϕ). Concretely, the model outputs the set of Fourier coefficients {F m l,p } for at each p ∈ B i . Implementing Γ as an Equivariant Neural Network: We implement Γ using a modified version of EquiFormerV2 [17] , a fully SE(3)-equivariant, GNN-based network designed for handling nodebased data. In the original EquiFormerV2 [17] , all nodes are fully connected. However, in order to handle the larger number of points in our point clouds, we developed a UNet-style version of the model, which has a greater capacity (inspired by [38] ). Figure 2 illustrates the model architec-ture. Given the point cloud B i , successive blocks in our model sequentially downsample the points using FPS. As the number of points decreases, the radius of the edge in the graph increases, allowing distant points to communicate, expanding the receptive field per block, and capturing global features. To recover the features of each point in the original point cloud after downsampling, we upsample point features back by inverting the edges: during downsampling, we record the source and target edges in each block, and during upsampling, we swap these edges. This ensures that features gathered during downsampling are effectively propagated back to the original points. In addition, we employ skip connections between each downsample block and its corresponding upsample block to aggregate features and prevent degradation. Details of the architecture are in Appendix B. This model satisfies the desired SE(3) equivariance constraint described in Section 4, 3), if the input point cloud is rotated gB i , the equivariance constraints ensure the output Fourier coefficients are also transformed D l (g)F l,p . Evaluating at a transformed grasp g • u thus gives the same quality of a successful grasp Inferring Grasp Pose: Since our neural network model only infers grasp quality over S 2 , we must obtain the remaining orientation DoF somehow. Following [8] , we accomplish this by constraining one of the two gripper fingers to make contact such that the object surface normal at the contact point is parallel to the gripper closing direction (see Figure 3 ). Specifically, for a point p ∈ B i ⊂ B i in region B i with object surface normal n p , we constrain the hand y-axis (gripper closing direction) to be parallel to n p . Therefore, valid hand orientations form a submanifold in SO(3) homeomorphic to a 1-sphere S 1 which we call the orbit at pΓ(gB i ) = g Γ(B i ) = g • f p : S 2 → R | p ∈ gB i . For g ∈ SO((g • f p )(g • u) = f p (g -1 • g • u) = f p (u).EQUATIONwhere r 1 , n p , r 3 are the columns of the 3-by-3 rotation matrix R.Valid orientations are determined by the z-axis of the gripper (the approach direction of the hand) which may be any unit vector perpendicular to n p . We may thus specify valid grasps by their approach vector r 3 ∈ O p = {r 3 ∈ S 1 : n ⊤ p r 3 = 0} since r 1 = -n p × r 3 . The details of how we evaluate the f p can be found in Appendix C.In the above discussion, notice that we have constrained two DoFs of orientation, which could suggest that our model f p should infer only one DoF of orientation, not two. This raises the question: Why is f p defined over S 2 (the spherical harmonics) rather than S 1 (the circular harmonics)? The answer is interesting and important. Defining the output of f p over S 1 would require choosing a gauge, i.e. a mapping of S 1 to the tangent plane at each point. Assuming S 1 were parameterized by an angle θ, we would need to select a vector that defined θ = 0. Notice that it would be hard to make this selection in a consistent way across the point cloud. In contrast, by defining the output of f p over S 2 , we avoid this problem because we can express this function in a single global coordinate system for all points in the cloud. One way to think about this is that we have gained consistency in representation by adding an extra dimension, i.e. going from S 1 to S 2 . 
experiments: 
 Simulation Environment: We evaluated our model in simulation using PyBullet [39] . Our setting is the same as that used in [18, 5, 8, 9] . The workspace size is a 30 cm 3 cube and contains varying numbers of objects randomly placed within it. The object dataset includes 303 training and 40 test objects from [40, 41, 42, 43] . A Franka-Emika Panda floating gripper is used to grasp objects in the workspace. We evaluated two camera configurations. The first is a single-view setting where a camera is positioned randomly on a spherical region around the workspace. The second is a threecamera multi-view setting. Following [5] , we evaluate two different grasping tasks: a Pile setting where objects are randomly dropped into the workspace and a Packed setting where objects are placed upright in random poses. For more details, see Appendix D. 1 . We compared the OrbitGrasp with various baselines in terms of grasp success rate (GSR) and declutter rate (DR), the same metrics as used in [8] . For the single-view setting, we tested pretrained models from [8] and obtained results for GIGA, GIGA-HR, and ICGNet directly from [9] . For the multi-view setting, we retrained VNEdgeGrasp and tested the pretrained VGN model for comparison. OrbitGrasp (3M) is trained on the 3M dataset, which is similar in size to that used to train ICGNet and EdgeGrasp. OrbitGrasp (6M) is trained on the full 6M dataset and reaches a slightly higher level of performance. The best results are marked with bold and the second best results are underlined.Generating Training Data: We generate data in simulation by loading a random number of objects into the workspace and then generating point clouds in both single-view (one depth camera) and multi-view settings (three depth cameras spaced evenly around the workspace). We add Gaussian noise sampled from N (0, 0.001) to the point cloud to make the model more robust to real world sensor noise. After creating the scenes, we must generate training data for the grasp model. First, object masks are obtained using the Segment Anything Model (SAM) [44] Training Details: Since our training data is segmented into objects, we use the center of every mask as the center point c i and obtain the local point neighborhood B i . During training, we balance the positive and negative grasp labels in each B i . The model is trained using the AdamW optimizer [45] , starting with a learning rate of 1e-4 and using a cosine annealing scheduler [46] . We apply binary cross-entropy loss for each grasp pose and a dropout rate [47] of 0.1 to prevent overfitting. Our network is trained for 15 epochs, taking approximately 25 hours on 22k point clouds. Each SGD step takes 0.25 seconds with a batch size of 1. The model is trained on an NVIDIA RTX 4090 GPU.Baselines: We compare our method with several strong baselines. In the single random view setting, we compare it with volumetric-based methods: GIGA and GIGA-HR (high resolution) [18] . Since GIGA was trained under a fixed view, we compare it with the results from [9] , which retrains it under a random view setting. We also compare our method with EdgeGrasp and VNEdgeGrasp [8] , two point cloud-based methods, where VNEdgeGrasp is SE(3)-invariant. Finally, we compare our method with ICGNet [9] , an approach focused on partial observation to object-centric grasping that reconstructs the full 3D shape of objects before finding grasps. In the multi-view setting, we compare our method with VGN [5] , which takes a TSDF as input and generates a single grasp pose per voxel, as well as VNEdgeGrasp [8] trained in the multi-view setting.Results: We report the comparison result in Table 1 under two different settings, single-view and multi-view, and two different tasks, Packed and Pile. For a fair comparison with the baselines, we also trained our model on a similar amount of data (3M labeled grasp poses uniformly randomly downsampled from the original 6M dataset.), consistent with EdgeGrasp [8] and ICGNet [9] . Additional performance details under different amounts of data are pro- The results in Table 1 indicate that our method outperforms all baselines across both settings and tasks in terms of both GSR and DR, in both the 3M and 6M training sets. The high GSR indicates that our model can predict accurate grasp quality. On the other hand, the high DR signifies that our model infers accurate grasp poses that do not move objects outside of the workspace. Notably, our model performs well even though point centers are selected differently at training time (centered on object segments) and test time (selected using FPS), i.e., as described in Section 4. This demonstrates that the grasp quality function Γ output by our method exhibits strong robustness to different point cloud geometries. Table 2 . We compared the results of OrbitGrasp with VNEdgeGrasp [8] using the same metrics as in the simulation experiments. Our experiment settings for single-view and multi-view are shown in Figure 4 . Sometimes, a single trial resulted in grasping two objects simultaneously.To assess our method's real-world performance, we conduct physical experiments involving two tasks under two camera settings, replicating those in the simulation. We directly transfer the trained model from the simulation to the real-world setting to evaluate the performance gap between them.Setup: As shown in Figure 4 (a), a UR5 robot arm is equipped with a Robotiq-85 Gripper. We mount and calibrate 4 RealSense D455 cameras on the table. In the multi-view setting, 3 low-mounted cameras capture the scene for full observation. In the single view setting, we use the top-mounted camera. After obtaining the point cloud, we crop it to the workspace, filter outliers, and calculate the surface normals at each point. We do not crop the table from our point clouds. Objects are placed on the table as follows. In the Packed setting, we draw 5 objects uniformly at random from the set of 10 objects shown at the top of Figure 4 (b) and place them in upright orientations with a randomly selected planar position and orientation. In the Pile setting, we draw 5 objects uniformly at random from the set of 25 objects shown at the top of Figure 4 (c) and dump them onto a tabletop out of a box. We conduct 10 rounds of experiments for the Packed scene and 4 rounds for the Pile scene for each view setting. We configure our method as follows. We use k = 10 center points selected using height-thresholded FPS to form the neighborhoods B 1 , . . . , B k . From the orbit of each point, we evenly sample 36 grasp poses and evaluate them based on the grasp quality function from the network. After filtering out unreachable and low-quality poses (i.e. grasp poses with a quality < 0.95), we select the pose with the highest Z value.Results: We baseline our method against VNEdgeGrasp [8] . The results are shown in Table 2 . In the single view Packed task, the GSR of our model outperformed VNEdgeGrasp by an average of 3.6% and by 5% in the single view Pile task. In the multi-view setting, our model's GSR outperformed by 5.1% on the Packed task and by 2.2% on the Pile task. We analyze the failure modes we encountered in Appendix F.In our ablation study, we measure the significance of network equivariance by comparing to Point-Net++ [48] , a well-known non-equivariant model under a single random camera setting, keeping all settings identical except for the network structure. For fairness, we set up PointNet++ so that it has the same number of parameters as our method. We also measure the influence of different degrees of spherical harmonics, where higher degrees correspond to more basis functions and larger numbers of Fourier coefficients. The results, shown in Table 3 , indicate that PointNet++ performs poorly (GSR of 60.5% versus 96.7% for the Pile task), presumably due to the lack of equivariance in SO(3) space. Interestingly, even with spherical harmonic basis functions of degree 1, our network performs well (GSR of 93.9% for the Pile task), a result we attribute to the capabilities of our UNet-based EquiFormerV2, which effectively captures both local and global information. As the degree increases, the performance improvement from additional parameters diminishes. Although the performance of ours (l = 3, m = 2) is slightly better (GSR of 96.7% for the Pile task) than (l = 2, m = 2), it requires nearly double the parameters. This trade-off should be considered when computing power is limited. We also analyze the impact of larger neighborhoods B i on performance in simulation, along with the effect of different training data generation methods (Mask versus FPS) in the real world, as detailed in Appendix G. 
conclusion and limitations: 
 In this paper, we propose OrbitGrasp, a grasp detection method that exploits SE(3)-equivariance to achieve state-of-the-art performance. Our model learns a continuous grasp function over S 2 for each point in the point cloud. Using a geometric prior, we constrain potential grasp poses around the surface normal direction of each candidate contact point, forming an orbit. By evaluating densely sampled poses along the orbit with the learned grasp function, multiple good grasp poses can be identified. Simulation experiments demonstrate that our model outperforms several strong baselines across different grasping tasks and settings. Physical experiments show high grasp success rates and good generalization across diverse objects.Our work has several limitations. First, as discussed in Appendix E, the inference time is relatively long because PyTorch's lack of optimization for the geometric operations used in EquiFormerV2 and the need to evaluate multiple point neighborhoods, B 1 , . . . , B k . One potential approach is to incorporate gauge equivariance [49] and predict directly in the tangent space [50] , which would lower the computational complexity and dimensionality of the problem and speed up inference. Another limitation is the absence of a direct mechanism for constraining grasps to specific objects or object parts. Future work may address this challenge by integrating methods for conditioning on language or object crops.A Mask-Based Sample Generation To obtain object masks on a point cloud, all RGB-D images are first cropped to focus on the workspace area. The Segment Anything Model (SAM) is then applied to each RGB image to acquire 2D segmentation masks of all objects. Given the pixel-wise correspondence between the RGB image, masks, and depth map, we can map these 2D masks onto the point cloud. Note that each point p i may belong to multiple masks, such as p i ∈ M a and p i ∈ M b , where M a and M b are different masks. By combining the point clouds and mask information from multiple cameras, a raw mask-centric point cloud representation is reconstructed. After preprocessing and filtering, we finally obtain the refined mask-centric point cloud scene, illustrated in Figure 5 .In our implementation, we use the centers of the individual object masks as center points to construct B i for generating grasp poses during training data collection. This method provides more balanced object-level point sampling compared to the FPS-based method, as it assigns a center point to each object (mask), regardless of size. In contrast, FPS tends to allocate more points to larger objects and fewer to smaller ones, which leads to an uneven distribution of grasp points. Additionally, it avoids generating many unstable poses on the edges and corners of objects. Although FPS is still used to construct B i during inference, we found that our model effectively reduces the probability of grasping the edges or corners of objects. 
 
 Appendix: The detailed architecture of EquiFormerV2, as mentioned in Implementing Γ as an Equivariant Neural Network in Section 4, is illustrated in Figure 6 . Compared to the original structure, we have made several key modifications. The original SO(3) embedding has been replaced with a single SO(3)equivariant linear layer that directly takes the point coordinates and normal directions as input. This embedding is then used in subsequent blocks. After the first equivariant graph attention block, we apply FPS to downsample the point cloud. For each downsampled point, we use KNN to find its neighbors and build edges between them. During upsampling, we reverse these edges from each downsampling block by swapping the source and destination of these edges. This allows us to gradually transfer information from the downsampled points to those points in the upsampling blocks.Meanwhile, this process aggregates information from distant points and gradually extends this information to local points. The theoretical foundation and mathematical proof of EquiFormerV2 can be found in [17, 37, 51] .Figure 7 visualizes the process of evaluating grasp poses along the orbit of point p in the point cloud.Starting with the point's normal vector n p and the set of Fourier Coefficients {F m l,p } output by the network, the spherical harmonics basis functions are multiplied with these coefficients to reconstruct the spherical signal on S 2 , according to Equation 1. This signal serves as the grasp quality function, i.e., f p : S 2 → R. The grasp sampler then uses the normal vector n p to generate a set of approach vectors {r 3 ∈ Op = r 3 ∈ S 1 : n ⊤ p r 3 = 0}, representing possible grasp poses. These vectors are then used to query f p to yield the quality of each grasp pose. The spherical signal f p , output by the model, evaluates grasp orientations via the approach vector r 3 . Notice that the unit vector r 3 is a direction that can be described by spherical coordinates (θ, ϕ). While f p may hypothetically be evaluated at any u ∈ S 1 , in practice, we evaluate it only at u ∈ O p during training and inference. Using Equation 1, f p is evaluated over O p and the pose that maximizes f p is selected through dense sampling of O p . To prevent the object from rotating when one finger makes contact before the other, we apply an offset along the gripper's closing direction to center the object between the fingers.We provide several figures (Figure 8, 9 ) to give more information about the simulation environment and grasp pose evaluation process. Figure 8 (a) shows a pile scene with objects randomly dropped onto the table while Figure 8 (b) indicates a packed scene with objects placed upright in random poses. Figure 9 (a)(b)(c) presents the downsampled point cloud, the orbital sampled grasp poses, and the optimal pose within the entire scene, respectively. Training Data Efficiency. While we use 6M labeled grasp poses (both positive and negative) to train our model to ensure sufficient data for learning and to prevent data limitations from being a bottleneck, our method also performs well with less training data. To demonstrate this, we trained our model with 3M data points (matching the amount used in EdgeGrasp [8] and ICGNet [9] ) and with 1M data points (a significantly smaller dataset). Table 4 indicates that our model consistently outperforms the baselines across all scenarios. Even with just 1M data points, our method still performs strongly and surpasses other methods, as seen in Table 1 . These findings emphasize the efficiency and robustness of our method.Inference Time. Rapid inference of grasping poses is critical for grasp detection methods. Although our approach requires dense sampling of grasp poses for each point in the point cloud, this process is efficient because it only requires a single forward pass through the model per region B i . Specifically, one forward pass yields a vector of spherical harmonic coefficients at each point in B i .To find the optimal grasp, we identify the argmax of the grasp quality function by sampling values on S 2 and taking the maximum. This sampling is performed efficiently through matrix multiplication with spherical basis functions, as described in Equation1, which eliminates the need for additional forward passes.To validate our approach efficiency, we report the inference, pre-processing, and post-processing times under a single-view setting in Table 5 . For k = 1, object-centric grasping is employed, where a segmentation model is used to locate the target object and crop the point cloud for prediction. This is similar to conditional grasping, as it focuses on a specific target. For k = 10, 10 regions B 1,...,10 are generated. This configuration helps our method to understand the entire scene and generate potential grasp poses for all points, which is suitable for scenarios with multiple objects or when the target is not predefined. The quantitative results suggest that our method can respond within 0.25 seconds when grasping a target object and evaluate grasp poses for nearly the entire scene in 1.5 seconds-fast enough for typical open-loop grasping tasks. While our inference time is comparable to VN-Edgegrasp for k = 1, it is slower for k = 10. However, given that our method outperforms theirs in accuracy and can evaluate almost the entire scene, this difference in speed is acceptable. the FPS-based trained network tends to produce grasps biased toward the object's edges or specific small regions. These edge-focused poses are generally less stable than those around the center of mass. We interpret this as a result of FPS-based training lacking object-centric awareness, which causes the network to focus on specific areas and struggles to assess grasps across the object. Maskbased training data, however, incorporates object-centric information. This enables the network to evaluate grasp poses more evenly across the entire object. Therefore, even when FPS is used for input during inference, the network trained with mask-based data maintains enough robustness to handle unseen geometric variations.  


Decision: Thank you for your submission to CoRL 2024. The reviewers appreciated many aspects of the paper, including its clarity, use of formalisms, the novelty of the approach, method aspects (e.g., use of spherical harmonics), and the wealth of real-world and simulated demonstrations and comparisons to baselines.

Reviewers raised concerns about efficiency, heuristic and sampling limitations (including imposed constraints), and would like to see more experiments requiring full 6-DOF grasping.

However, the reviewers and ACs found the authors' thoughtful and thorough responses convincing. While there are lingering concerns (which we encourage the authors to address for the camera-ready version), several reviewers raised their final rating.

Reviews: review: title: title: Major concern over the efficiency of sampling-based training and inference
paper_summary: paper_summary: This paper proposed to learn the grasping quality through an SE(3)-equivariant model. Using fourier coefficients as the output, the learned grasping quality is a continuous function on the 2-sphere, from which the authors claim better generalization.
main_review: main_review: The content of the paper is organized well. The presentation of the problem and the method is mostly clear. The literature review and background are proper and help readers understand the context. 

Methodology-wise, my major concern is efficiency. While the Fourier coefficient output models a continuous function, the pooling of this function relies on sampling. It makes the evaluation of the learned model more tedious. The grasping quality of a single point needs multiple evaluations of the sampled approaching orientations, and it has to be done for all points to find the optimal grasping pose. This is a very inefficient process, both for training and inference. In order to train the network, this paper generated 6M grasp poses for each camera setting, which is a lot for the grasping-learning task. In my understanding, the superior performance in the tables is likely due to the dense sampling of the grasping points and orientations, which could significantly sacrifice efficiency. The authors should include the actual inference time for accomplishing the task, in comparison with the baselines. 

On the other hand, the heuristics used in sampling the grasping pose could be a limiting factor. The hand y-axis Is set to the surface normal direction at the grasping point. While it could alleviate the efficiency problem, it may sacrifice the flexibility of grasping. The surface normal of a point could be unreliable due to observation noise or specific local geometry. Could you comment and analyze on this limitation?
questions: questions: See above. Please answer the questions in terms of efficiency and the surface-normal heuristics.
limitations: limitations: 2
review_summary: review_summary: The limitation in efficiency could fundamentally limit the value of this work.
title: title: Interesting approach to learning 6DoF grasping.
paper_summary: paper_summary: The paper develops an SE(3) equivariant grasp synthesis network that takes in point cloud input and generates full 6DoF grasps. The approach learns a mapping from each point in the point cloud to a grasp quality function on the sphere centering at the point, where the function is encoded via spherical harmonics. At inference time, the point cloud is separated into small regions of point clouds, and each is inference separately. The authors compare the results against another SE(3) invariant
main_review: main_review: Strength:
1. A novel equivariance approach to full 6DoF grasping, with a sufficient demonstration of the method on real robot environments in a tabletop setting. 
2. The reviewer thinks that representing point-level grasp quality function via spherical harmonics is an interesting idea. 
3. Sufficient demonstration of model performance and comparison with baseline methods. 

Weakness:
1. The proposed method needs to split a large point cloud into smaller sub-point clouds, which may not be as efficient compared to other methods that do not rely on such equivariance. 
2. All experiments are done in a tabletop setting, which in many cases does not necessarily justify the use of 6DoF grasping. A comparison with prior SE(2) methods [1] on grasp success and/or simulation of moving objects under more complex motion constraints (i.e. shelves) would further justify the improvement of the proposed method.

[1] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and R. Platt. Sample efficient grasp learning using equivariant models. arXiv preprint arXiv:2202.09468, 2022.
questions: questions: 1. What are some challenges you run into when performing sim2real transfer? How large is each $B_i$? Does increasing or decreasing the number of sub-point clouds change the performance? 
2. Given that grasps are sampled on the point cloud (object surface) instead of inside the object, will this parametrization result in more collision and object rotation when closing the gripper?
3. In the simulation, all the $B_i$ are generated via SAM: “In our experiments, we generated the center points using object masks for training and FPS for inference.” (line 155-156). Why isn’t SAM run in real?
limitations: limitations: 3
review_summary: review_summary: While there are some limitations, such as the need to split large point clouds and the focus on tabletop settings, these do not significantly reduce the paper's contributions. The work presents a new perspective on grasp synthesis and shows promise for learning better grasp representations. Thus I recommend accepting the paper.
title: title: Review
paper_summary: paper_summary: The work proposes an $SE(3)$-equivariant model predicting grasp qualities, given a point-cloud representing an object. The grasp quality is modelled as a function over the sphere, decomposed in spherical harmonics.
main_review: main_review: # Strengths
The paper is clearly written, especially regarding the mathematical formalism and notation.

The experimental analysis is comprehensive. It includes experiments in both simulated and real-world environments, with comparisons to several baselines. 

# Weaknesses  
I believe that a major weakness of the proposed approach is the constraint of sampling grasp poses where the y-axis of the hand (green axis in Figure 3) is parallel to the surface normal. This is restrictive, and not realistic for several objects. For example, how does that allow to grasp a cup by its handle? 

The above limitation is also reflected by a redundancy in the proposed model. Specifically, as explained in Section 4.6, the model outputs a grasp pose evaluation function $f_p: S^2 \rightarrow [0,1]$ for each $p$ in the point-cloud. However, because of the above constraint, only a one-dimensional submanifold of the sphere is relevant – specifically, a copy of $S^1$ in $S^2$. This makes the model’s output redundant. Why not designing a model that outputs a function defined directly on $S^1$ instead?

Another weakness is the fact that $f_p$ is not normalized, even though it represents a grasp quality. Since grasp quality is, intuitively, a distribution from which one wants to sample, I would expect some form of normalization. Even further, I think that the lack of normalization is reflected by an issue in the training loss. Specifically, as explained in Section 5 (Training Details), the model is trained via binary-cross entropy on each output of $f_p$ independently. In other words, the values $f_p(x) \in [0,1]$, $x \in S^2$, are interpreted as independent Bernoulli distributions that detect a grasp. But I believe this is a mathematically ill-posed optimization problem, and that some dependency as $x$ varies is necessary (for example, normalizing $f_p$ to integrate to $1$ over $S^2$). Since the model is trained only on a dataset of ‘positive’ grasps (i.e.,  for which ideally $f_p$ outputs $1$), why doesn’t the constant function $f_p(x) = 1$ minimize the loss?  



# Additional Comments
I believe that the paragraph in the background section on the representation theory of $SO(3)$ (lines 96-104) is redundant. This paragraph introduces plenty of mathematical notation – especially for Wigner matrices. I understand, as explained in the next paragraph, that these matrices are used to formalize the equivariance properties of spherical harmonics, but they are not used anywhere else in the paper. Therefore, I think the amount of formalism introduced is not justified by its use in the paper.
questions: questions: I would like the authors to address the questions above regarding 1) the constraint and redundancy in the model, and 2) the normalization issue, especially as related to the issue in the optimization problem.
limitations: limitations: 2
review_summary: review_summary: I believe that the approach, despite interesting and original, suffers from the technical issues explained above. Therefore, I lean towards rejecting it in the current state.