EquiGraspFlow: SE(3)-Equivariant 6-DoF Grasp Pose Generative Flows
Authors: Byeongdo Lim
Jongmin Kim
Jihwan Kim
Yonghyeon Lee
Frank C. Park
Venue: CoRL 2024 (2024)
Keywords: 6-DoF grasp pose generation
equivariance
generative models
continuous normalizing flows
TL;DR: We propose an SE(3)-equivariant grasp pose generative model by constructing a framework to learn SE(3)-invariant conditional distributions with Continuous Normalizing Flows

Abstract: Traditional methods for synthesizing 6-DoF grasp poses from 3D observations often rely on geometric heuristics, resulting in poor generalizability, limited grasp options, and higher failure rates. Recently, data-driven methods have been proposed that use generative models to learn the distribution of grasp poses and generate diverse candidate poses. The main drawback of these methods is that they fail to achieve SE(3)-equivariance, meaning that the generated grasp poses do not transform correctly with object rotations and translations. In this paper, we propose \textit{EquiGraspFlow}, a flow-based SE(3)-equivariant 6-DoF grasp pose generative model that can learn complex conditional distributions on the SE(3) manifold while guaranteeing SE(3)-equivariance. Our model achieves the equivariance without relying on data augmentation, by using network architectures that guarantee the equivariance by construction. Extensive experiments show that \textit{EquiGraspFlow} accurately learns grasp pose distribution, achieves the SE(3)-equivariance, and significantly outperforms existing grasp pose generative models. Code is available at https://github.com/bdlim99/EquiGraspFlow.

EquiGraspFlow: SB(3)-Equivariant 6-DoF Grasp Pose Generative Flows
Byeongdo Lim
Jongmin Kim
Jihwan Kim Yonghyeon Lee"
Frank C. Park
Scoul National University Korea Institute for Advanced Study (KIAS) SSIGGE [bdlim, jmkim, jihwankim]@rote @robotics......ooooooooooo snu. ac. kr ylee@kias. re. fcpēsnu kr
Abstract: Traditional methods for synthesizing 6-DoF grasp poses from 3D ob- servations often rely geometric heuristics, resulting in poor generalizability, limited grasp options, and higher failure rates. Recently, data-driven methods have been proposed that use generative models to learn the distribution of grasp poses and generate diverse candidate poses. The main drawback of these methods is that they fail to achieve SE(3)))quivar)
meaning that the generated grasp poses do not transform correctly with object rotations and translations. In this paper, we propose EquiGraspFlows flow-based SE(3)- 6-DoF grasp pose generative model that learn complex conditional distributions on the SE(3) manifold while guaranteeing SE(3)-cquivatiinnce Our model achieves the equiv- ariance without relying on data augmentation, by using network architectures that guarantee it by construction. Extensive experiments show that EquiGraspFlow accurately learns grasp pose distribution, achieves the -euuae-qqu, and significantly outperforms existing grasp pose generative models. Code avail- able https://github.c https:/ggitubb.co//llliii wowp//diiiinnnneeiaaraarl.....
Keywords: 6-DoF grasp pose generation, equivariance, generative models, con- tinuous normalizing flows
1 Introduction
Synthesizing six degrees freedom (DoF) grasp poses from 3D observations of an object (e.g., surface point cloud) is fundamental task in roboties. Approaches that produce limited number of grasp poses [1, 2, carry high risk of grasping failures, especially in con- strained environments. For example, obstacles or kine- matic constraints may prevent the robot from reaching certain candidate grasp poses; therefore, the more diverse candidates, the higher the probability grasp success.
Grasp pose generation
Transformation
Grasp pose generation
An example of SF(3)- equivariant grasp pose generation: the generated grasp poses identically trans- form with the object.
Figure 1:
Earlier approaches generate diverse grasp poses rely geometric heuristics to suggest multiple candidates based on an object's geometry [4, 5]. These methods typically of surface points with opposite surface normals as the contact points of the gripper's fingertips. However, their reliance on antipodal points limits the diversity of the grasp poses and makes the methods sensitive to observational noise in surface points and normals.
identify antipodal surface points pairs
Recently, data-driven approaches have gained significant attention. These methods first generate sufficiently diverse grasp poses through simulation [6], then use this data to learn the distribution of grasp poses and train grasp pose generative models [7, 8). The trained models not only generate di- verse poses but can also be made robust to observational noise by employing domain randomization.
"Equal contribution
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Mousavian et al. [7] employed the Variational Autoencoder (VAE) [9] as grasp pose generative model, while Urain et al. [8] more recently trained an energy-based model via the denoising score matching [10], demonstrating improved performance in learning complex grasp pose distribution.
However, the primary flaw with existing grasp pose generative models that they do not produce consistent grasp poses for rotated objects, leading to significant failure in some cases. An ideal model should generate grasp poses that transform identically for rotated and translated objects, as shown in Figure 1. Such models are considered SE(3) eqquivaria. Existing methods fail to ensure this equivariance, despite efforts to achieve it through data augmentation. Equivariant models, which have drawn significant attention from robot learning [II, 12], provide a promising solution to this issue, enabling robust and accurate grasp pose generation.
Our main contribution is EquiGraasPFow, where the equivariance is guaranteed by the network architectures, hence no data augmentation is required. Specifically, we adopt the Continuous Normalizing Flows (CNFs) framework [13, 14, 15), which, by utilizing time-dependent velocity fields as an infinite number of infinitesimally small transformations, learns complex distributions more effectively than diffusion models or discrete normalizing flows. We then formulate the necessary conditions to guarantee the -qquivaaaae of grasp pose generation for CNF on SE(3) conditioned on point cloud input.
6-DoF grasp pose generative model
Among these conditions, a non-trivial one requiring careful consideration is (B(33)cquivarian)) of the time-dependent velocity fields. Our method leverages the Vector Neurons (VNs) [16], which a popular choice in constructing SE(3) shown remarkable success, it allows only vector variables as inputs and cannot accommodate sealar variables such time. This limitation makes it challenging use VNs for constructing SE((3- equivariant time-dependent velocity fields. To address this, we introduce novel equivariant lifting layer that equips scalar variables with equivariant vector basis, enabling the effective use of VNs in constructing SE(3) -equivvrvva dieendeped velocity fields.
point cloud networks. Although VNs have
Through experiments conducted in both simulation and real-world environments, we have validated that our method surpasses existing 6-DoF grasp pose generative models [7, 8]. Notably, our method achieves superior results and consistent performance, even with changes in the object's rotation, without employing any augmentation strategies for equivariance. Real-world experiments validate that our method seamlessly applies to real-world applications.
2 Related Work
Synthesizing 6-DoF grasp poses is active research tackled with numerous methods [17, 18, 19]. Specifically, we focus on generative model-based approaches [7, 8, 20] and methods that guarantee equivariance [21, 22, 23]. which are of particular relevance to our work. Additionally, we present generative models that incorporate equivariance across various domains.
Generative models for grasping Mousavian et al. [7] develop 6-DOF GraspNet, a grasp pose generative model based on VAE [9], refining the generated grasp poses using an additional data- driven grasp evaluator. Urain al. [8] propose diffusion model [24] on the SE(3) manifold, termed Sifus-DDifssionFFieldd for grasp pose generation. Weng et al. [20] develop CAPGrasp, an R3 x (2(()))eeiivaa) grasp pose generative model under the assumption of amprrocch-constraiinn grasp. Despite these advancements, existing grasp pose generative models have not fully explored <33())=qquiaarianc;; they rely on augmentation strategies or achieve only partial equivariance under the assumption of the grasp. In contrast, our method fully ensures the (el())))quvvaiannce without relying on any assumptions or augmentation strategies.
Equivariance for grasping Zhu al. [21, 22] incorporate equivariquee for generating pla- nar grasp poses from top-down image observations, achieving enhanced sample efficiency. Huang et al. [23] construct Edge Grasp Network, SE(3) grasp quality function, resulting in improved grasp quality prediction performance for 6-DoF grasp poses. However, because these

---

methods do not utilize generative models, they either produce only limited set of grasp poses or rely on geometric heuristics that lack sufficient diversity. Unlike these methods, our method employs a generative model to produce a diverse set of grasp poses while ensuring full
Equivariant Generative Models Katsman et al. [25] introduce equivariant generative models on manifolds by applying equivariant manifold flows. Chen et al. [26] propose an (0()) conditional generative model for generating 2D trajectories in Fuclidean space, and Rozenberg and Freedman [27] propose an E(3)- and permutation-equuivrrinn conditional generative model for gen- erating 3D graphs Euclidean space. Additionally, Zwartsenberg et al. [28] develop permutation- cquivariant conditional generative model for generating traffic scenes and object bounding boxes in Euclidean space. These methods primarily focus learning distributions either unconditional or in Euclidean space, making them difficult to apply to SE(3)-equivarian 6-DoF grasp pose genera- tion, which requires conditional distributions on the SE(3) manifold. To address this limitation, we propose an SF((3) -qquivariat conditional generative model on the SF(3) manifold.
3 Preliminaries: Continuous Normalizing Flows on SE(3)
As preliminaries, we introduce the Continuous Normalizing Flows (CNFs) [13, 14, 15] tailored for SE(3) cases. We begin by defining key notations; we denote an element of SE(() by T = (R, where R E SO(3) and æ E R3 and refer to it as a pose or grasp pose. Given three-dimensional vector (a), 02, ag), [a] is 3 x 3 skew symmetric matrix with entries defined [a]12
-a3, [a]13 02 [a|23 time-dependent angular and linear velocity fields, : [0, 1] x SE(3) - R and v [0, 1] SE(3) R which map a time and a pose to angular and linear velocity vectors, respectively. We collectively define these two velocity fields as a time-dependent vector field w : [0, x SE(3) +
-1. By decomposing flows on SE(3) into SO(3) and R3, construct
Unconditional Continuous Normalizing Flows on SE(3) The CNF on SF(() models target distribution (T)) by transforming a prior distribution PT(T) using the time-dependent angular and linear velocity fields we and us where and é are trainable parameters. This transformation is guided by the following ordinary differential equations (ODEs)
R= [wo
: = uo(,7)
It generates a flow and a probability density path Pe(T") for time t E [0, 1]. Sampling from p, (T) involves sampling from the prior (o) and transforming these initial samples along the flow by solving the ODEs over & E [0. The parameters and á are trained to ensure that the transformed distribution P1 (T) matches the target distribution (T)).
Conditional Continuous Normalizing Flows on SE(3) To model target conditional distri- bution (T(cc) for some condition variable c, we use prior conditional distribution (T and velocity fields conditioned on e the ODEs become
R == (wo(t,c,TT]a e, T))R, ==0o(t,,,7)
which lead to a conditional probability density path Pe (T(c) for t E [0. 1]. In our setting, the condition variable e represents a point cloud an object and the target conditional distribution (TT(c) corresponds the distribution of successful grasp poses for that object.
4 EquiGraspFlow: SE(3)-Equivariant 6-DoF Grasp Pose Generative Flows
In this section, introduce EquiGraspFloww which generates diverse 6-DoF grasp poses from point cloud inputs using the conditional CNF approach while ensuring SE(3) (es()---qivariannc..... Denoting point cloud by P {Iv E R } EquiGraspFlow utilizes the time-dependent conditional velocity fields wt (t,, P. T), (t, P, P)T) and prior conditional distribution po (T)P) = (x)))))()(P) where
3

---

Generation process
Ground truth
(=D
00.25
0.5
0.75
To Po (T
To.s Po.s (TIP)
I (T(TP)
(TT)P)
Figure 2: Grasp pose generation process of EquiGraspFlow.
po (R) is uniform over SO(3) and po (P) is Gaussian R2 with its mean located the center of the point cloud. As illustrated in Figure 2, the model transforms po (T) into pı (T)P) through flow constructed from the veloc- ity fields, detailed in Figure 3.
Velocity Fields g. V
ODE solver
We use a dataset D == {(Ps, [Tig) , consisting of pairs of the point cloud P; of the i-th object and cor- responding set of successful grasp poses MM { We assume that each of grasp poses is sampled from the respective ground-truth conditional distribution P(T)P:). The neural velocity fields we and vy trained with 'D to ensure that the transformed conditional distribution pi (T)P:) closely approximate (T))))) We employ a Flow Matching framework [29, 30] to train the velocity fields and Guided Flows [31] to enhance the sample quality. Details are provided in Appendix B.2.
Figure 3: Construction of the flow from + At using the velocity fields wa. Ug and an ODE solver.
In the following section, we explain the incorporation of SE(3)=zquivariance into EquiGraspFlow. First, we will establish SF(3) iivaarinveee of conditional distributions, which is essential for SF(3)- equivariant grasp pose generation. We will then derive the necessary conditions to ensure that the transformed conditional distribution P1 (T) is -invaaaat...
4.1 SE(3) -333-nnnaarann Invariant Conditional Distributions
We first define transformations for point clouds, grasp poses, and diree-dimensiona vectors, Given an element T" (I²,,')) ) E SE(3), a point cloud p { is transformed T'P :- [R' +x)), grasp pose T (R,x) is transformed to T"T (R' R. R'x + x'), and dicensimenio vector is transformed to R'a. Next, we define (Ev(3)-inaariance of conditional distributions:
Definition 1. A distribution on SE(3) conditioned on a point cloud, denoted by p(T(P). is SE(3)- invariant if ((T'') = p(T(P) for any T" E SE(3).
This a formal description, in the language of probability distributions, that leads to SE((3- equivariant grasp pose generation. For a transformed point cloud, equivalently transformed grasp poses have the same conditional likelihood as before the transformation.
It is well-established mathematical fact that distributions transformed from an invariant prior via cquivariant maps remain invariant [25, 26, 27, 28, 32, 33J. However, existing works primarily focus on distributions either in Euclidean space, unconditional, or both, making them unsuitable for grasp pose generation, which requires conditional distributions on the SE(3) manifold. To address this, we extend these works model invariant conditional distributions on SE(3), thereby satisfying the required conditions. We thus define -equivreqiva of time-dependent conditional vector fields on SE(3):
Definition 2. A time-dependent vector field on SE(3) conditioned on a point cloud, denoted by u(t, P.T), is -Elu)--euvvariian if u(t, T'P, TT)) R' u(t, P,T) TT for any =(R', (R',x') E SE(3).
This geometric condition is essential for modeling invariant conditional distributions via CNFs on SE(3). Starting from an invariant prior conditional distribution, equivariant velocity fields preserve the invariance of transformed conditional distributions over time by the following proposition.
Proposition 1. Suppose a prior conditional distribution Po(T)P) is SE(3)- -invarian.. If the time- dependent angular and linear velocity fields w, SE(3)--quivaarian, then the transformed con-

---

Time t
1x1
Lifting layer*
1x3
Lifting layer*
Matrix multiplication
Concat
Time
1x1
1
1x3
NN290 -N
VN-MLP
VN-Linear
Point cloud P
Point cloud encoder
x
X 16
Grasp pose T
Velocity fields dàg. D
Velocities
Figure 4: Structure of the velocity fields wa, us and the lifting layer. The VN-DGCNN encodes the point cloud P into representation z consisting of 341 three-dimensional vectors. The lifting layer uses this representation along with the grasp pose to lift the time variable to a three- dimensional vector. Finally, the VN-MLP takes as input the concatenated list of the lifted time, representation, and grasp pose, and outputs the angular and linear velocities.
ditional distribution (T)P) any time O, defined via the flow of ODEs R [t),P, and à vtt, T), is (FF(3)-nvarrinn)
is trivial to show that our prior conditional distribution Po(T(P) SE(3)-invariant. In the subse- quent section, we propose neural network architecture that guarantees the remaining conditions in Proposition the -333vvvquva of the angular and linear velocity fields.
4.2 SE(3) Eqqiivarit Time- DeeeeDepnndett Conditional Velocity Field Networks
We separate the SEi3ieqquie into the equivariances on IR and SO(3). The RR==quivvriaace is achieved by subtracting the point mean = Σx xx/ K from P and x in T. The SO(3)- cquivariance achieved by adopting the Vector Neuron (VN) architectures [16], which are designed be -equivvaquvaaiant The structure the velocity fields is depicted in Figure
However, directly using the VN architectures is not straightforward since they require lists of three- dimensional vectors as input. Among the inputs of the time-dependent conditional velocity fields, the point cloud P is a set of three-dimensional vectors, and the pose T can be represented as a list of dinee-immesional vectors R2, R3, 2). where R3 is the i-th column vector of R. However, the time a scalar variable, making its incorporation into the VN architectures challenging. To use VNs with minimal modification, we propose an equivariant lifting layer that converts any scalar variables into three-dimensional equivariant vectors, so that they be concatenated to the list of vectors while maintaining the equivariance of the VN architectures.
Consider list of C scalar variables represented as a column vector s E R and list of C2 three- dimensional vectors represented as matrix V E RRxx3 - in our case, the time variable corresponds the case when Ci 1. We propose lifting layer, a mapping fin RRix x1 xIRC2x3 - RC1x3 and consider it to be -33uiqqqui it satisfies funn(s, VR = [in V)R" for any R E SO(3).
We construct the equivariant lifting layer as (un V) sfegui (V) where fequi C2 + R Ixa is any equivariant mapping, i.c., foqui (V R²)) = (V)).. This implics that fequi produces an equivariant vector from the input vectors V, and this equivariant vector is subsequently scaled by each of the C₁ scalar variables s, resulting in list of C, vectors. construction leads the -eeuuvaauieeee of fua. For fequi, we use the VN architecture [16]. Finally, with the proposed equivariant lifting layer, we can construct equivariant neural network for the time- dependent conditional velocity fields wo P, and (t, P.T).
trivial to show that this
5 Experiments
5.1 Experiment Settings
Baselines We compare our model with existing grasp pose generative models, 6-DOF GraspNet [7, 34] and Diffssiiitssooniillll [8]. To exclusively compare the generation performance of the generative models, we exclude the grasp evaluator used in 6-DOF GraspNet. SE(3)-DiffusionFilddd
5

---

Augmentation strategy
None
(0(3) -aug
Object category
Laptop
Mug
Bowl
Pencil
Laptop
Mug
Bowl
Pencil
6-DOF GraspNect [7]
0.7990
0.8285
1.0072
0,5216
0,4666
0,6683
0.6550
0.4025
PoiNt-SB(3)-Dit [8]
0.7184
0.7648
0.8775
0.5530
0.6054
0.6164
0.5047
0,5846
EquiGraspFllow (Ours)
0.3579
0.4804
0.3114
0.2988
0.3417
0.4828
0.3126
0.2872
(a) Average EMD (lower values are better)
Augmentation strategy
None
SO(3) aug
Object category
Laptop
Mug
Bowl
Pencil
Laptop
Mug
Bowl
Pencil
6-DOF GraspNet [7]
32.96
32.58
40.93
59.35
59.11
43.64
59.67
88.71
PoiNt-SFF3)--iff [8]
13.03
19.14
28.87
52.33
95.65
80.98
99.87
97.85
EquiGraspFlow (Ours)
99.72
90.56
100.00
98.98
99.90
92.26
100.00
99.71
(b) Average grasp success (%) (higher values better)
Table 1: EMD and grasp success rate for various objects in simulation experiments.
include two versions: SE(3)-Dif, which uses predefined object's shape codes, and Poi3t-SE(33- Dif, which encodes object's point cloud into a latent code. To compare with our model which also encodes object's point cloud, we choose PoiNt-SE(3)-Dif as a baseline. Since the baselines can incorporate F e'uivvivare by simply subtracting the point mean from the point cloud and adding the mean back the generated grasp poses, evaluate how effectively these models achieve -equvvvrrvaa through data augmentation.
Dataset We utilize a dataset obtained from the Laptop, Mug, Bowl, and Pencil categories of the ACRONYM dataset [6], comprising 175 laptops, 94 mugs, 38 bowls, and 82 pencils, each with poses configured for grasping by Franka Panda gripper. Our objective is to train a general model capable of grasping every type of object, train a single model. The objects are provided in mesh form; therefore, we uniformly sample 1024 points on the mesh surface to obtain a full point cloud. Experiments utilizing partial point clouds are provided in Appendix C.2. For the data augmentation the training dataset, we construct two strategies: None denotes no augmentation, and (3)))- denotes augmenting by random arbitrary rotation in SO(3). The validation and test datasets are augmented with evenly sampled rotations using Super-Fibonacci Spirals algorithm [35].
utilize unified dataset containing all object types to
Evaluation Metrics The evaluation metrics we utilize are Earth Mover's Distance (EMD) [36] and grasp success rate. The EMD measures the distance between the distributions of the generated and ground-truth grasp poses, defined by the minimum geodesic distance on the SE(3) manifold required to align the samples. The grasp success rate is assessed by determining whether the Franka Panda gripper successfully holds the object following the grasping action. Both metrics are first averaged across the rotations for each object, and then averaged across all objects.
5.2 Simulation Experiments
We conduct two types of experiments in the Nvidia Isaac Gym simulator [37]. The first experi- ment measures the average EMD and grasp success rate, while the second experiment assesses the consistency of the EMD and grasp success rate with changes in the object's rotation. In the first experiment, the test dataset is augmented with three rotations, while in the second experiment, it is augmented with ten rotations. To measure the grasp success rate, we generate 100 grasp poses in the first experiment and 25 grasp poses in the second experiment. To eliminate inconsistency arising from sampling the prior distribution in the second experiment, we fix initial samples in PoiNt-SE(33- Dif [8] and EquiGraspFlow, and then rotate them along with the object.
The average EMD and grasp success rate for various objects under different augmentation strategies are presented in Table 1. The existing methods do not account for SO(3) (euuiaarrvar,,,,,, leading

---

to insufficient performance with high EMD and low grasp success rate in the None set- ting. Conversely, EquiGraspFlow incorporates -quivaanuv,, improving performance in learning the grasp pose distribution and gen- erating graspable poses in this setting, evi- denced by lower EMD and higher grasp success rate. Even when arbitrary rotations augment the training dataset in the (3(3) (aug -aug setting, there exist discrepancies in the EMD and grasp suc- cess rate values between EquiGraspFlow and the baselines, underscoring our method's su- perior performance. Notably, unlike SE(3)- DiffusionFields, which requires additional data to learn the signed distance function, our model achieves superior performance without addi- tional data or modules. Additionally, by more accurately learning the grasp pose distribution, our model generates more diverse grasp poses than the baselines, depicted in Figure 6.
Grasp success rate (%)
200
0.6
80-
EMD
0.5
60
0.3-
40
Laptop Mug
Bowl Pencil
Laptop Mug
Bowl Pencil
6.DOF GraspNet
PaiNt SE(3) DIf
EquiGraspFlow
Figure 5: Average and standard deviation of EMD and grasp success rate for models trained with the (O(3) -3aug -ang setting.
G-DOF GraspNet
PoiNt-SE(3)-DDi
EquiGraspFlow
(0.4264, 48%)
10.6223, 72%)
(0.4422, 100%)
Figure and Figure 6 illustrate the consistency of performance as the object rotation varies for models trained with the (O(3)-aug setting. Figure presents the average EMD and grasp success rate, with standard deviations indicated by error bars. The standard deviations cal- culated with respect to the object rotation and are averaged across all objects. Figure 6 illus- trates the generated grasp poses alongside EMD and grasp success rate values for various object rotations, where both the object and the gener- ated grasp poses are inversely rotated align all scenes. As indicated by the bars in Figure 5 and the variance of values in Figure 6, our model exhibits more consistent perfor- mance compared to the baselines, attributed to No-
(0. (0.5064, 2880
10.7582, 20%)
(0.4497, 100%)
(0-4510, 48%)
10,8750, 88%)
(0.4432, 100%)
Figure 6: trained with the SO(3) setting for rotated ob- jects. Both the object and grasp poses are in- versely rotated to align all scenes. Green indi- cates successful grasps, while red indicates fail- ures. The EMD and grasp success rate for each scene are annotated below.
Grasp poses generated by models
its incorporation of (0(3)-equivarian tably, our model shows zero standard deviations in Figure 5 and maintains an identical value regardless of the object's rotation in Figure 6, demonstrating the perfect equivariance of our model.
5.3 Real-World World Experiments
In the real-world experiments, evaluate the grasp suc- cess rate of the baselines and our model trained with the SO(3) setting. Two mugs and two bowls as shown in Figure 7 are used. For the mugs, the handles are ran- domly oriented to test grasping, and both mugs are further rotated 90 degrees along the horizontal axis for additional testing, indicated (R) in Table 2. The bowls are placed on the desk and also rotated certain angle along the horizontal axis, also indicated as (R) in Table 2.
Figure 7: The mugs and bowls used in the real-world experiments along with the Franka Panda robot. The right side the image shows the mugs and bowls in rotated poses.
7

---

Mug (R)
Mug 2 (R)
Bowl 1 (R)
Bowl 2 (R)
Object
Mug
Mug 2
Bowl
Bowl 2
Total
6-DOF GraspNet [7]
10/ 10
4/ 10
10/ 10
5/ 10
10/ 10
2/ 10
10/ 10
3/ 10
54 80 (67.5%)
PoiNi-SE())-Dif [8]
10/ 9/
8/10
9/ 10
10/
10/ 10/10
10/
76 80 (95.0%)
EquiGraspFlow (Ours)
10/ 10 9/
10/10
9/ 10
10/
10/10
10/ 10
10/ 10
78/ / 80 (97.5%)
Table 2: Grasp success rates for various objects in real-world experiments. (R) indicates that objects are rotated poses.
After capturing RGB-D data of the object from multiple viewpoints, the object's partial point clouds are fused to create a full point cloud, which is subsequently downsampled to 1,024 points. This data is passed through each model to generate 100 grasp poses. Following this, grasp poses that are infeasible due to the constraints such as the manipulator's 's workspace, singularities, and collisions with the environments are excluded. From the remaining feasible grasp poses, ten grasp poses are randomly selected to measure the success rate.
Table 2 shows the results of real-world grasping. Our model achieves performance comparable to that in simulation, demonstrating its seamless application to real-world tasks without sim-to- real issues. Even when using noisy real-world data, our model successfully generates grasp poses. Additionally, our model outperforms the baselines, similar to the simulation results.
Conclusion
In this paper, introduce EquiGraspFlow, SE(3)-equivariant 6-DoF grasp pose generative model. Our approach revolves around two key ideas: (i) constructing framework for learning in- variant conditional distributions on the SE(3) manifold which is essential for equivariant grasp pose generation, and (ii) designing a novel equivariant lifting layer for our method. Unlike existing grasp pose generative models, our model ensures -equvvvrrqva in generating grasp poses, resulting in improved performance and consistency. From simulation experiments, we conduct quantitative evaluations against baselines, demonstrating the superior grasp pose generation performance of our model. Furthermore, the consistent performance demonstrated across varying object rotations ver- ifies the equivariance of the generated grasp poses and underscores the robustness of our method. Additionally, real-world experiments confirm the seamless applicability of our method to real-worke scenarios, underscoring its practical relevance and effectiveness in real-world applications.
Limitations and Future Works Our model, trained on full point clouds, may struggle to generate accurate grasp poses when robots encounter occlusions that prevent full observation of an object. To address this issue, we conducted additional experiments on generating grasp poses from partial observations of objects, as detailed Appendix C.2. The results of this experiment demonstrate that, even with partial point clouds, our method effectively generates grasp poses and outperforms the baselines. In future work, this model could be applied in real-world experiments generate grasp poses in constrained environments where objects are difficult to observe from multiple viewpoints.
In the real-world experiments, we randomly selected ten grasp poses that provided statistically sim- ilar results to the generated grasp poses. However, if the objective to identify the most promising grasp poses among the generated ones, one could leverage the probabilities of generated samples calculated from the CNFs or diffusion models. Selecting grasp poses with the highest probabilities could improve the success rate. This approach, however, has the drawback of primarily selecting grasp poses from regions with the highest probability, leading to reduction in diversity.
Additionally, our model, which generates grasp pose without considering obstacles, requires a collision- Celisiio-cheeking algorithm for real-world grasping. However, by utilizing dataset that includes grasp poses for target object in cluttered environments, we can develop collision-free grasp pose generative model without the need for additional modules. Incorporating equivariance into such model is a direction for future work.
8

---

Acknowledgments
This work was supported in part by IITP-MSIT grant RS-2021-1111068 (SNU AT Innovation Hub), (Training and Inference Methods for Goal Ori- ented AI Agents), MSIT(Ministry of Science, ICT), Korea, under the Global Research Support supervised by the ITPP[Intiiute for In- formation & Communications Technology Planning & Evaluation), KIAT grant P0020536 (HRD Program for Industrial Innovation), SRRC NRF grant 55--2033-0020803 SNU-AIIS, SNU-IPAI, SNU-IAMD, SNU BK21+ Program in Mechanical Engineering, SNU Institute for Engineering Re- search, and Microsoft Research Asia.
IITP-MSIT grant 2022-220480,
Program in the Digital Field (ogramm(RS-2024-004466666
Yonghyeon Lee was the beneficiary of an individual grant from CAINS supported by KIAS Indi- vidual Grant (AP092701) via the Center for Al and Natural Sciences at Korea Institute for Advanced Study.
References
[1] Y. Jiang, S. Moseson, and A. Saxena. Efficient grasping from rgbd images: Learning using new rectangle representation. In 201 IKEE International conference on robotics and automa- tion, pages 3304-3311. IEEE, 2011.
[2] I. Lenz, H. Lee, and A. Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Research, 34(4-5):705-) 77--4244 2015.
[3] G. Zhai, D. Huang, S.-C. Wu, H. Jung, Y. Di, F. Manhardt, F. Tombari, N. Navab, and
B. Busam. Monograspnet: 6-dof grasping with a single rgb image. In 2023 IEEE Interna- tional Conference Robotics and Automation (ICRA), pages 1708-1714. IEEE, 2023.
[4] I.-M. Chen and W. Burdick. Finding antipodal point grasps on irregularly shaped objects.
IEEE transactions on Robotics and Automation, 944:5507-512, 1993.
[5] G. Smith, F. Lee, K. Goldberg, K. Bohhinger, and Craig. Computing parallel-jaw grips. In Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No. 99CH36288C), volume pages 1897-1903. IEEE, 1999.
[6] C. Eppner, A. Mousavian, and D. Fox. Acronym: A large-scale grasp dataset based on sim- ulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1222-6227. IEEE, 2021.
[7] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2901-2910, 2019.
[8] Urain, N. Funk, Peters, and G. Chalvatzaki. dif3)ddiffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 5923-5930. IEEE, 2023.
[9] D. Kingma and M. Welling, Auto-encoding variational bayes. In The Second International Conference on Learning Representations, 2014.
[10] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[11] S. Kim, B. Lim, Y. Lee, and F. C. Park. Se (2)-equivariant pushing dynamics models for tabletop object manipulations. In Conference on Robot Learning, pages 427-436. PMLR, 2023.
[12] B. Lee, Y. Lee, S. Kim, M. Son. and F. C. Park. Equivariant motion manifold primitives. In 7th Annual Conference Robot Learning, 2023.

---

[13] R. T. Chen, Y. Rubanova, J. Bettencourt. and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31:6572-6588,, 2018.
[14] E. Mathieu and M. Nickel. Riemannian continuous normalizing flows. Advances in Neural Information Processing Systems, 33:2503-2515, 2020.
[15] L. Falorsi. Continuous normalizing flows on manifolds. arXiv preprint arXiv:2104.14959
2021.
[16] C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas. Vector neu- rons: A general framework for (3)eeqivar networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12200-12209, 2021.
[17] Mahler, F. Pokorny, B. Hou, M. Roderick, M. Laskey, M. Aubry, K. Kohlhoff, Kröger, Kuffner, and K. Goldberg. Dex-net 1.0: cloud-based network 3d objects for robust grasp planning using multi-armed bandit model with correlated rewards. In 2016 IEEE international conference robotics and automation (ICRA), pages 1957-1964. IERE, 2016.
[18] A. Ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research,
2017.
2017.
[19] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. Contact-graspnet Efficient 6-dof grasp generation in cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13438-13444. IEEE, 2021.
[20] Z. Weng, H. Lu, J. Lundell, and D. Kragic. Capgrasp: An R approach-constrained generative grasp sampler. IEEE Robotics and Automation Letters, 2024.
A x SQ(u)-equivvarian continuous
[21] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and R. Platt. Sample efficient grasp learning using equivariant models. In Robotics: Science and Systems, 2022.
[22] X. Zhu, D. Wang, G. Su, O. Biza, R. Walters, and R. Platt. On robot grasp learning using equivariant models. Autonomous Robots, 47(8):1175-113) 2023.
[23] H. Huang, D. Wang, X. Zhu, R. Walters, and R. Platt. Edge grasp network: A graph-based SE(3)-invarian) approach to grasp detection. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3882-3888, IEEE, 2023.
[24] Y. Song, Sohl-Dickstein, P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In The nineth International Conference Learning Representations, 2021.
[25] I. Katsman, A. Lou, D. Lim, Q. Jiang, S. N. Lim, and C. M. De Sa. Equivariant manifold flows. Advances in Neural Information Processing Systems, 34: 54:10600-10612, 2021.
[26] K. Chen, X. Chen, Z. Yu, M. Zhu, and H. Yang. Equidiff: A conditional equivariant diffusion model for trajectory prediction. In 2023 IEEE 26th International Conference Intelligent Transportation Systems (ITSC), pages 746-751, IEEE, 2023.
[27] E. Rozenberg and D. Freedman. Semi-equivariant conditional normalizing flows, with appli- cations to target-aware molecule generation. Machine Learning: Science and Technology. (3):035037, 2023.
[28] B. Zwartsenberg, A. Scibior, M. Niedoba, V. Lioutas, J. Sefas, Y. Liu, S. Dabiri, J. W. Lav- ington, T. Campbell, and F. Wood. Conditional permutation invariant flows. Transactions on Machine Learning Research, 2023.
[29] Y. Lipman, R. T. Chen, H. Ben-Hamu M. Nickel, and M. Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Representations, 2023.
10

---

[30] R. T. Chen and Y. Lipman. Riemannian flow matching on general geometries. arXiv preprint arXiv:2302.03660, 2023.
[31] Q. Zheng, M. Le, N. Shaul, Y. Lipman, A. Grover, and R. T. Chen. Guided flows for generative modeling and decision making, arXiv preprint arXiv:2311. 2023.
[32] Köhler, Klein, and F. Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In International conference on machine learning, pages 5361-5370. PMLR, 2020.
[33] H. Lu, S. Szabados, and Y. Yu. Structure preserving diffusion models. arXiv preprint arXiv:2402. 19369, 2024.
[34] J. Lundell. 6-dof graspnet pytorch. URL 6dof- sof--ge grof-ggaapnet.
[35] M. Alexa. Super-fibonacci spirals: Fast, low-discrepancy sampling of SO(3). In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8291-8300 2022.
[36] A. Tanaka. Discriminator optimal transport. Advances in Neural Information Processing Sys- tems, 32, 2019.
[37] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simu- lation for robot learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[38] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint 1iXiv:2207.1259. 2022.
[39] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In The Thrid Interna- tional Conference on Learning Representations, 2015.
[40] H. Munthe-Kase Runge-kutta methods on lie groups. BIT Numerical Mathematics, 38:92-
111, 1998.
[41] Guerrero. Language segment-anything. lang-sam.
URL httpg///oooog/////ooo/oo/oooooor com/paulguerrerr/
[42] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, I.. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643, 2023.
[43] S. Liu, 7. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. L. Yang, H. Su, J. Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
[44] R. Swinbank and James Purser. Fibonacci grids: novel approach to global modelling. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography, 1326619):1799-1793 2006.

---

Appendix
A Proof of Proposition
In this section, we restate and prove Proposition Section 4.1.
Proposition 1. Suppose a prior conditional distribution po (T(P) is SE(3) -i33-iiaaaaant If the time- dependent angular and linear velocity fields w. are SE(3) -equivarrv,, then the transformed con- ditional distribution pe PT) at any time Z 0, defined via the flow of ODEs R [wt,,, R and & ut, P, ,). is SE(3) -ivvaraan..
To this end, we first introduce the concept of conditional flow derived from the time-dependent conditional velocity fields and define -quiivaaaae of the conditional flow. Subsequently, üime-dependent conditional velocity fields induce an SE(3)-
we demonstrate that equivariant conditional flow. Finally, we prove Proposition by establishing that, starting from an -Enaa)-nnariant prior conditional distribution, an -equivrauuaaaiant conditional flow preserves the invariance over time.
A.1 (((3)))quivaian)) Conditional Flows
Consider trajectory on the SE(3) manifold, starting from an initial pose T SE(3) and guided by the time-dependent angular and linear velocity fields, w and v, conditioned on point cloud P. This trajectory called an integral curve for w and conditioned P and starting at T, and denoted by y IR- SE(3). By decomposing the S3(3) and R components such that 7t) (yu((), 7/ (t)), the integral curve is defined via the following ordinary differential equations (ODEs):
nR (t) [wt (t)) (t) = v(t, P. (t)(,), 7(0)
(3)
conditional flow of the velocity fields w and v conditioned on P defined as a mapping P(,,P,T) SEE(), where 1. IR and SE(3) represent time and the initial pose, respec- tively. Here, f(t, PT) (() where y the integral curve for and conditioned and starting at T.
Now, we define the (F((3)eeuivariaave) of conditional flow as follows:
Definition 3. A flow SE(3) conditioned point cloud, denoted by f(t,P,T) is SE(3)- equivariant if ft, T T'T) = f(, P) for any T" E SE(3).
Next, We demonstrate that -3uuiiqquiaa dime-dependent conditional velocity fields induce the SE(3) equiiavaquie of their conditional flow through the following Proposition:
Proposition 2. For any time-dependent conditional angular and linear velocity fields w and v, their conditional flow f is -equivrraan if as and are -ele))))qqvariantt
Proof. Consider arbitrary point cloud P and fix initial pose T as arbitrary element in SE(3). Then, ftt, P)T) (t) = (TR(t), Y/ (t)) represents the integral curve for w, conditioned on P and starting at T. The ODEs governing this integral curve are given by the same equations as (3).
=())= = (fn(t), (1)) where ÿ the integral curve for w, conditioned on 7"P and starting T'.. The ODEs for this integral curve are given by:
For any T' x²' E SE(3), (t. T'P, TTT)
(n(() (w, T'P, (())) (t) wtt, (t,,TTP 7(0) (0)=T'T TT.
(4)
Now, consider an integral curve & defined as {()) = (1x 9/ (0) = (R'ra((), (1) + x')
(Tn(()), Th (1)) = ('x(1) T',,,, This integral curve results from transforming the inte- gral curve (t) by TV.
To prove the -euuivaauie of the conditional flow, need to show that and are the same integral curve. Specificallyy need to show that 7(t) f(t, = T' f(t P. (t).
12

---

Noting that R[a]R* = [Ra] for any R E SO(3) and a E I.' we analyze (t) as follows:
(t))) = + (R'rR(t)
dt
R''iR(t)
R' (t)(t)aaa(t
(5)
-I''(LI,P: (R)))
Similarly, for (t), we have:
(1) (1'o00))+ +1))+
dt
-l'Fia11)
R'vlt,
(6)
=u(t,T'P,T''
= =((tTTp,,(())
Finally, note that 7(0) = T')(0) = T'. Thus, ((t) and (t) satisfy the same ODEs, and the uniqueness of the solution of the ODE ensures that & and are the same integral curve. Con- sequently, we have TTTTPTTT T' f(t P,T) T) for any T' SE(3), demonstrating that f is SE(3) -equivvaaiva
A.2
Conditional Distributions
To demonstrate that an SE33)-equivariant conditional flow preserves the invariance of an SE(3)- invariant prior conditional distribution, we present the following proposition.
Proposition 3. Suppose prior conditional distribution PP(T)P) is SF(3)-invariant. If condi- tional flow is -quivvaaan, then the transformed conditional distribution Pt (TT(P) at any time 2 0 defined via the flow is SE(3)-invarians
Proof. To prove this proposition, we first extend Theorem 3 from [25], which involves a general Riemannian manifold and general group, conditional version.
Consider a Riemannian manifold (M, h) with group G. Denote the action an element E G on .M by the map Lg : : + M. The map Lg is isometric if, for any tangent vectors u and u at any point x M, the following condition holds:
huu, v), where d(Lg)z represents the differential of Lg z. If Ly is isometric, then det JL, (z) 1 for any M, where Jz, (z) denotes the Jacobian matrix of the map Ly evaluated at and expressed in local coordinates.
Let e denote a condition variable. The conditional flow at time f is represented by the map ft,e M + M. This flow transforms prior conditional distribution po (x]c) into the condi- tional distribution (x]c). The likelihood of the transformed conditional distribution is given by the following change of variables formula:
Pe(xlc) po (f (z))c) J
Assuming the action of G on e is well-defined and denoted by conditional distribution p(zlc) is G-invariant if p((1 p(L. (x) c) = p(r|c) for any g E G. The conditional flow St, is
G-equivariant if, et.gee(L. (L. = (z) for any E G, ie., ft.e o L.g = -Tg St, and
1 fije = fie L
Assuming that the map Lg is isometric for any g G. Riemannian manifold M and an isometric group G as follows:
prove Proposition 3 in general
13

---

Pi(Lg(a))ogc))
po
Po (Leg c) det 1igi (Lo())))
det J (Lg(z))
(invariant prior)
((Lg 1 fige o Lg) (z)]c)
det J1,- (ffege Lo) (x))
det J 15g (L,(x))
det JL, (x))
(PLg ttgge o Lg) (x)]c)
(8)
det Jt,-1 (Veige Lu) (x)) J 1. (I
(multiplicativity)
Po (( Stire Lg) (x) det 1. (z)
(chain rule)
Po ((1gg lige o Lg) (z) ()) det 1 obg (2))
(Lg-1= = Lg''
Po (ffed o L Lg) (z)
det I sn :otoo,tooo" tobo (x))
(equivariant flow)
po (Z) J/ (x)
= (x]cc)
Proposition 3 is a special case where .M- SE(3), SE(3), P, and the group action of T" E SE(3) on E SE(3), denote by LT (T) = TT, is the left translation map which is isometric. Hence, Proposition proved.
We now prove Proposition by utilizing Proposition 2 and Proposition 3.
Proof of Proposition 1. Since the angular and linear velocity fields w and -E33)-equivariant, it follows from Proposition 2 that their flow f is (F3)e-aauivariant. Thus, by Proposition 3, the conditional distribution pe (T))), defined via the flow of the velocity fields, is -ivvariint. O
B
Implementation Details
B.1 Details for Networks
To model the time-dependent conditional velocity fields (t P,T) and (t, with SO(3)- equivariance, we employ Vector Neuron (VN) architectures [16], which are specifically designed for -quuiieeuvv The structure of these velocity fields is illustrated in Figure
To encode the point cloud P, we utilize the backbone of the VN-DGCNN designed for classifica- tion tasks, up to the invariant layer, excluding batch normalization layers, and adding a 170-size EdgeConv module at the sixth module position. mean pooling layer is then applied to pool the point dimension, extracting a representation 2 of 341 dimensiimali vectors. The grasp pose T reconfigured by concatenating the three rotation column vectors and the translation vector, resulting in four three-dimensional vectors. Time / converted into three-dimensional vector through the lifting layer. The VN-MLP then concatenates these list of three- dinee-dimessional vectors input and outputs the angular and linear velocities. The VN-MLP consists of five hidden VN-Linear layers, each followed by VN-LeakyReLU activation with a negative slope 0.2, and one output VN-Linear layer. The sizes of the hidden layers are (256, 256, 128, 128, 128), and the output layer size is 2, corresponding to two three-dimensional velocity vectors.
The lifting layer the representation and the grasp pose T to convert the scalar time into threc-dimensional vector. This process involves VN-Linear layer with size of 1, producing an
14

---

Time t
1x1
Lifting layer*
1x3
Lifting layer*
Matrix multiplication
Concat
Time
1x1
1x3
1x3
NN290 -N
VN-MPP
VN-Linear
Point cloud P
Point cloud encoder
x
X 16
Grasp pose?
Velocity fields d2g. V
Velocities
Figure 8: Structure of the velocity fields and the lifting layer. The VN-DGCNN encodes the point
cloud P into representation z consisting of 341 three-dimensional vectors. The VN-Linear layer the lifting layer uses this representation along with the grasp pose T, to produce matrix of size 1 x 3. This matrix lifts the time variable f (of size x 1) to sarre-dimensionall vector. Finally, the VN-MLP in the velocity fields takes as input the concatenated list of the lifted time, representation, and grasp pose, and outputs the angular and linear velocities.
output matrix of size 1 3. This output matrix is then multiplied to the time variable / (size 1 x 1), resulting in single three-dimensional vector.
B.2 Details for Training and Sampling
Dataset We use a dataset comprising 175 laptops, 94 mugs, 38 bowls, and 82 pencils obtained from the ACRONYM dataset [6]. The number of objects per each category and each dataset split are detailed in Ta- ble 3. For each category laptop, mug, bowl, and pencil average of 1,643, 1,124, 1,747, and 1,737 6-DoF grasp poses are provided, respectively.
Laptop
Mug
Bowl
Pencil
training validation
105
54
21
50
35
18
7
16
test
22
16
Table 3: Number of objects per each ob-
ject category and each dataset split.
Flow Matching We employ the Flow Matching (FM) framework [29, 30] to train our CNF model. The core element of FM involves designing the per-sample target vector field uy (T(T;) and the corresponding probability path (T)T),, where Ti = (R1 represents particular sample from the target distribution (PT)P. our approach, we separate the rotation and translation components in ut (T(T;) (wo (R) (Ir(z1)). We then define the target angular and linear velocity fields wi R2) and uz (arla)) as follows:
log( (2TTR, Rt)
(R)))
(a((a)) $1
(9)
1-t
Consequently, the training objective for EquiGraspFlow is designed as
L -
2 (t. P.T)) log( RdT R2)
(s
31
(10)
where || ||r denotes the Frobenius norm, and T (R, x) and 7i (R1,21).
One thing to note is that it might seem natural to design the vector field on the SE(3) manifold instead of separating the rotation and translation components, similarly how we design the angular velocity field the S3(3) manifold shown in (9). However, this approach results in screw motion-shaped paths of grasp poses, where the translation may not follow straight line toward the target grasp pose. In the context of our grasp pose generation task, separating the rotation and translation and ensuring that the translation motion directly heads toward the target grasp pose is more intuitive and appropriate vector field formulation.
Guided Flows Guided Flows [31] a technique that enhances the sample quality and efficiency of conditional generative models by integrating classifier-free guidance [38] into Flow Matching framework. This method employs guided velocity fields during sampling, defined as a weighted sum of unconditional and conditional velocity fields. Using empty set as a null condition for
15

---

the point cloud imput, we define the guided angular and linear velocity fields Wy and üg as follows, utilizing the weight parameter 13:
wo P. + (wo(t, (t..P.
(1I)
(t (1-
When is imput, the point cloud encoder outputs list of zero vectors as The guided velocity fields remain -euui--quivaariat,, as they are a linear combination of unconditional and conditional velocity fields, both of which SE(3) -E33i-qquiaaii To train the unconditional velocity fields, randomly replace P with the empty set Ø with probability of 20% during training. For sampling, we use 3 = 2 to evaluate average performance and 3 2.5 to assess the consistency of performance.
4 is utilized to train the baselines and our model. L2 regularization with hyperparameter 1 x 10 employed for training EquiGraspFlow.
Optimizer Adam optimizer [39] with learning rate x 10-4
Sampling The fourth-order Runge- Kutta MK method on Lic groups [40] is utilized as the ODE solver, with 20 steps employed.
B.3 Details for Grasping Experiments
Grasping in Simulation In the simulation experiments, the gripper and object float without any obstacles. After grasping the object, the gripper is shaken to robustly assess the success of the grasp.
Grasping in Real-World For the real-world experiments, a Franka Panda gripper is equipped with 7-DoF Franka Emika Panda robot, and the object is placed on table. The object's point cloud is obtained using an Intel RealSense Depth Camera D435 mounted on the gripper. The depth camera captures an RGB-D image of the scene, and the object is segmented using Language Segment- Anything [41], which built on the Segment Anything Model [42] and Grounding DINO [43], resulting in a partial point cloud of the object. By combining partial point clouds from multiple viewpoints, we reconstruct the full point cloud of the object. We then downsample the full point cloud to uniformly downsampled point cloud with 1,024 points using voxel downsampling.
The robot motion for grasping an object in real-world experiments is designed as follows. To prevent collisions with the object during the movement of the gripper toward the generated grasp pose, we first move the gripper to a pre-grasp pose. This pre-grasp pose is offset from the grasp pose by small distance the -2 direction the gripper's ' frame (the 2-axis of the gripper's frame represents the direction of gripper's palm). Next, we move the gripper the grasp pose and execute the grasp. Once the object is grasped, the gripper is lifted by small distance. The success of the grasp is manually determined based on whether the object is held securely by the gripper. After each grasping attempt, we manually reset the position and orientation of the object to its initial state.
C Additional Experiments and Results
C.1 Sampling Time
We conduct an experiment to measure the sampling time of the baselines and EquiGraspFlow. Each model gener- ates 100 grasp poses from point cloud with 1024 points. Sampling is conducted on single NVIDIA GeForce RTX 3090 GPU, and results are averaged over 100 experiments.
Sampling time (ms)
Model
6-DOF GraspNet [7]
41.08
PoiN [8]
1666.56
EquiGraspF (Ours) 256.76
Table shows the average sampling time of each model. While 6-DOF GraspNet [7] about six times faster than EquiGraspFlow, its grasping performance is less-than- desirable. On the other hand, EquiGraspFlow is not only
Table 4: Sampling times for generat- ing 100 grasp poses from point cloud consisting of 1024 points.
about six times faster but also shows better grasping performance than PoiNt-SE(3)-Diit [8].
16

---

Augmentation strategy
None
(0(3) -ang
Object category
Laptop
Mug
Bowl
Pencil
Laptop
Mug
Bowl
Pencil
6-DOF GraspNect [7]
0.8778
0.8726
1.0766
0.7241
0.6012
0.7628
0.8724
0.4233
PoiNt-SB(3)-Dit [8]
0.7875
0.8892
1.0407
0.5957
0.5851
0.7186
0.4733
0.4498
EquiGraspFllow (Ours)
0.3564
0.4801
0.3204
0.2879
0.3032
0.4086
0.2675
0.2648
(a) Average EMD (lower values better)
Augmentation strategy
None
S0(3) aug
Object category
Laptop
Mug
Bowl
Pencil
Laptop
Mug
Bowl
Pencil
6-DOF GraspNet [7]
7.31
12.16
5.42
17.47
17.18
24.99
18.89
60.97
PoiNt-SF(3y-Dif [8]
14.50
20.69
12.84
34.86
87.75
50.12
95.82
97.44
EquiGraspFlow (Ours)
95.86
84.91
100.00
99.64
96.11
84.10
99.87
99.53
(b) Average grasp success (%) (higher values better)
Table 5: EMD and grasp success rate of the grasp poses generated from partial point cloud.
C.2 Grasp Pose Generation from Partial Point Cloud
To evaluate the performance of grasping in scenarios where objects cannot be fully observed from all viewpoints, we train and test the baselines and EquiGraspFlow when partial point clouds are input. Instead of uniformly sampling points from the mesh surface, we obtain a partial point cloud by sampling 512 points only from the mesh surface visible from viewpoint defined by 3D unit vector originating from the object's s center. All grasp poses in the dataset are utilized regardless of the viewpoint, requiring the model to generate grasp poses even for unobserved parts of the object.
The data augmentation for object rotation follows the same approach as the full point cloud exper- iments, utilizing either the None or (O33)aau strategy in the training dataset, while the validation and test datasets utilize evenly sampled rotations using Super-Fibonacci Spirals algorithm [35]. The viewpoint vector on S is randomly sampled in the training dataset, and is evenly sampled using Fibonacci lattice algorithm [44] in the validation and test datasets. Fach object in the validation and test datasets is augmented with three rotations, and each rotated object is further augmented with three viewpoints. The grasp success rate is measured in Nvidia Isaac Gym simulator [37] using 25 generated grasp poses.
The average Earth Mover's Distance (EMD) and grasp success rate under different augmentation strategies are presented in Table 5. Despite slight performance degradation from observing only portions of objects, our method still demonstrates strong performance, indicating its applicability in scenarios where objects are partially observed. Additionally, similar to the full point cloud exper- iments, EquiGraspFlow achieves lower EMD and higher grasp success rate compared to the base- lines both the None and SO(3)-aug settings. This demonstrates superior performance learning grasp pose distribution and generating graspable poses even when objects are partially observed. Especially, EquiGraspFlow demonstrates high grasp success rate for mugs, which have com- plex shapes and are challenging for generating graspable poses under partial observation. This is attributed to the enhanced data efficiency resulting from the incorporation of (3u--equuva which enables the model to better learn how to grasp mugs from partial observation compared to the baselines.
C,3
Grasp Pose Generation from Full Point Cloud
C.3.1 Consistency with Respect to the Number of Rotations
We conduct additional experiments investigate how performance consistency varies with the num- ber of rotations. Using the same setup in the consistency experiments described in Section 5.2, augment the test dataset with 2, 4, 6, and 8 rotations, respectively.
17

---

Laptop
Mug
Bowl
Pencil
EMD
Grasp success
100
100
100
100
rate (%)
80
80
60
60
60
60
40
40
40
40
6
Number of rotations
Number of rotations
Number of rotations
Number of rotations
6-DOF GraspNet
PoiNt-SE(3)-DIF
EquiGraspFlow
Figure 9: Average and standard deviation of EMD (lower values are better) and grasp success rate (higher values are better) with respect to the number of rotations. The standard deviation is calcu- lated with respect the object's rotation and then averaged across all objects.
Figure 9 presents an error bar plot illustrating the changes in the average and standard deviation of EMD and grasp success rate across different numbers of rotations for each object category. As indicated by the increasing size of the error bars, the baselines exhibit less consistency in grasp pose distribution and grasping performance as the number of rotations increases. This suggests that the more diverse the rotations, the less consistent the baselines are in generating grasp poses. In contrast, EquiGraspFlow exhibits zero standard deviations EMD values across all number of rotations, indicating that our model generates a consistent grasp pose distribution. This consistency results from the -quuviqquivariaanc,, which ensures the generation equivariant grasp poses relative the object's rotation. Although there are slight variations in the grasp success rate for EquiGraspFlow in some cases, these attributed to the simulator's 's non-equivariance. As these deviations remain close to zero across all rotation counts, they further underscore the robustness of our model.
C.3.2 Additional Visualizations
Figures 10 to 13 present additional visualizations of the generated grasp poses. These figures show the generated grasp poses of a laptop, mug, bowl and pencil across ten object rotations, along with the EMD and grasp success rate values. The object's point cloud is rotated and input into each model; however, in these figures, both the objects and the generated grasp poses inversely rotated to align all scenes. Successful and failed grasp poses are indicated in green and red, respectively.
For all object types, the grasp poses generated by EquiGraspFlow are widely distributed across vari- ous parts of the objects, demonstrating that our model generates more diverse grasp poses compared the baselines. For example, while PoiNt-SE(3)-Dir tends to concentrate generated grasp poses on specific parts of the laptop and bowl for certain object rotations, EquiGraspFlow distributes the grasp poses evenly across the objects regardless of their rotation. Additionally, (EINt-SE(3)-DDD) -Dff struggles to generate grasp poses that target the handle of the mug, whereas EquiGraspFlow generates grasp poses that are evenly distributed between the handle and the body of the mug.
Furthermore, the variance in the EMD and grasp success rate values indicates that EquiGraspFlow exhibits more consistent results across different object rotations. While the baselines show vari- ability in grasp pose distribution and grasping performance depending on the object's rotation, model maintains identical values across the ten object rotations, demonstrating the perfect equivari- ance of our approach.
18

---

(D 68%)
IDEGAA, 7250)
(0.4187 1009))
(6477), 728)
18.995,, 100%)
t0 30.4996, 100%)
10.7871 B4R9
10.1107 1009)
10,20R 60%)
(0 054471.997 92%)
(0 40.098 1009
jo 125221 9898
1D 10-086 100%)
5c200 1995
10.7715 78800
(0.4187, 100%))
Figure 10: Generated grasp poses for a laptop acTOss ten rotations.
Figure Generated grasp poses for mug across ten rotations.
(0. 70.7728, 100%
(0. 40.4 100%
10 8055, 48%
10.7382. Bong
00.1077 1009)
10 4890
10. 00,9999, 50%)
(G.095, 100%)
aesee 9996)
(0.4187, 100%)
(0 A2000 0899
(0.5041, 929
10,409, 109%
D.ATES S99A
00.0002 Bo9g
00.1077 1009)
10 4800, 8944
10.044,, 100%)
10,4095 100%)
Jo.E40, 58%)
00.8770. 50%)
10.4107, 100%)
e0 3483, spn)
(0-080, 100%
10.6112 80%)
10 105477 80%
00.4155 08.1 100%)
5858,, B4R
(0,095, 100%
D.S8SA 60%)
10.417r, 100%))
10-0790, S2N)
No,500, 100%)
1o 004095, 100%)
(04755, 89%)
ta 4880, 7290
(041877 100%))
6.D GraspNet
PaiNt-SE(3)-Dii
EquiGraspFlow
6-DOF G-DOF GraspNet
PoiNt-SE(3)-Dir
EquiGrespFlow
19

---

1oss 1009
3522 (00%)
10 1005 W0Y
(0.3621, 100%)
TD CESOO
(0 AOST 1000
(0.5621, 100%)
Figure 13: Generated grasp poses for pencil across ten rotations.
Vocco
BLC
EED
100000 (2AN
100%)
0.3821,
BOOO 7t
207012 SENU
100%)
10 303621
ppooss
0039000 W990
6377 6099
(0,382), (005))
Sopee 20 To CeEaa
(0 0018, (80K)
R0044 100% (06414, 100%)
(00%)
Figue 12: Cenneraee
(0.8621,
10,3810, (e%)
100%)
3.3621,
(0, 9986 Wabi
(0.4025 100%
100%)
0.3621,
(03770, 100%))
3.88, 100%)
6-DOF GraspNet
PoiNt PoiNt-SE(3)-Dii Dif
EquiGraspFlow
GraspNet
Pei))-SE(3)-Dif
EquiGraspFlow
20


Reviews:
[{"review_id": "vWAeMDY1qp", "reviewer": "Reviewer_jkBE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the authors present a new grasp generative model. Compared to prior works, their model is SE(3) equivariant by definition, which results in improved performance. This is achieved on a methodological level through an equivariant encoder (VN) and a flow-based generative model.", "summary_of_recommendation": "The paper is very well written and has substantial contributions, which are nicely showcased in the experiments. Moreover, the experimental results are convincing and fully support the claims made in the paper.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- What is the initial distribution from which you start sampling the grasp poses?\n- In the real-world experiment, would SE(3)-Dif benefit and slightly reduce the number of failures when selecting the 10 grasp poses according to the energy?\n- What procedure would the authors propose in a real-world scenario when they can only select a single grasp from the generated ones? I.e., what alternative strategies could be implemented with the presented approach apart from sampling at random?\n- How sensitive is the method w.r.t. the pointcloud being complete?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aigQjQnvPq", "reviewer": "Reviewer_Qvx1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors devise a SE(3)-Equivariant continuous normalizing flow model for generative 6-DoF Grasp Synthesis. Such equivariance can help the model generalize well with less data-preprocessing such as data augmentation. To this end, the authors  introduced necessary conditions for constructing such flows and a lifting layer in the conditional velocity field networks. The superior performance is validated in both simulated and real-world experiments for two object categories.", "summary_of_recommendation": "For the technically sound appraoch and convincing results, I vote for weak accept while looking forward to the manuscript with the points above addressed.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See the list of weaknesses above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JRQWhrandV", "reviewer": "Reviewer_2vdQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The work proposes a flow-based $SE(3)$-equivariant generative model on $SE(3)$ that is conditioned on a point-cloud in $\\mathbb{R}^3$. The method is applied to grasp pose generation.", "summary_of_recommendation": "I think that the work combines equivariant deep learning with flow-based generative modelling in a clever way for the purpose of grasp pose sampling. Besides the weakness above – which has a theoretical nature – the work deserves acceptance, in my opinion, for its methodological and applied contributions.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I have no questions for the rebuttal.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: PRE REBUTTAL:

High level summary of reviews:

Strengths:

- All reviewers state that the paper is clear, well-written, and well-organized.
- Multiple viewers suggest that the paper is technically sound and presents convincing methodological contributions.
- Multiple reviewers highlight the novelty of the approach (the idea of incorporating invariance in the generated distribution).
- The experimental section is thorough with real-world experiments. Reviewer jkBE highlights that experimental results are convincing and fully support the claims made in the paper.

Weaknesses:

- It lacks discussion of discrete normalizing flows and evidence on more object categories.
- The performance gain on the real-robot experiment is limited.
- Reviewer 2vdQ suggests that the main theoretical result (Proposition 1) is a basic and well-known mathematical fact. This should be clarified in updated versions of the manuscript.
- Reviewers Qvx1 and jkBE request the authors clarify a few details of the paper. Please see their extensive comments for details (e.g. run-time analysis, performance on more object categories, etc).

POST REBUTTAL:

During the rebuttal the authors clarified a number of points made by the reviewers and added additional experiments. This is clearly a strong submission and is stronger following the rebuttal.
Decision: Accept