EquiGraspFlow: SE(3)-Equivariant 6-DoF Grasp Pose Generative Flows

Authors: byeongdo  lim
jongmin  kim
jihwan  kim
yonghyeon  lee
frank c park
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 09:34:27

Abstract: Traditional methods for synthesizing 6-DoF grasp poses from 3D observations often rely on geometric heuristics, resulting in poor generalizability, limited grasp options, and higher failure rates. Recently, data-driven methods have been proposed that use generative models to learn the distribution of grasp poses and generate diverse candidate poses. The main drawback of these methods is that they fail to achieve SE(3)-equivariance, meaning that the generated grasp poses do not transform correctly with object rotations and translations. In this paper, we propose \textit{EquiGraspFlow}, a flow-based SE(3)-equivariant 6-DoF grasp pose generative model that can learn complex conditional distributions on the SE(3) manifold while guaranteeing SE(3)-equivariance. Our model achieves the equivariance without relying on data augmentation, by using network architectures that guarantee the equivariance by construction. Extensive experiments show that \textit{EquiGraspFlow} accurately learns grasp pose distribution, achieves the SE(3)-equivariance, and significantly outperforms existing grasp pose generative models. Code is available at https://github.com/bdlim99/EquiGraspFlow.

Main paper: 
 related work: 
 Synthesizing 6-DoF grasp poses is an active research area tackled with numerous methods [17, 18, 19] . Specifically, we focus on generative model-based approaches [7, 8, 20] and methods that guarantee equivariance [21, 22, 23] , which are of particular relevance to our work. Additionally, we present generative models that incorporate equivariance across various domains.Generative models for grasping Mousavian et al. [7] develop 6-DOF GraspNet, a grasp pose generative model based on VAE [9] , refining the generated grasp poses using an additional datadriven grasp evaluator. Urain et al. [8] propose a diffusion model [24] on the SE(3) manifold, termed SE(3)-DiffusionFields, for grasp pose generation. Weng et al. [20] develop CAPGrasp, an R 3 × SO(2)-equivariant grasp pose generative model under the assumption of approach-constrained grasp. Despite these advancements, existing grasp pose generative models have not fully explored SE(3)-equivariance; they rely on augmentation strategies or achieve only partial equivariance under the assumption of the grasp. In contrast, our method fully ensures the SE(3)-equivariance without relying on any assumptions or augmentation strategies.Equivariance for grasping Zhu et al. [21, 22] incorporate SE(2)-equivariance for generating planar grasp poses from top-down image observations, achieving enhanced sample efficiency. Huang et al. [23] construct Edge Grasp Network, an SE(3)-invariant grasp quality function, resulting in improved grasp quality prediction performance for 6-DoF grasp poses. However, because these methods do not utilize generative models, they either produce only a limited set of grasp poses or rely on geometric heuristics that lack sufficient diversity. Unlike these methods, our method employs a generative model to produce a diverse set of grasp poses while ensuring full SE(3)-equivariance.Equivariant Generative Models Katsman et al. [25] introduce equivariant generative models on manifolds by applying equivariant manifold flows. Chen et al. [26] propose an SO(2)-equivariant conditional generative model for generating 2D trajectories in Euclidean space, and Rozenberg and Freedman [27] propose an E(3)and permutation-equivariant conditional generative model for generating 3D graphs in Euclidean space. Additionally, Zwartsenberg et al. [28] develop a permutationequivariant conditional generative model for generating traffic scenes and object bounding boxes in Euclidean space. These methods primarily focus on learning distributions either unconditional or in Euclidean space, making them difficult to apply to SE(3)-equivariant 6-DoF grasp pose generation, which requires conditional distributions on the SE(3) manifold. To address this limitation, we propose an SE(3)-equivariant conditional generative model on the SE(3) manifold. 
preliminaries: continuous normalizing flows on se(3): 
 As preliminaries, we introduce the Continuous Normalizing Flows (CNFs) [13, 14, 15] tailored for SE(3) cases. We begin by defining key notations; we denote an element of SE(3) by T = (R, x), where R ∈ SO(3) and x ∈ R 3 , and refer to it as a pose or grasp pose. Given a three-dimensional vector a = (a 1 , a 2 , a 3 ), [a] is a 3 × 3 skew symmetric matrix with entries defined as[a] 12 = -a 3 , [a] 13 = a 2 , [a] 23 = -a 1 .By decomposing flows on SE(3) into SO(3) and R 3 , we construct time-dependent angular and linear velocity fields, ω :[0, 1] × SE(3) → R 3 and v : [0, 1] × SE(3) → R 3, which map a time and a pose to angular and linear velocity vectors, respectively. We collectively define these two velocity fields as a time-dependent vector field u :[0, 1] × SE(3) → R 3 .The CNF on SE(3) models a target distribution q(T ) by transforming a prior distribution p 0 (T ) using the time-dependent angular and linear velocity fields ω θ and v ϕ where θ and ϕ are trainable parameters. This transformation is guided by the following ordinary differential equations (ODEs)EQUATIONIt generates a flow and a probability density path p t (T ) for time t ∈ [0, 1]. Sampling from p τ (T ) involves sampling from the prior p 0 (T ) and transforming these initial samples along the flow by solving the ODEs over t ∈ [0, τ ]. The parameters θ and ϕ are trained to ensure that the transformed distribution p 1 (T ) matches the target distribution q(T ).To model a target conditional distribution q(T |c) for some condition variable c, we use a prior conditional distribution p 0 (T |c) and velocity fields conditioned on c; the ODEs becomeEQUATIONwhich lead to a conditional probability density path p t (T |c) for t ∈ [0, 1]. In our setting, the condition variable c represents a point cloud of an object and the target conditional distribution q(T |c) corresponds to the distribution of successful grasp poses for that object.4 EquiGraspFlow: SE(3)-Equivariant 6-DoF Grasp Pose Generative FlowsIn this section, we introduce EquiGraspFlow, which generates diverse 6-DoF grasp poses from point cloud inputs using the conditional CNF approach while ensuring SE(3)-equivariance. Denoting a point cloud by P = {x k ∈ R 3 } K k=1 , EquiGraspFlow utilizes the time-dependent conditional velocity fields ω θ (t, P, T ), v ϕ (t, P, T ) and a prior conditional distribution p 0 (T |P) = p 0 (R)p 0 (x|P), where p 0 (R) is uniform over SO(3) and p 0 (x|P) is Gaussian in R 3 with its mean located at the center of the point cloud. As illustrated in Figure 2 , the model transforms p 0 (T |P) into p 1 (T |P) through a flow constructed from the velocity fields, as detailed in Figure 3 .We use a datasetD := {(P i , {T ij } Mi j=1 )} N i=1, consisting of pairs of the point cloud P i of the i-th object and its corresponding set of successful grasp poses {T ij } Mi j=1 . We assume that each set of grasp poses is sampled from the respective ground-truth conditional distribution q(T |P i ). The neural velocity fields ω θ and v ϕ are trained with D to ensure that the transformed conditional distribution p 1 (T |P i ) closely approximate q(T |P i ). We employ a Flow Matching framework [29, 30] to train the velocity fields and Guided Flows [31] to enhance the sample quality. Details are provided in Appendix B.2.In the following section, we explain the incorporation of SE(3)-equivariance into EquiGraspFlow. First, we will establish SE(3)-invariance of conditional distributions, which is essential for SE(3)equivariant grasp pose generation. We will then derive the necessary conditions to ensure that the transformed conditional distribution p 1 (T |P) is SE(3)-invariant. 
se(3)-invariant conditional distributions: 
 We first define transformations for point clouds, grasp poses, and three-dimensional vectors. Given an element T ′ = (R ′ , x ′ ) ∈ SE(3), a point cloud P = {x k } is transformed to T ′ P := {R ′ x k + x ′ }, a grasp pose T = (R, x) is transformed to T ′ T = (R ′ R, R ′ x + x ′ ), and a three-dimensional vector a is transformed to R ′ a. Next, we define SE(3)-invariance of conditional distributions: Definition 1. A distribution on SE(3) conditioned on a point cloud, denoted by p(T |P), is SE(3)invariant if p(T ′ T |T ′ P) = p(T |P) for any T ′ ∈ SE(3). This is a formal description, in the language of probability distributions, that leads to SE(3)equivariant grasp pose generation. For a transformed point cloud, equivalently transformed grasp poses have the same conditional likelihood as before the transformation.It is well-established mathematical fact that distributions transformed from an invariant prior via equivariant maps remain invariant [25, 26, 27, 28, 32, 33] . However, existing works primarily focus on distributions either in Euclidean space, unconditional, or both, making them unsuitable for grasp pose generation, which requires conditional distributions on the SE(3) manifold. To address this, we extend these works to model invariant conditional distributions on SE(3), thereby satisfying the required conditions. We thus define SE(3)-equivariance of time-dependent conditional vector fields on SE(3): Definition 2. A time-dependent vector field on SE(3) conditioned on a point cloud, denoted byu(t, P, T ), is SE(3)-equivariant if u(t, T ′ P, T ′ T ) = R ′ u(t, P, T ) for any T ′ = (R ′ , x ′ ) ∈ SE(3).This geometric condition is essential for modeling invariant conditional distributions via CNFs on SE(3). Starting from an invariant prior conditional distribution, equivariant velocity fields preserve the invariance of transformed conditional distributions over time by the following proposition. It is trivial to show that our prior conditional distribution p 0 (T |P) is SE(3)-invariant. In the subsequent section, we propose a neural network architecture that guarantees the remaining conditions in Proposition 1: the SE(3)-equivariance of the angular and linear velocity fields.We separate the SE(3)-equivariance into the equivariances on R 3 and SO(3). The R 3 -equivariance is achieved by subtracting the point mean µ = k x k /K from P and x in T . The SO(3)equivariance is achieved by adopting the Vector Neuron (VN) architectures [16] , which are designed to be SO(3)-equivariant. The structure of the velocity fields is depicted in Figure 4 . However, directly using the VN architectures is not straightforward since they require lists of threedimensional vectors as input. Among the inputs of the time-dependent conditional velocity fields, the point cloud P is a set of three-dimensional vectors, and the pose T can be represented as a list of three-dimensional vectors (R 1 , R 2 , R 3 , x), where R i is the i-th column vector of R. However, the time t is a scalar variable, making its incorporation into the VN architectures challenging. To use VNs with minimal modification, we propose an equivariant lifting layer that converts any scalar variables into three-dimensional equivariant vectors, so that they can be concatenated to the list of vectors while maintaining the equivariance of the VN architectures.Consider a list of C 1 scalar variables represented as a column vector s ∈ R C1×1 and list of C 2 threedimensional vectors represented as a matrix V ∈ R C2×3 -in our case, the time variable corresponds to the case when C 1 = 1. We propose a lifting layer, a mapping f lift : R C1×1 ×R C2×3 → R C1×3 , and consider it to be SO(3)-equivariant if it satisfies f lift (s, V R T ) = f lift (s, V )R T for any R ∈ SO(3).We construct the equivariant lifting layer as f lift (s, V ) = sf equi (V ) where f equi : R C2×3 → R 1×3 is any equivariant mapping, i.e., f equi (V R T ) = f equi (V )R T . This implies that f equi produces an equivariant vector from the input vectors V , and this equivariant vector is subsequently scaled by each of the C 1 scalar variables in s, resulting in list of C 1 vectors. It is trivial to show that this construction leads to the SO(3)-equivariance of f lift . For f equi , we use the VN architecture [16] . Finally, with the proposed equivariant lifting layer, we can construct an equivariant neural network for the time-dependent conditional velocity fields ω θ (t, P, T ) and v ϕ (t, P, T ). 
experiment settings: 
 Baselines We compare our model with existing grasp pose generative models, 6-DOF GraspNet [7, 34] and SE(3)-DiffusionFields [8] . To exclusively compare the generation performance of the generative models, we exclude the grasp evaluator used in 6-DOF GraspNet. SE(3)-DiffusionFields 3). The validation and test datasets are augmented with evenly sampled rotations using Super-Fibonacci Spirals algorithm [35] .The evaluation metrics we utilize are Earth Mover's Distance (EMD) [36] and grasp success rate. The EMD measures the distance between the distributions of the generated and ground-truth grasp poses, defined by the minimum geodesic distance on the SE(3) manifold required to align the samples. The grasp success rate is assessed by determining whether the Franka Panda gripper successfully holds the object following the grasping action. Both metrics are first averaged across the rotations for each object, and then averaged across all objects.We conduct two types of experiments in the Nvidia Isaac Gym simulator [37] . The first experiment measures the average EMD and grasp success rate, while the second experiment assesses the consistency of the EMD and grasp success rate with changes in the object's rotation. In the first experiment, the test dataset is augmented with three rotations, while in the second experiment, it is augmented with ten rotations. To measure the grasp success rate, we generate 100 grasp poses in the first experiment and 25 grasp poses in the second experiment. To eliminate inconsistency arising from sampling the prior distribution in the second experiment, we fix initial samples in PoiNt-SE(3)-Dif [8] and EquiGraspFlow, and then rotate them along with the object. Figure 5 and Figure 6 illustrate the consistency of performance as the object rotation varies for models trained with the SO(3)-aug setting. Figure 5 presents the average EMD and grasp success rate, with standard deviations indicated by error bars. The standard deviations are calculated with respect to the object rotation and are averaged across all objects. Figure 6 illustrates the generated grasp poses alongside EMD and grasp success rate values for various object rotations, where both the object and the generated grasp poses are inversely rotated to align all scenes. As indicated by the error bars in Figure 5 and the variance of values in Figure 6 , our model exhibits more consistent performance compared to the baselines, attributed to its incorporation of SO(3)-equivariance. Notably, our model shows zero standard deviations in Figure 5 and maintains an identical value regardless of the object's rotation in Figure 6 , demonstrating the perfect equivariance of our model. In the real-world experiments, we evaluate the grasp success rate of the baselines and our model trained with the SO(3)-aug setting. Two mugs and two bowls as shown in Figure 7 are used. For the mugs, the handles are randomly oriented to test grasping, and both mugs are further rotated 90 degrees along the horizontal axis for additional testing, indicated as (R) in Table 2 . The bowls are placed on the desk and also rotated at a certain angle along the horizontal axis, also indicated as (R) in After capturing RGB-D data of the object from multiple viewpoints, the object's partial point clouds are fused to create a full point cloud, which is subsequently downsampled to 1,024 points. This data is passed through each model to generate 100 grasp poses. Following this, grasp poses that are infeasible due to the constraints such as the manipulator's workspace, singularities, and collisions with the environments are excluded. From the remaining feasible grasp poses, ten grasp poses are randomly selected to measure the success rate.Table 2 shows the results of real-world grasping. Our model achieves performance comparable to that in simulation, demonstrating its seamless application to real-world tasks without sim-toreal issues. Even when using noisy real-world data, our model successfully generates grasp poses. Additionally, our model outperforms the baselines, similar to the simulation results. 
conclusion: 
 In this paper, we introduce EquiGraspFlow, an SE(3)-equivariant 6-DoF grasp pose generative model. Our approach revolves around two key ideas: (i) constructing a framework for learning invariant conditional distributions on the SE(3) manifold which is essential for equivariant grasp pose generation, and (ii) designing a novel equivariant lifting layer for our method. Unlike existing grasp pose generative models, our model ensures SE(3)-equivariance in generating grasp poses, resulting in improved performance and consistency. From simulation experiments, we conduct quantitative evaluations against baselines, demonstrating the superior grasp pose generation performance of our model. Furthermore, the consistent performance demonstrated across varying object rotations verifies the equivariance of the generated grasp poses and underscores the robustness of our method. Additionally, real-world experiments confirm the seamless applicability of our method to real-world scenarios, underscoring its practical relevance and effectiveness in real-world applications.Limitations and Future Works Our model, trained on full point clouds, may struggle to generate accurate grasp poses when robots encounter occlusions that prevent full observation of an object. To address this issue, we conducted additional experiments on generating grasp poses from partial observations of objects, as detailed in Appendix C.2. The results of this experiment demonstrate that, even with partial point clouds, our method effectively generates grasp poses and outperforms the baselines. In future work, this model could be applied in real-world experiments to generate grasp poses in constrained environments where objects are difficult to observe from multiple viewpoints.In the real-world experiments, we randomly selected ten grasp poses that provided statistically similar results to the generated grasp poses. However, if the objective is to identify the most promising grasp poses among the generated ones, one could leverage the probabilities of generated samples calculated from the CNFs or diffusion models. Selecting grasp poses with the highest probabilities could improve the success rate. This approach, however, has the drawback of primarily selecting grasp poses from regions with the highest probability, leading to a reduction in diversity. Additionally, our model, which generates grasp pose without considering obstacles, requires a collision-checking algorithm for real-world grasping. However, by utilizing a dataset that includes grasp poses for a target object in cluttered environments, we can develop a collision-free grasp pose generative model without the need for additional modules. Incorporating equivariance into such a model is a direction for future work.Noting that R[a]R T = [Ra] for any R ∈ SO(3) and a ∈ R 3 , we analyze γR (t) as follows:EQUATIONSimilarly, for γx (t), we have:EQUATIONFinally, note that γ(0) = T ′ γ(0) = T ′ T . Thus, γ(t) and γ(t) satisfy the same ODEs, and the uniqueness of the solution of the ODE ensures that γ and γ are the same integral curve. Consequently, we have f (t, T ′ P,T ′ T ) = T ′ f (t, P, T ) for any T ′ ∈ SE(3), demonstrating that f is SE(3)-equivariant. 
grasp pose generation: 
 TransformationSynthesizing six degrees of freedom (DoF) grasp poses from 3D observations of an object (e.g., surface point cloud) is a fundamental task in robotics. Approaches that produce a limited number of grasp poses [1, 2, 3] carry a high risk of grasping failures, especially in constrained environments. For example, obstacles or kinematic constraints may prevent the robot from reaching certain candidate grasp poses; therefore, the more diverse candidates, the higher the probability of grasp success.Earlier approaches to generate diverse grasp poses rely on geometric heuristics to suggest multiple candidates based on an object's geometry [4, 5] . These methods typically identify antipodal surface points -pairs of surface points with opposite surface normals -as the contact points of the gripper's fingertips. However, their reliance on antipodal points limits the diversity of the grasp poses and makes the methods sensitive to observational noise in surface points and normals.Recently, data-driven approaches have gained significant attention. These methods first generate sufficiently diverse grasp poses through simulation [6] , then use this data to learn the distribution of grasp poses and train grasp pose generative models [7, 8] . The trained models not only generate diverse poses but can also be made robust to observational noise by employing domain randomization.Mousavian et al. [7] employed the Variational Autoencoder (VAE) [9] as a grasp pose generative model, while Urain et al. [8] more recently trained an energy-based model via the denoising score matching [10] , demonstrating improved performance in learning complex grasp pose distribution.However, the primary flaw with existing grasp pose generative models is that they do not produce consistent grasp poses for rotated objects, leading to significant failure in some cases. An ideal model should generate grasp poses that transform identically for rotated and translated objects, as shown in Figure 1 . Such models are considered SE(3)-equivariant. Existing methods fail to ensure this equivariance, despite efforts to achieve it through data augmentation. Equivariant models, which have drawn significant attention from robot learning [11, 12] , provide a promising solution to this issue, enabling robust and accurate grasp pose generation.Our main contribution is EquiGraspFlow, a SE(3)-equivariant 6-DoF grasp pose generative model where the equivariance is guaranteed by the network architectures, hence no data augmentation is required. Specifically, we adopt the Continuous Normalizing Flows (CNFs) framework [13, 14, 15] , which, by utilizing time-dependent velocity fields as an infinite number of infinitesimally small transformations, learns complex distributions more effectively than diffusion models or discrete normalizing flows. We then formulate the necessary conditions to guarantee the SE(3)-equivariance of grasp pose generation for CNF on SE(3) conditioned on point cloud input. Among these conditions, a non-trivial one requiring careful consideration is SE(3)-equivariance of the time-dependent velocity fields. Our method leverages the Vector Neurons (VNs) [16] , which is a popular choice in constructing SE(3)-equivariant point cloud networks. Although VNs have shown remarkable success, it allows only vector variables as inputs and cannot accommodate scalar variables such as time. This limitation makes it challenging to use VNs for constructing SE(3)equivariant time-dependent velocity fields. To address this, we introduce a novel equivariant lifting layer that equips scalar variables with an equivariant vector basis, enabling the effective use of VNs in constructing SE(3)-equivariant time-dependent velocity fields.Through experiments conducted in both simulation and real-world environments, we have validated that our method surpasses existing 6-DoF grasp pose generative models [7, 8] . Notably, our method achieves superior results and consistent performance, even with changes in the object's rotation, without employing any augmentation strategies for equivariance. Real-world experiments validate that our method seamlessly applies to real-world applications. 
 
 Appendix: To demonstrate that an SE(3)-equivariant conditional flow preserves the invariance of an SE(3)invariant prior conditional distribution, we present the following proposition. Proof. To prove this proposition, we first extend Theorem 3 from [25] , which involves a general Riemannian manifold and a general group, to a conditional version.Consider a Riemannian manifold (M, h) with a group G. Denote the action of an element g ∈ G on M by the map L g : M → M. The map L g is isometric if, for any tangent vectors u and v at any point x ∈ M, the following condition holds:h(d(L g ) x (u), d(L g ) x (v)) = h(u, v), where d(L g ) x represents the differential of L g at x. If L g is isometric, then det J Lg (x) = 1 for any x ∈ M, where J Lg (x) denotes the Jacobian matrix of the map L g evaluated at x and expressed in local coordinates.Let c denote a condition variable. The conditional flow at time t is represented by the map f t,c : M → M. This flow transforms a prior conditional distribution p 0 (x|c) into the conditional distribution p t (x|c). The likelihood of the transformed conditional distribution is given by the following change of variables formula:EQUATION)Assuming the action of g ∈ G on c is well-defined and denoted by g• c, a conditional distribution p(x|c) is G-invariant if p(L g (x)|g • c) = p(x|c) for any g ∈ G. The conditional flow f t,c is G-equivariant if, f t,g•c (L g (x)) = L g (f t,c (x)) for any g ∈ G, i.e., f t,g•c • L g = L g • f t,c and L -1 g • f -1 t,g•c = f -1 t,c • L -1g . Assuming that the map L g is isometric for any g ∈ G, we can prove Proposition 3 in a general Riemannian manifold M and an isometric group G as follows:EQUATION)Proposition 3 is a special case where M = SE(3), G = SE(3), c = P, and the group action ofT ′ ∈ SE(3) on T ∈ SE(3), denote by L T ′ (T ) = T ′ T , is the left translation map which is isometric.Hence, Proposition 3 is proved.We now prove Proposition 1 by utilizing Proposition 2 and Proposition 3.Proof of Proposition 1. Since the angular and linear velocity fields ω and v are SE(3)-equivariant, it follows from Proposition 2 that their flow f is SE(3)-equivariant. Thus, by Proposition 3, the conditional distribution p t (T |P), defined via the flow of the velocity fields, is SE(3)-invariant.To model the time-dependent conditional velocity fields ω θ (t, P, T ) and v ϕ (t, P, T ) with SO(3)equivariance, we employ Vector Neuron (VN) architectures [16] , which are specifically designed for SO(3)-equivariance. The structure of these velocity fields is illustrated in Figure 8 .To encode the point cloud P, we utilize the backbone of the VN-DGCNN designed for classification tasks, up to the invariant layer, excluding batch normalization layers, and adding a 170-size EdgeConv module at the sixth module position. A mean pooling layer is then applied to pool the point dimension, extracting a representation z of 341 three-dimensional vectors. The grasp pose T is reconfigured by concatenating the three rotation column vectors and the translation vector, resulting in four three-dimensional vectors. Time t is converted into a three-dimensional vector through the lifting layer. The VN-MLP then concatenates these list of three-dimensional vectors as input and outputs the angular and linear velocities. The VN-MLP consists of five hidden VN-Linear layers, each followed by VN-LeakyReLU activation with a negative slope 0.2, and one output VN-Linear layer. The sizes of the hidden layers are (256, 256, 128, 128, 128), and the output layer size is 2, corresponding to two three-dimensional velocity vectors.The lifting layer uses the representation z and the grasp pose T to convert the scalar time t into a three-dimensional vector. This process involves a VN-Linear layer with a size of 1, producing an output matrix of size 1 × 3. This output matrix is then multiplied to the time variable t (size 1 × 1), resulting in a single three-dimensional vector.Laptop Dataset We use a dataset comprising 175 laptops, 94 mugs, 38 bowls, and 82 pencils obtained from the ACRONYM dataset [6] . The number of objects per each category and each dataset split are detailed in Table 3 . For each category -laptop, mug, bowl, and pencil -an average of 1,643, 1,124, 1,747, and 1,737 6-DoF grasp poses are provided, respectively.We employ the Flow Matching (FM) framework [29, 30] to train our CNF model. The core element of FM involves designing the per-sample target vector field u * t (T |T 1 ) and the corresponding probability path p t (T |T 1 ), where T 1 = (R 1 , x 1 ) represents a particular sample from the target distribution q(T |P). In our approach, we separate the rotation and translation components in u * t (T |T 1 ) = (ω * t (R|R 1 ), v * t (x|x 1 )). We then define the target angular and linear velocity fields ω * t (R|R 1 ) and v * t (x|x 1 ) as follows:EQUATIONConsequently, the training objective for EquiGraspFlow is designed asEQUATIONwhere || • || F denotes the Frobenius norm, and T = (R, x) and T 1 = (R 1 , x 1 ).One thing to note is that it might seem natural to design the vector field on the SE(3) manifold instead of separating the rotation and translation components, similarly to how we design the angular velocity field on the SO(3) manifold as shown in (9) . However, this approach results in screw motion-shaped paths of grasp poses, where the translation may not follow a straight line toward the target grasp pose. In the context of our grasp pose generation task, separating the rotation and translation and ensuring that the translation motion directly heads toward the target grasp pose is a more intuitive and appropriate vector field formulation.Guided Flows Guided Flows [31] is a technique that enhances the sample quality and efficiency of conditional generative models by integrating classifier-free guidance [38] into Flow Matching framework. This method employs a guided velocity fields during sampling, defined as a weighted sum of unconditional and conditional velocity fields. Using an empty set ∅ as a null condition for the point cloud input, we define the guided angular and linear velocity fields ωθ and ṽϕ as follows, utilizing the weight parameter β: ωθ (t, P, T ) = (1 -β)ω θ (t, ∅, T ) + βω θ (t, P, T ), ṽϕ (t, P, T ) = (1 -β)v ϕ (t, ∅, T ) + βv ϕ (t, P, T ).When ∅ is input, the point cloud encoder outputs a list of zero vectors as z. The guided velocity fields remain SE(3)-equivariant, as they are a linear combination of unconditional and conditional velocity fields, both of which are SE(3)-equivariant. To train the unconditional velocity fields, we randomly replace P with the empty set ∅ with a probability of 20% during training. For sampling, we use β = 2 to evaluate average performance and β = 2.5 to assess the consistency of performance.Optimizer Adam optimizer [39] with learning rate 1 × 10 -4 is utilized to train the baselines and our model. L2 regularization with hyperparameter 1×10 -5 is employed for training EquiGraspFlow.Sampling The fourth-order Runge-Kutta MK method on Lie groups [40] is utilized as the ODE solver, with 20 steps employed.Grasping in Simulation In the simulation experiments, the gripper and object float without any obstacles. After grasping the object, the gripper is shaken to robustly assess the success of the grasp.Grasping in Real-World For the real-world experiments, a Franka Panda gripper is equipped with a 7-DoF Franka Emika Panda robot, and the object is placed on a table. The object's point cloud is obtained using an Intel RealSense Depth Camera D435 mounted on the gripper. The depth camera captures an RGB-D image of the scene, and the object is segmented using Language Segment-Anything [41] , which is built on the Segment Anything Model [42] and Grounding DINO [43] , resulting in a partial point cloud of the object. By combining partial point clouds from multiple viewpoints, we reconstruct the full point cloud of the object. We then downsample the full point cloud to a uniformly downsampled point cloud with 1,024 points using voxel downsampling.The robot motion for grasping an object in real-world experiments is designed as follows. To prevent collisions with the object during the movement of the gripper toward the generated grasp pose, we first move the gripper to a pre-grasp pose. This pre-grasp pose is offset from the grasp pose by a small distance in the -z direction in the gripper's frame (the z-axis of the gripper's frame represents the direction of gripper's palm). Next, we move the gripper to the grasp pose and execute the grasp. Once the object is grasped, the gripper is lifted by small distance. The success of the grasp is manually determined based on whether the object is held securely by the gripper. After each grasping attempt, we manually reset the position and orientation of the object to its initial state.C. We conduct an experiment to measure the sampling time of the baselines and EquiGraspFlow. Each model generates 100 grasp poses from a point cloud with 1024 points. Sampling is conducted on a single NVIDIA GeForce RTX 3090 GPU, and results are averaged over 100 experiments.Table 4 shows the average sampling time of each model. While 6-DOF GraspNet [7] is about six times faster than EquiGraspFlow, its grasping performance is less-thandesirable. On the other hand, EquiGraspFlow is not only about six times faster but also shows better grasping performance than PoiNt-SE(3)-Dif [8] . To evaluate the performance of grasping in scenarios where objects cannot be fully observed from all viewpoints, we train and test the baselines and EquiGraspFlow when partial point clouds are input. Instead of uniformly sampling points from the mesh surface, we obtain a partial point cloud by sampling 512 points only from the mesh surface visible from a viewpoint defined by a 3D unit vector originating from the object's center. All grasp poses in the dataset are utilized regardless of the viewpoint, requiring the model to generate grasp poses even for unobserved parts of the object.The data augmentation for object rotation follows the same approach as in the full point cloud experiments, utilizing either the None or SO(3)-aug strategy in the training dataset, while the validation and test datasets utilize evenly sampled rotations using Super-Fibonacci Spirals algorithm [35] . The viewpoint vector on S 2 is randomly sampled in the training dataset, and is evenly sampled using Fibonacci lattice algorithm [44] in the validation and test datasets. Each object in the validation and test datasets is augmented with three rotations, and each rotated object is further augmented with three viewpoints. The grasp success rate is measured in Nvidia Isaac Gym simulator [37] using 25 generated grasp poses.The average Earth Mover's Distance (EMD) and grasp success rate under different augmentation strategies are presented in Table 5 . Despite a slight performance degradation from observing only portions of objects, our method still demonstrates strong performance, indicating its applicability in scenarios where objects are partially observed. Additionally, similar to the full point cloud experiments, EquiGraspFlow achieves lower EMD and higher grasp success rate compared to the baselines in both the None and SO(3)-aug settings. This demonstrates superior performance in learning grasp pose distribution and generating graspable poses even when objects are partially observed. Especially, EquiGraspFlow demonstrates a high grasp success rate even for mugs, which have complex shapes and are challenging for generating graspable poses under partial observation. This is attributed to the enhanced data efficiency resulting from the incorporation of SO(3)-equivariance, which enables the model to better learn how to grasp mugs from partial observation compared to the baselines. Figure 9 presents an error bar plot illustrating the changes in the average and standard deviation of EMD and grasp success rate across different numbers of rotations for each object category. As indicated by the increasing size of the error bars, the baselines exhibit less consistency in grasp pose distribution and grasping performance as the number of rotations increases. This suggests that the more diverse the rotations, the less consistent the baselines are in generating grasp poses. In contrast, EquiGraspFlow exhibits zero standard deviations in EMD values across all number of rotations, indicating that our model generates a consistent grasp pose distribution. This consistency results from the SO(3)-equivariance, which ensures the generation of equivariant grasp poses relative to the object's rotation. Although there are slight variations in the grasp success rate for EquiGraspFlow in some cases, these are attributed to the simulator's non-equivariance. As these deviations remain close to zero across all rotation counts, they further underscore the robustness of our model.Figures 10 to 13 present additional visualizations of the generated grasp poses. These figures show the generated grasp poses of a laptop, mug, bowl and pencil across ten object rotations, along with the EMD and grasp success rate values. The object's point cloud is rotated and input into each model; however, in these figures, both the objects and the generated grasp poses are inversely rotated to align all scenes. Successful and failed grasp poses are indicated in green and red, respectively.For all object types, the grasp poses generated by EquiGraspFlow are widely distributed across various parts of the objects, demonstrating that our model generates more diverse grasp poses compared to the baselines. For example, while PoiNt-SE(3)-Dif tends to concentrate generated grasp poses on specific parts of the laptop and bowl for certain object rotations, EquiGraspFlow distributes the grasp poses evenly across the objects regardless of their rotation. Additionally, PoiNt-SE(3)-Dif struggles to generate grasp poses that target the handle of the mug, whereas EquiGraspFlow generates grasp poses that are evenly distributed between the handle and the body of the mug.Furthermore, the variance in the EMD and grasp success rate values indicates that EquiGraspFlow exhibits more consistent results across different object rotations. While the baselines show variability in grasp pose distribution and grasping performance depending on the object's rotation, our model maintains identical values across the ten object rotations, demonstrating the perfect equivariance of our approach. 


Decision: PRE REBUTTAL:

High level summary of reviews:

Strengths:

- All reviewers state that the paper is clear, well-written, and well-organized.
- Multiple viewers suggest that the paper is technically sound and presents convincing methodological contributions.
- Multiple reviewers highlight the novelty of the approach (the idea of incorporating invariance in the generated distribution).
- The experimental section is thorough with real-world experiments. Reviewer jkBE highlights that experimental results are convincing and fully support the claims made in the paper.

Weaknesses:

- It lacks discussion of discrete normalizing flows and evidence on more object categories.
- The performance gain on the real-robot experiment is limited.
- Reviewer 2vdQ suggests that the main theoretical result (Proposition 1) is a basic and well-known mathematical fact. This should be clarified in updated versions of the manuscript.
- Reviewers Qvx1 and jkBE request the authors clarify a few details of the paper. Please see their extensive comments for details (e.g. run-time analysis, performance on more object categories, etc).

POST REBUTTAL:

During the rebuttal the authors clarified a number of points made by the reviewers and added additional experiments. This is clearly a strong submission and is stronger following the rebuttal.

Reviews: review: title: title: Initial Review
paper_summary: paper_summary: In this paper, the authors present a new grasp generative model. Compared to prior works, their model is SE(3) equivariant by definition, which results in improved performance. This is achieved on a methodological level through an equivariant encoder (VN) and a flow-based generative model.
main_review: main_review: Overall, the paper is clear and well-written. Moreover, it nicely motivates why SE(3) equivariant grasp pose generation is desired and presents convincing methodological contributions regarding the grasp generation architecture and exploiting a flow-based generative model that fits the needs of the desiredata. Additionally, real robot experiments are provided.

I still have a few remarks I will list in the following.

1) In Section 3, the authors explain continuous normalizing flows in SE(3). In line 105, they define a time-dependent vector field on SE(3), which, however, only maps to a single three-dimensional vector. Shortly after it becomes evident that rotation and translation are separated. It would nevertheless be good if the authors made clear from the beginning that they need two three-dimensional vectors for the flow in full SE(3).

2) In Section 4, and in general, the authors have to start from some sort of prior distribution. However, I could not find which prior distribution the authors used in the paper. Since this is important information, it should be added.

3) In the last paragraph of Section 4.2 the authors explain how to construct the equivariant lifting layer. The authors could provide more details at this point of the paper. If I understand it right, this lifting layer consists of an additional VN-MLP network outputting just a single point which is then scaled by time. Is this intuition correct? If yes, adding more context at this point of the paper would help clarity.

4) In the experiment section, Table 1 could be improved. For me, it would have helped to provide names for the first three rows of the table, i.e., something like Metrics, Object, and Data Augmentation. The latter, I guess, would help a lot in understanding the table correctly, as I initially thought that None corresponds to no augmentation, i.e., only using the nominal object poses during evaluation. However, if I understand correctly, it is related to the data augmentation strategies only.

5) At the end of Section 5.2. the authors mention in lines 281-283 that their method shows zero standard deviation. I found this statement a bit confusing at first, and I think the paper would benefit in clarity if the definition of this standard deviation (which is provided in Fig. 5) would be also added in the main text.

6) In the real-world experiments, the authors generate a bunch of grasp poses and subsequently, from the remaining feasible grasp poses, randomly select 10. While this is a valid procedure, SE(3)-Dif fields, since it outputs the energy values corresponding to each grasp pose, would allow for a more informed grasp selection strategy. Would this additional consideration further improve the performance of SE(3)-dif? In line with this question, especially in the limitations sections, the authors could discuss a bit in more detail how, in practice for their proposed method they would select the best grasp pose that is to be executed on the robot in a pick-and-place experiment, as testing ten generated grasp poses is not something one would do in practice.

7) As mentioned by the authors in the real-world experiments description (lines 222-231) and also in the limitations section, their method relies on a full / maximally complete pointcloud of the objects. I think it could be interesting to further improve the paper by providing an ablation w.r.t. how sensitive the current approach is w.r.t. the pointcloud being complete. I.e., does the performance drop to 0% success rate if only a single view pointcloud measurement is used? I think this information is not needed in the main paper but could be interesting information for the Appendix.

8) (Very Minor Comment). Section 3 is a background / preliminaries section. I would prefer that this is clear from its heading already, so the authors might consider to rename the section title to Background: Continuous ...
questions: questions: - What is the initial distribution from which you start sampling the grasp poses?
- In the real-world experiment, would SE(3)-Dif benefit and slightly reduce the number of failures when selecting the 10 grasp poses according to the energy?
- What procedure would the authors propose in a real-world scenario when they can only select a single grasp from the generated ones? I.e., what alternative strategies could be implemented with the presented approach apart from sampling at random?
- How sensitive is the method w.r.t. the pointcloud being complete?
limitations: limitations: 3
review_summary: review_summary: The paper is very well written and has substantial contributions, which are nicely showcased in the experiments. Moreover, the experimental results are convincing and fully support the claims made in the paper.
title: title: A Promising Approach for SE(3)-Equivariant 6-DoF Grasp Synthesis
paper_summary: paper_summary: The authors devise a SE(3)-Equivariant continuous normalizing flow model for generative 6-DoF Grasp Synthesis. Such equivariance can help the model generalize well with less data-preprocessing such as data augmentation. To this end, the authors  introduced necessary conditions for constructing such flows and a lifting layer in the conditional velocity field networks. The superior performance is validated in both simulated and real-world experiments for two object categories.
main_review: main_review: ### Strengths
- The problem tackled by this work is of highly practical relevancy for the learning-based grasping community;
- The presentation of the paper is generally clear and easy to follow except for some minor, less-friendly words for readers without sufficient background knowledge: L54: "no augmentation" and L56: "time-dependent velocity fields."
- The authors propose a technically sound formulation of the necessary conditions to guarantee the SE(3)-equivariance on the conditional continuous normalizing flows;
- A practical solution for constructing SE(3)-equivariant time-dependent velocity fields based on Vector Neurons is introduced;
- Comprehensive baselines, including other generative models such as VAE, GAN, and Diffusion models, are included in the experimental comparison;

### Weaknesses
- Lack of discussion of discrete normalizing flows: why do the authors focus on the continuous version?
- For the proposed lifting layer, instead of constructing the proposed layer, how about simply replicating the time variable to construct the vector? The concern of the current design is that, after the matrix multiplication, will the time information be lost or distorted, which might harm the actual effect of the velocity field?
- Lack of a run-time analysis in the discussion or experiment part;
- Lack of evidence on more object categories with various geometries;
- Lack of an ablation study on how the performance degrades with increasing rotations across the baselines. This ablation study is informative in the sense of knowing the limits of different approaches.
- In Table 2, it would be more understandable if "percentage" could be used.
- Performance gain on the real-robot experiment seems quite limited, i.e., only 2 successful grasps more. An explanation or failure analysis for this would be nice;
questions: questions: See the list of weaknesses above.
limitations: limitations: 3
review_summary: review_summary: For the technically sound appraoch and convincing results, I vote for weak accept while looking forward to the manuscript with the points above addressed.
title: title: Review
paper_summary: paper_summary: The work proposes a flow-based $SE(3)$-equivariant generative model on $SE(3)$ that is conditioned on a point-cloud in $\mathbb{R}^3$. The method is applied to grasp pose generation.
main_review: main_review: # Strengths
The idea of incorporating invariance in the generated distribution is original and sound in the context of grasp pose generation. 

The paper is extremely well-written and well-organized. In particular, the relevant mathematics is introduced succinctly, with an elegant and clear formalism. 

The experimental section is thorough. It contains both simulated and real-world experiments, evaluated via probabilistic distances (geodesic earth-mover distance) and downstream tasks (grasp success rate). 


# Weaknesses  
The main theoretical result (Proposition 1) is a basic and well-known mathematical fact, even in the applied machine learning literature. It holds, with the same simple proof, on any (compact) Lie group: the flow of an equivariant vector field on a Lie group $G$ transports $G$-invariant distributions to $G$-invariant ones. Similarly to this work, the result is used in several papers on group-invariant flow-based generative modelling – see, for example, Theorem 1 and Theorem 2 in [1], or Theorem 1 in [2], or Proposition 1 in [3]. I believe the authors should stress out that the result is not novel, and is a standard argument in the field (with appropriate references). 

[1] Köhler et al., Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities, ICML 2020. 

[2] Zwartsenberg et al., Conditional Permutation Invariant Flows, TMLR 2023. 

[3] Lu et al., Structure Preserving Diffusion Models, 2024. 

# Additional Comments
-Line 18: The acronym DoF (degrees of freedom) is never introduced explicitly – I suggest doing it here. Even though it is standard terminology, it is customary to introduce all the acronyms in a paper for better accessibility.
questions: questions: I have no questions for the rebuttal.
limitations: limitations: 3
review_summary: review_summary: I think that the work combines equivariant deep learning with flow-based generative modelling in a clever way for the purpose of grasp pose sampling. Besides the weakness above – which has a theoretical nature – the work deserves acceptance, in my opinion, for its methodological and applied contributions.