ET-SEED: EFFICIENT TRAJECTORY-LEVEL SE(3) EQUIVARIANT DIFFUSION POLICY
Authors: Chenrui Tie
Yue Chen
Ruihai Wu
Boxuan Dong
Zeyi Li
Chongkai Gao
Hao Dong
Venue: ICLR 2025 Poster (2025)
Keywords: Robotics; Manipulation; Equivariance
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduces ET-SEED, an SE(3) equivariant diffusion model that leverages spatial symmetries to improve data efficiency and spatial generalization in robotic manipulation tasks while reducing training complexity.

Abstract: Imitation learning, e.g., diffusion policy, has been proven effective in various robotic manipulation tasks.
However, extensive demonstrations are required for policy robustness and generalization.
To reduce the demonstration reliance, we leverage spatial symmetry and propose ET-SEED, an efficient trajectory-level SE(3) equivariant diffusion model for generating action sequences in complex robot manipulation tasks.
Further, previous equivariant diffusion models require the per-step equivariance in the Markov process, making it difficult to learn policy under such strong constraints.
We theoretically extend equivariant Markov kernels and simplify the condition of equivariant diffusion process, thereby significantly improving training efficiency for trajectory-level SE(3) equivariant diffusion policy in an end-to-end manner.
We evaluate ET-SEED on representative robotic manipulation tasks, involving rigid body, articulated and deformable object.
Experiments demonstrate superior data efficiency and manipulation proficiency of our proposed method,
as well as its ability to generalize to unseen configurations with only a few demonstrations. Website: https://et-seed.github.io/

Published as a conference paper at ICLR 2025
ET-SEED: EFFICIENT TRAJECTORY-LEVEL SE(3) EQUIVARIANT DIFFUSION POLICY
Chenrui Tie 1.2+
Tie!.
Yue Chen
Ruihai Wu
Boxuan Dong*
Zeyi Li
Chongkai Gao Hao Dong"
'Peking University NNational University of Singapore
chenrui. t.tetiieuu. nus edu
yuechenêsta .kuu.pku..a edu.
wuruiha @pku edu. en
Succres Simulation Tanka (3G)
Diftukee Talk
ICLR
Transforned
Equivariant Sequence
Average Succoss Rtare of Simolation Taske (%)
Gives 25 Dano
Givan Scaee
AC P
Figure 1: ET-SEED is visual imitation learning algorithm that marries SE(3) equivariant visual representations with diffusion policies. (a) ET-SEED achieve surprising efficiency and spatial gen- eralization than baselines. (b) When the input object observation is rotated translated, the output action sequence change equivariantly. (c) Visualizations of simulation environments.
ABSTRACT
Imitation learning, e.g., diffusion policy, has been proven effective in various robotic manipulation tasks. However, extensive demonstrations are required for policy robustness and generalization. To reduce the demonstration reliance, we leverage spatial symmetry and propose ET-SEED, an efficient trajectory-level SE(3) equivariant diffusion model for generating action sequences in complex robot manipulation tasks. Further, previous equivariant diffusion models require the per-step equivariance in the Markov process, making it difficult learn policy under such strong constraints. We theoretically extend equivariant Markov kernels and simplify the condition of equivariant diffusion process, thereby significantly improving training efficiency for trajectory-level S1E(3) equivariant diffusion pol- icy. We evaluate ET-SEED on representative robotic manipulation tasks, involving rigid body, articulated and deformable object. Experiments demonstrate superior data efficiency and manipulation proficiency of our proposed method, as well its ability generalize to unseen configurations with only a few demonstrations. Videos and code are available at: https://et-seed /ttss//ttteeedggituu.ooo///
INTRODUCTION
Imitation learning has achieved promising results for acquiring robot manipulation skills (Zhu et al. 2023 Fu al. 2024 Chi al.
2023 Wang et al. 2025). Though, one of the main challenges of imitation learning is that it requires ex extensive demonstrations to learn a robust manipulation pol-
icy (Brohan al. 2022 Liu et al. 2024 Mandlekar al. 202T). Especially once the spatial pose of the object to be manipulated runs out of the demonstration distribution, the policy performance will easily decrease. Although some works seek tackle these issues through data augmentation (Yu et al. 2023) or contrastive learning (Ma et al.) 2024), they usually require task-specific knowledge or extra training, and without theoretical guarantee of spatial generalization ability.
"Equal contribution
1 "Coresponndig author

---

Published as a conference paper at ICLR 2025
Another promising idea is to leverage symmetry. Symmetry is ubiquitous in the physical world, and many manipulation tasks exhibit a specific type of symmetry known as SE(3) Equivariance. SE(3) is group consisting of 3D rigid transformations. For example, shown in fig. [Tb,, a real robot arm is required to write characters "ICLR" on a paper or fold a garment, when the pose of the paper the garment changes, the manipulation trajectories of the end-effector should transform equivalently. Employing such symmetries into policy learning can not only improve the data effi- ciency but also increase the spatial generalization ability. Recent works on 3D manipulation have explored using SE(3) equivariance in the imitation learning process. Most of these works focus on equivariant pose estimation of the target object or end-effector (Ryu al. 2024] Hu al. [024] Gao et al. 2024). Trajectory-level imitation learning has achieved state-of-the-ar performances on di- verse manipulation tasks (Chi et al. 2023, Yang et al. 2024b). By generating a whole manipulation trajectory, this kind of method is capable to tackle more complex manipulation task beyond pick- and-place. For trajectory-level equivariance, Equivariant Diffusion Policy (Wang et al. 2024) and Equibot (Yang et al. 2024a) propose equivariant diffusion process for robotic manipulation tasks.
However, previous trajectory-level diffusion models for robotic manipulation have two key limita- tions. First, maintain equivariance throughout the diffusion process, these models assume that every transition step must preserve equivariance. As we will show section 4.11 training neu- ral networks with equivariance is more challenging than neural networks with invariance, requir- ing additional computational resources and leading to slower convergence. This design constrains the model's efficiency, making it hard for tackling complex long-horizon manipulation tasks. Sec- ond, these models define the diffusion process in Euclidean space, which is not a natural defini- tion, and limits the expressiveness. Since the focus on equivariant diffusion processes within the SE(3) group, it is more natural to define both the diffused variables and the noise as elements of the SE(3) group, which will lead better convergence and multimodal distributions representa- tion (Urain et al. 2023).
In this work, we propose ET-SEED, a new trajectory-level SF(3) equivariant diffusion model for manipulation tasks. ET-SEED improves the sample efficiency and decreases the training difficulty by restricting the equivariant operations during the diffusion denoising process. We extend the equivaraint Markov process theory and prove that during the full denoising process, at least only one equivariant transition is required. Then, we integrate the diffusion process on SE((3 manifold (Jiang et al. 2024) and (3) transformers (Fuchs et al. 2020) to design new trajectory-less equivariant diftusion model on SE(3) space. In experiment, we evaluate our method on several common and representative manipulation tasks, including rigid body manipulation (rotate triangle, open bottle cap), articulated object manipulation (open door), long-horizon tasks (robot calligraphy), and de- formable object manipulation (fold and fling garment). Experiments show our method outperforms SOTA methods in terms of data efficiency, manipulation proficiency and spatial generalization abil- ity. Further, in real-world experiments, with only 20 demonstration trajectories, our method is able to generalize to unseen scenarios.
In summary, our contributions are mainly as followed:
We propose ET-SEED, an efficient trajectory-Ievel SE(3) equivariant diffusion policy de- fined on SE()) manifold, which achieves proficient and generalizable manipulation pol- icy with only a few demonstrations.
We extend the theory of equivariant diffusion processes and derive a novel S SF(3) equivariant diffusion process, for simplified modeling and inference.
We extensively evaluate our method standard robot manipulation tasks in both simu- lation and real-world settings, demonstrating its data efficiency, manipulation proficiency, and spatial generalization ability, significantly outperforming baseline methods.
RELATED WORK
2.1
LEVERAGING EQUIVARIANCE FOR ROBOTIC MANIPULATION
Previous research has demonstrated that leveraging symmetry or equivariance in 3D Euclidean
space can improve spatial generalization in a variety of robotic manipulation tasks. Lim et al.
2024): Hu et al. (2024); Simeonov al. (2022)) Xue et al. (202bb) Gao al. 2024 proposed
2

---

Published as a conference paper at ICLR 2025
(3) equivariant model for grasp pose prediction. Other works have also leveraged this symmetry in tasks such as part assembly (Wu et al., 2023; Scarpellini et al. 2024), object manipulation on desktop (Wang et al., 2024), articulated and deformable object manipulation (Yang et al. 2024b) and affordance learning (Chen et al. 2024 Most of these studies either focus solely on generating single 6DD pose or fail guarantee end-to-end equivariance across the entire F space. In this paper, our proposed method is capable of generating manipulation trajectories while theoretically maintaining end-to-end equivariance over the entire (3)) group.
2.2 EQUIVARIANT DIFFUSION MODELS
Diffusion models (Sohl-Dickstein et al., 2015 Ho et al. 2020) compose a powerful family gen- erative models that have proven effective in robotic manipulation tasks (Chi et al. 2023). Previous studies (Guan et al,, 2023 Schneuing et al. 2022) have investigated the effectiveness of combin- ing spatial equivariance in the diffusion process to increase data efficiency and improve the spa- tial generalization ability of the model. GeoDiff (Xu et al. (022) gave a theoretical proof of SE(3) equivariant Markov process. Diffusion-EDFss (Ryu et 2024) and Orbitgrasp (Hu al., 2024) introduced (3)(3 equivariant diffusion processes for target grasp pose prediction, but lack the capability to generate entire manipulation trajectories. [Wan al. (2024) proposed an equivari- ant diffusion policy capable of addressing SO(2) equivariant tasks. EquiBot (Yang et al. 2024a) extended equivariant diffusion policies SIM(3) transformations, with the assumption that every transition step in the diffusion process is equivariant, which demands high training cost, We further discuss the conditions of (E)) equivariant diffusion process and prove that not each, but at least one equivariant step is required. Based on this condition. propose novel SE(3) equivariant diffusion model achieving better performance than previous works.
2.3 DIFFUSION ON SE(3) MANIFOLD
Most diffusion models define the diffusion process on pixel space (Ho et al. 2020) or 3D Euclidean space CCi et al. 2023). Leach et al. (2022) introduces denoising diffusion model on 50(3) group.
Siffuuiifousoon Fields (Uain et al. 2023) suggests that in 6-DoF grasp pose generation scenarios, formulating the diffusion process in SE(3) manifold provides better coverage and representation of multimodal distributions, resulting in improved sample efficiency and performance. Jiang et al. (2024) proposes a SE(3) diffusion model for robust 6D object pose estimation. In this work, we introduce an equivariant diffusion model on SE(3) manifold for robot manipulation, revealing the superiority of defining equivariant diffusion process on S F:3)) over Fuclidean space.
3 PRELIMINARY BACKGROUND
SE(3) Group and its Lie Algebra. SF(() (Special Euclidian Group) is group consisting of 3D rigid transformations. Each (3) transformation can be represented as a 4 x 4 matrix (denoted as T",, indicating linear transformation
homogeneous 4-vectors. Formally, T [8 [Rt] 1]] where
R E R3x3 is a rotation matrix, t E R3 is translation vector. The Lie algebra se(3) is a linear 6D vector space corresponding to the tangent space of SE(3) Each element of se(3)
6D
vector 8 E R The mutual mapping between SE(3) and se(3) is achieved by the logarithm map Log : SE(3) + R and the exponential map Exp : R + SE(3). More information about SE(3) and se(3) can be found in appendix A.
SE(3) Equivariant Function. Generally, we call a function X input space X' to output space y is equivariant on X and y respectively denoted by px and p such that Ygea p>' (g) o
y that maps elements from group G if there are group representations of G p (g)
+ X
In other words, the function commutes with representations of the group G. In this paper, focus on SE(3) group. and represent elements of SE(3) as 4 x 4 matrices. As special cases of general equivariance,
define SF) equivariant and invariant functions as:
3

---

Published as a conference paper at ICLR 2025
Definition SE(3) Equivariant Function.
A function f X + is called SE(3) equivariant if
VrESE(s): oo :
(I)
Definition 2 SE(3) Invariant Function.
function f x: - + is called (E(3) invariant if
VTSEE()) f
SE(3) Equivariant Trajectory. In many robotic manipulation tasks, the trajectories of the manip- ulator show a certain symmetry. If the representation of a trajectory under certain coordinate frame (3) invariant, we call the trajectory as SE(3) Equivariant. Formally, it can defined as
Definition 3 SE(3) Equivariant Trajectory.
A trajectory {s: is called SE(3) equivariant if exists a coordinate frame A, such that for any transformation T SE(3) applied on both the trajectory and the coordinate frame(denoted as { = T and A' AA,, the representation of 1 by the basis of A' same as the representation of [s:]:=1 by the basis of A.
This property means the trajectory is "attached" certain frame, and when the frame transforms, the trajectory transforms accordingly. In our experiments, we select 6 representative manipulation tasks with this symmetry. Further discussion can be found in appendix G.
4 METHOD
Problem Formulation. We formulate the problem an imitation learning setting, aiming learn a mapping from observation O to action sequence A, with some demonstrations from an expert. In our setting, the observation O is colored point clouds P {(11,c1), c1), (xN,CN)) cx)} e R. Xx6 The action is defined directly as the desired 6D pose H E SF(3) of the end-effectors. So in our setting, the action sequence means the trajectory of end-effectors. This experimental setup does not require additional input information, and the action definition is both intuitive and consistent with real robot control, making it applicable to a wide range of robotic manipulation tasks.
In this paper, propose ET-SEED, trajectory- level end-to-end SE(3) equivariant diffusion model for robotic manipulation. ET-SEED can theoreti- cally guarantee the output actions are equivari- ant to any SF(3) transformation applied the input observation, while only involving one equiv- ariant denoising step. Fig. 2 is general example to show how it works, given an observation and noisy action sequence, our model first implement K 1 invariant denoising steps, and pass the result into the last equivariant denoising step to generate S (3) equivariant denoised trajectory.
k
x
x
Invariant Transitions
Equivariant Transition
Illustration of the denoising pro- cess of ET-SEED. A random trajectory K first passes through an invariant transition for K times and finally passes an equivari- transition once. This process efficient while keeping SE(3) equivariance property.
Figure 2:
We will discuss equivariant Markov processes fur- ther to explain the correctness and advantages of our proposed diffusion process section [.1], with only one denoising step SE(3) equivariant and the rest SE invariant. Then introduce our modified SE(3) invariant and equivariant backbones in sec- tion 4.2 and illustrate our SE(3) equivariant diffu- sion process in section
4

---

Published as a conference paper at ICLR 2025
4.1 EQUIVARIANT MARKOV PROCESS
For Markov process x and any roto- roto-translationn transformation T E SE(3). Geodiff Xu
al. 2022) shows that if the initial probabilistic distribution SE invariant, i.e. t.e.,(ph p(x
p(7x and the Markov transitions (=**) are SF(3) equivariant for any 1 S k K
i.e., p(x 1|=²) p(Tx*~1(T'' then the density zo satisfies p(2)) = p(T'zo) Equibot (Yang
et al.) (024a) adapts the theory and makes it more consistent with the robotics setting. They in-
volve an additional condition c (can be seen as an observation) and show that if the initial distri- bution p(x110 Ic) and transitions are all equivariant, i.c., p(c.,((x' (c) p(T (Tc), p(= " |,*,
p(Tx* [Tx., Tc) then the marginal distribution satisfies p(x*(c) = p(Tx"TT))
In this paper, discover that the condition of getting equivariant marginal distribution p(=* can be weaker. Formally, we first define three Markov transitions with different properties.
(1 l c) = =pi(=* |12" Tc) (2
* la c) p2 - 12 Tc)
(a a 1 c) = pa 11'= Tc)
Then derive the marginal distribution using the three types of Markov transitions. We have the following statement. See appendix B for the detailed proof.
Proposition 1 For Markov process K if the initial distribution p(x (c) pl first K n + transitions follow the property of pt. the middle I transition follows p2, and the last n - 2 transitions follow Ps. then the final marginal distribution satisfies p(z"lc)= (TT))
Previous works (Wang et al. 2024 Yang et al. 2024a) make all transitions Pa-like, which is special case of proposition! In practice we observe that training neural networks to approximate the properties of P2 and P3 much more challenging compared to Pt, both in terms of performance and training cost. When the condition e is transformed by a SE(3) element, the distributions in p2 and p3 change equivalently, while the distribution in Pi remains unchanged. Learning to output an equivariant feature is clearly more challenging for neural networks than producing an invariant feature. Additionally,
most of implementations of equivariant networks, building and training model whose output is S SFF(3) equivariant to the input takes up more computing resources than S SF(3) invariant version. We design experiments to validate these facts. The results show that whether in single step multiple steps setting, training invariant model is casier than equivariant model. Details of this contirmatory experiment can be found appendix D
In ET-SEED, set the parameter ferred to as "SE(3) Invariant Denoising Steps") and one Py-like transition (referred to as the "3E(3) Equivariant Denoising Step"). This key design choice significantly reduces the training complexity, thereby enhancing the overall performance of our method.
2, meaning there are K pı-like transitions (re-
4.2 SF()) EQUIVARIANT BACKBONE
In order to generate whole manipulation trajectories, it's necessary that the network has the ability to output a translation vector at anywhere in the 3D space (even beyond the convex hull of the object), which not be achieved by directly using existing equivariant backbones (Fuchs al., 2020 Deng et al., 2021 Liao & Smidt 2002). In this paper, based on SE(3) Transformer (Fuchs) al. 2020). we propose SE(3) equivariant backbone Eequiv and invariant backbone Ein,. which are surtable for predicting (3) action sequences while theoretically satisfying definition ] and 2 The implementation details can be found at appendix [+] The input of backbone is of points coordinates X' E Nx3 with some type-0 features D and type-1 features D attached on each point. Type-0 vectors are invariant under oototrranslation transformations and type-I vectors rotate and translate according to SE(3) transformation of point cloud. The output can be elements of SE(3). each element represented as a 4 x matrix. For the SE()) equivariant model Eeguios we have
VTESE(3) T (X; D6 D( Enguiv (TX; D8 " TD())
5

---

Published as a conference paper at ICLR 2025
And for the SE(3) invariant model Eiv,, we have
VreSE(3) (,; D D( = Einu (Tx: D. "D
(5)
Algorithm Training phase repeat
Algorithm 2 Inference phase
for & K..., 2 do
Sample A ~Pdata
Predict A = (0,400
Sample k ~ Uniform(11,
for H 1 do
for H. A do
H --1 Exp(Ao Log Log(H?-
H Exp I F (Vār;HP,
Log(H)))
Assign A = ( [
Assign 1a [H) 1
Predict A = (0, A kt)
Predict A'-00 = Eeguso A0)4;
Optimize
loss 10 (
for H; A
Update H = H
until converged
Return: (H)) i=
4.3 SE(3) EQUIVARIANT DIFFUSION PROCESS
Repeat
Input
Equivariant
Self-Atenntonn
TensorField Network
Invariunt Denoise
Lincar projection
Graph Pooling
Noisy Action Sequences Ax
Equivariant
Action Sequences A
normalization
Inv. SE(3) Transformer
Equivariant
Self-Atlention
TensorField Network
Equivariant Denoise
Linear projection
Graph Pooling
Point Cloud
Equivariant
Noise Free Action Sequences A
normalization
SE(3) Transformer
Repeat x
Figure 3: Overview of our pipeline. A colored point cloud and random sampled action sequence are first passed through K SE33 invariant denoising steps and then SE(33 equivariant de- noising step generate noise free action sequence. Although Inv. SF((3 Transformer and Eqy. SF(3) Transformer have same network architecture, the feature types of input and output differ, re- sulting in different coefficient matrices in network forward. Details can be referred to in appendix
Inspired by standard diffusion model, ET-SEED progressively disturbs the noise free action H SE(3) into noisy action H As standard diffusion process assume the final noisy variable zy fol- lows the standard Gaussian distribution A (10, I), we assume the noisy action H follow a Gaussian distribution FF3) centered the identity transformation H. So (E(3) diffusion formula, which represent the H ~ g( qHH H
interpolation-bate at noise step k(1 < k < K) as
H Exp (rvI are) F (Vār: H E
Perturbation
Interpolation
The interpolation function F (Vār H HH) is an intermediate transformation between the origin action H and the identity transformation H. By applying perturbation noise Exp (7v1 ā,e) the intermediate transformation, get diffused action H² As shown in algorithm 1

---

Published as a conference paper at ICLR 2025
design model Sy to predict the applied noise in supervised learning fashion. More explanation aboutf6 found in Diang et al. (2024) or appendix
This formulation is analogy of DDPM (Ho et al. 2020 which represent the noisy image as
21 = 0, + -EE,E~N(00.)
We can treat the first term of as interpolation between Zo and 0, second term as external noise.
The goal of (3)() reverse process is to train denoising network, gradually refine the noisy action to the optimal ones. As illustrated in fig. 3 the imput of reverse process is an observation O, an noisy action sequence A [H H ,.... H where Tр is the action prediction horizon, and each H is drawn from a S (3) Gaussian distribution centered at identity transformation IHI. The denoising process forms a Markov chain A - AK +
- A In each denoising step, the input of denoising network so consists of observation O, noisy action sequence A and scalar condition k, outputs the predicted relative transformation between 4 and noise-free action sequence A Formally, have
Å (6,0AAAH))
(8)
To ensure the overall SE(3) equivariance of our pipeline, we propose novel design of denoising network sg. consists one SE(3) invariant backbone Einu and one SE(3) equivariant backbone Eequin- We use Eive to denoise in the first K iterations and use Eequio to denoise in the last iteration. Formally, 8g is defined as
Ein (O, Enquio (0,A" :)
(OAA:))=
As illustrated algorithm 2. the first denoising iteration, we use SF(3) invariant backbone Exn to predict noise, and for each H A we use the i-th transformation H of predicted noise sequence k to implement denoise step by
H Exp(Ao Log H HH) A1 Log(H
(10)
This formulation, by minimizing the KI. divergence of the posterior distribution and prior distribu- tion of H is able to infer more reliable distribution for H (Jiang et al. 2024).
By implementing the above SE)3) invariant denoising step for K times, we get the partially denoised action sequence A'. which is invariant to any SE(3) transformation of O. In the last denoising iteration, we use SE)(3 equivariant backbone Espuio predict noise - and directly apply each H!! 1- of 1 on corresponding action H. H! And finally get the noise-free action sequence A H H H..1.
With such design, the end-to-end equivariance is guaranteed. When the input observation O is transformed by any SE(3) element T, the output denoised action sequence A will be equivariantly transformed. Formally, we have following proposition, with detailed proof attached in appendix C
Proposition 2 For a Markov process AK.0 if AK is sampled from Gaussian distribution, A AAK ()) Then, VT e SE(3): TA EETSEED(TO,A"

---

Published as a conference paper at ICLR 2025
Table 1: Success rates (1) and standard deviation of different tasks in simulation.
Open Cap
Open
Trinagle
NP
NP
T
NP
Method
25
25
50
25
50
25
50
25
25
50
2024
61:22.24
16:55.88
89-22.24
DP1-Aup
35.0::5.0
44143
38:4447
464742
4348.37
5416652
3024,18
40:28.22
35:3.54
42:44447
52155.00
4114.18
EquiRor Yang ot al.
30246
6322..7
73:22:4
63:5.70 70
7747758
56:22.24
72422.44
5827.56
77:77.88
67:2.74
84:22.24
6418666
814-2344
74:66.52
66:22.24
66-22.74
7::22:24
93:22.44
85-22.24
Calligmphy
Pold Garment
Fling Garment
NP
NP
Method
50
DP3 Ze al 2024
28:12.74
50:1351
0::0000
34274
46:12.24
60:14.18
41:.448
8:1-1488
361:5,48
67:14.48
4::5.48
9:28.22
842774
21
34204
17:1151
13:5.70
2727.58
17
31
28:7.58
38:4448
11::4118
3122.24
EquiBot Yang
20246
24
43-58.37
14:4
40th
34:14.18
5842274
354:274
60:7790
35:66.12
61:6.52
56166.52
64-8.22
(O:SEED(Our))
38:2.74
55+:354
36:6652
54::8222
47:22.74 67:2.74
49:22.24
09:24.18
50:5.00
67:4448
48:74.47
02:5.70
Table 2: SE(3) Geodesic distances (L) of different tasks in simulation.
Open
Triangle
Method
25
50
25
50
25
25
50
25
50
DP3 Ze al. 2024
0.257
0.197
1,413
1.785
0.384
0.354
0:478
0.442
0.265
0.192
1.812
1.627
DPS-Ang
0.283
0.234
0.276
0.218
0.391
0.315
0.442
0.320
0.247
0.187
0.578
0.447
EquiBet (Yang
0.197
0.170
0.241
0.107
0214
ET-SEED (Ours)
0.127
0.124
0.127
0.101
0.098
0
0.104
0.087
Calligraphy
Fold Garment
Fling Garment
T
NP
T
NP
T
NP
Method
50
25
25
50
2024
4.88
4.662
0.298
4.129
0.348
4.993
4,752
4,365
1,318
0.976
1.524
1.219
1.318
0.976
1.524
1.219
Equitet (Yang ct.
20246
0.291
0.117
0.282
0.119
0
0.293
0.387
0.288
0.418
0.343
0.437
0,338
ET SEED (Ours)
0.124
0,083
0.121
0,089
0.299
0.149
0.287
0.136
0.349
0.179
0.337
0.186
5 EXPERIMENTS
We systematically evaluate ET-SEED through both simulation and real-world experiments, aim- ing to address the following research questions: (1) Does our method demonstrate superior spatial generalization compared to existing imitation learning approaches?
(2) Can our method achieve comparable performance with fewer demonstrations? (3) Is our method applicable to real-world robotic manipulation tasks?
5.1 SIMULATION EXPERIMENTS
Tasks. We design six representative robot manipulation tasks: Open Bottle Cap, Open Door, Rotate Triangle, Calligraphy, Cloth Folding, and Cloth Fling. These tasks encompass manipulation of rigid bodies, articulated bodics, and deformable objects, as well as dual-arm collaboration, long-horizon tasks, and complex manipulation scenarios. A brief overview is illustrated in fig. 1 For each task, we set up multiple cameras to capture full point clouds of the objects to be manipulated. We assume each robot manipulator operates within a complete 6DoF SE(3) action space. Further details and discussions of their equivariant properties can be found in appendix G
Baselines. We compare our method against the following baselines:
3D Diffusion Policy (DP3) (Ze et al. 2024): A diffusion-based 3D visuomotor policy.
3D Diffusion Policy with Data Augmentations (DP3+Aug): Same architecture as DP3, with SF)) data augmentation added.
8

---

Published as a conference paper at ICLR 2025
EquiBot (Yang al. 2024a): A baseline combines SIM(3)-equivariant neural network architectures with diffusion policy.
DP3 and Aug+Aug are used to compare ET-SEED with baseline methods that utilize data augmen- tation to achieve spatial generalization, while EquiBot allows for comparison between different architectures of equivariant diffusion process.
Augmentations. The Aug+Aug baseline utilizes augmentations during training. In all environ- ments, training data is augmented by (1) rotating the observation along all three axes by random angles between 0 and 90p, and (2) applying random Gaussian offset to the observation. The standard deviation of the Gaussian noise is set to 10% of the workspace size.
Evaluation. Following the setup of al. (2024), collect demonstrations and train our policy under the Training setting (T), subsequently testing the trained policy on both T and New Poses (NP), where target object poses undergo random SE(3) transformations. We evaluate all methods using two metrics, based on 20 evaluation rollouts, averaged over 5 random seeds. Since we generate complete manipulation trajectories, the final success rate alone inadequate for fully assessing the trajectory's is quality. We calculate the geodesic distance between each step of the predicted trajectory and the ground truth trajectory, providing a more comprehensive reflection of the trajectory's 's overall quality. The geodesic distance between each step of the predicted trajectory and the ground truth trajectory, we can obtain more accurate reflection of the trajectory's overall quality, The definition of geodesic distance between T, f E SE(3) is
D gea
T f) =
[|ITog(RTT))
|
+ l|ē tl|²
(11)
where R and t are the rotation and translation parts of T. We report D yeo in the same manner as success rates.
Results. Table and 2 provide quantitative comparison between our method and the baseline. Both DP3 and its augmented variant demonstrate strong performance in the training setting (T), but they exhibit significant drop in performance when faced with New Poses (NP) scenarios. This highlights that merely incorporating data augmentation is insufficient for the model to generalize effectively unseen poses. Instead, leveraging equivariance proves essential for enhancing spatial generalization.
Table 3: Ablation studies.
While EquiBot achieves commendable results in both success rate and Dzea. it struggles with more complex, long-horizon tasks such as Calligraphy and Fold Gar- ment. Also, when less demonstrations are given, the per- formance
Design
Average
Ours w/o SE(3)
24:1.4488
satisfactory. These challenges stem from the inherent complexity of its diffusion process design, where maintaining equivariance in each Markov transi- tion adds substantial difficulty to the learning task.
Ours w/o Eqv-Diff Ours
57::6.52
76-422224
In contrast, ET-SEED consistently outperforms across all six tasks, with minimal performance drop when facing unseen object poses. This advantage is especially pronounced when using limited number of demonstrations, showcasing ET-SEED's superior data efficiency, manipulation profi- ciency, and spatial generalization ability.
Ablation Studies. We conduct ablation studies on the New Pose (NP) scenario of the representative Opening Door task to evaluate the effectiveness of different components of our approach:
Ours w/o SE(3): Our method without SE(() invariance and equivariance in the backbone architecture. In this variant, use standard PointNet+ to predict noise at cach step.
Ours w/o Eqv-Diff: Our method without the SF(() equivariant denoising process. In-
stead, we use a non-equivariant diffusion process (DDIM), following the approach of Ze et (024)..
Table shows quantitative comparisons with ablations. Clearly each component improves our method's capability.

---

Published as a conference paper at ICLR 2025
5.2
EXPERIMENT
Open Bottle Cap
Open Door
Fold Garment
Calligraphy
Figure 4: Visualizations of the real-world environments used in our experiments. The tasks performed using multiple Microsoft Azure Kinect cameras and Intel@ RealSense for point cloud fusion and Franka robotic arm for execution.
Setup. We test the performance of our model on four tasks on real scenarios. All the tasks are visualized in Figure We use Segment Anything Model 2 (SAM2) (Ravi et al.) 2024) segment object from the scene and project the segmented image with depth to point cloud. Please refer to appendix and our website for more details and videos of real-world manipulations.
Expert demonstrations are collected by human tele-operation. The Franka arm and the gripper are teleoperated by the keyboard. Since tasks contain more than one stage and include two robots and various objects, making the process of demonstration collection very time-consuming, we only provide 20 demonstrations for each task.
In test setting, We place the object at 10 different positions with different poses that are unseen in the training data. Each position is evaluated with one trial.
Results. Results for our real robot tasks given in Table Con- sistent with our simulation findings, in real world experiments, ET-SEED performs better than baselines in all the four tasks, given only 20 demon- strations. The evaluation shows the effectiveness and spatial generaliza- tion ability of our method.
Table 4: Success rates in real-world robot experiments.
Melbol
Open Bottle Cap
Open Duur
Callignahh
Fold Gamment
DF3
0.2
0.0
DP3+Aug
0.2
0.3
0,0
0.2
EquiBet
0.5
0.0
CONCLUSION
In this paper, we propose ET-SEED, an efficient trajectory-level SE(() equivariant diffusion policy. Our method enhances both data efficiency and spatial generalization while reducing the training complexity typically encountered in diffusion-based methods. Through theoretical extensions of equivariant Markov kernels, we demonstrated that the SE(3) equivariant diffusion process can be achieved given a weaker condition, significantly simplifying the learning task. Experimental results on diverse robotic manipulation tasks show that ET-SEED performs better than SOTA methods. Real-world experiments further validate the generalization ability of our model to unseen object poses with only demonstrations, ET-SEED a novel approach for data efficient and generalizable imitation learning, paving the way for more capable and adaptive robots in real-world applications. However, the proposed method has certain limitations. 1) The equivariance ET-SEED is designed for object point clouds, so, pre-processing is required to extract segmented object point clouds from the scene point clouds. 2) The action space is defined as the 6D poses of the end-effectors, and assume the end-effectors controllers. Additionally, this action space is not able to tackle dexterous manipulation tasks as they need action space with higher dimension.
reach any feasible 6D pose by inverse kinematics solvers and
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
We gratefully acknowledge the financial support provided by the National Youth Talent Support Program Igrant number 8200800081] I] and the National Natural Science Foundation of China Igrant number 62376006]. This funding has been instrumental in enabling our research and the successful completion of this work.
REFERENCES
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Ri-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068/ 2022.
Alper Canberk, Cheng Chi, Huy Ha, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran for multi-purposs garment manipulation, 2022.
Song. Cloth funnels: Canonicalized-
Yue Chen, Chenrui Tie, Ruihai Wu, and Hao Dong. Eqvafford: Se (3) equivariance for point-level affordance learning. arXiv preprint arXiv:2408.nm 2400003000.,,53, 2024.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for so equivvarin networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 12200-12209, 2021.
Ethan Eade. Lie groups for 2d and 3d transformations. URL http://ethamaade. com/tie, pdf, revised Dec, 117:118, 2013.
Zipeng Fu, Tony Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body telcoperation. arXiv preprint arXiv:2401.022 2024.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se transfforrs:mr:s: 3d roto- translation equivariant attention networks. Advances in neural information processing systems, 33:1970-1981, 2020.
Chongkai Gao, Zhengrong Xue, Shuying Deng, Tianhai Liang, Siqi Yang. Lin Shao, and Huazhe Xu. Riemann: Near real-time squuiaaquia robot manipulation without point cloud segmentation, 2024. AtLLattts///axxttaa wrg./brgggaa/..a. http:/aaab//233114//
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equiv- ariant diffusion for target-aware molecule generation and affinity prediction. arXiv preprint arXiv:2303. 033543 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Boce Hu, Xupeng Zhu, Dian Wang, Zihao Dong, Haojic Huang, Chenghao Wang, Robin Walters, and Robert Platt. Orbitgrasp: se(3) -equivarrrr grasp learning, arXiv preprint arXiv:2407.03531, 2024.
Haobo Jiang, Mathieu Salzmann, Zheng Dang, Jin Xie, and Jian Yang. Se (3) diffusion model-based point cloud registration for robust 6d object pose estimation. Advances Neural Information Processing Systems, 36, 2024.
Adam Leach, Sebastian M Schmon, Matteo T. Degiacomi, and Chris G. Willcocks. Denoising diffuuson probabilistic models on SO(3) for rotational alignment. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022. htLhttp:: oppperevii......
Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic 1990, /1900, 2022.
graphs. arXiv preprint arXiv:2206.

---

Published as a conference paper at ICLR 2025
Byeongdo Lim, Jongmin Kim, Jihwan Kim, Yonghyeon Lee, and Frank C Park. Equigraspflow: Se (3)-euuvvrr 6-dof grasp pose generative flows. In Sth Annual Conference on Robot Learning, 2024.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
Teli Ma, Jiaming Zhou, Zifan Wang, Ronghe Qiu, and Junwei Liang. Contrastive imitation learning for language-guided multi-task robotic manipulation. arXiv preprint arXiv:2406.0m 097388 2024.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline 2021.
human demonstrations for robot manipulation. arXiv preprint
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao-Yuse Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.0077 2024. URL https::////. //tpii///aaaiv.///a arxiv. org. abs 24t../408.0000 00714
Hyunwoo Ryu, Jiwoo Kim, Hyunseok An, Junwoo Chang, Joohwan Seo, Taehan Kim, Yubin Kim, Chaewon Hwang, Jongeun Choi, and Roberto Horowitz. Diffusion-ede Bi-equivariant denoising generative modeling
(3) for visual robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18007-18018, 2024.
Gianluca Scarpellini, Stefano Fiorini, Francesco Giuliari, Pietro Morerio, and Alessio Del Bue. Diffassemble: A unified graph-diffusion model for 2d and 3d reassembly. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2024.
Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blun- dell, Pietro Lió, Carla Gomes, Max Welling, et al. Structure-basee based drug design with equivariant diffusion models. arXiv preprint arXiv:2210. 13695, 2022.
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se -3)-evurrri object rep- resentations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2394-6400 IEEE, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-22662222 2265. PMLR, 2015.
Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki. Se (3)-diffusionnieels:: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp- 5923-5930. IEEE, 2023.
Dian Wang, Stephen Hart, David Surovik, Tarik Kelestemur, Haojie Huang, Haibo Zhao, Mark Yeatman, Jiuguang Wang, Robin Walters, and Robert Platt. Equivariant diffusion policy, 2024.
Yuanfei Wang, Xiaojie Zhang, Ruihai Wu, Yu Li, Yan Shen, Mingdong Wu, Zhaofeng He, Yizhou Wang, and Hao Dong. Adamanip: Adaptive articulated object manipulation environments and policy learning. In International Conference on Learning Representations, 2025. URL https: 7 //openreview. net. http//orrmmmii dtw.rmmmmm.d..ssssssasaaa
Ruihai Wu, Chenrui Tie, Yushi Du, Yan Zhao, and Hao Dong. Leveraging se (3) equivariance for learning 3d geometric shape assembly. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14311-14320, 2023.
Minkai Xu, Lantao Yu, Yang Song. Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geo- metric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923, 2022.
12

---

Published as a conference paper at ICLR 2025
Han Xue, Yutong Li, Wenqiang Xu, Huanyu Li, Dongzhe Zheng. and Cewu Lu. Unifolding: To- wards sumple-efficient scalable, and generalizable robotic garment folding, 2023a.
Zhengrong Xue, Zhecheng Yuan, Jiashun Wang, Xueqian Wang, Yang Gao, and Huazhe Xu. Useek: Unsupervised se (3)-eqivvariant 3d keypoints for generalizable manipulation. In 2023 IEEE In- lernational Conference on Robolics and Automation (ICRA), pp. 1715-1722. IEEE, 2023b.
Jingyun Yang. Zi ang Cao, Congyue Deng, Rika Antonova, Shuran Song, and Jeannette Bohg. Equibot: Sim(3) equivaaaqu diffusion policy for generalizable and data efficient learning. arXiv preprint wrXV:22407.01479 2024a.
Jingyun Yang. Congyue Deng, Jimmy Wu, Rika Antonova, Leonidas Guibas, and Jeannette Bohg. Equivact: Sim -q)-vquiar visuomotor policies beyond rigid object manipulation. 2024 IEEE International Conference Roboties and Automation (ICRA), pp. 9249 9255. IEEE, 2024b.
Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Jodilyn Peralta, Brian Ichter, al. Scaling robot learning with semantically imag- ined experience. arXiv preprint arXiv:2302.11550 2023.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint arXiv:2403.03954, 2024.
Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. In Conference on Robot Learning, pp. 1199-1210. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2025
A SE(3) GROUP AND se(3) ALGEBRA
In this section, we briefly introduce the SE(3) Lie group and se(3) Lie algebra. to Eade (013) for more detailed explaination.
One can refer
A.1 BASIC TERMS
An element in SE(3) can be represented as a 4 x 4 matrix
R SO(3), t
R
R
(12)
C 0
This representation means we can compute the composition and inversion of elements in SE(3) by matrix multiplication and inversion.
An element 5 = se(3) can be represented by multiples of the generators
(u.w) w) E2
(13)
u is the translation, u E R2
is the rotation (exactly the axis-angle representation, its normal is the rotation angle, and its
direction is the rotation axis), w E R
There's s 1-1 map between SF((3 and se(3)
In(C)
(14)
C exp(()
A.2 INTERPOLATION
two elements a, b G, we would like to interpolate between the two elements according to parameter / E [0. 1), define an interpolation function
I: G:GxGRR+G
(15)
First define group element that tasks a to b
d: a
(16)
Compute the corresponding Lie algebra vector and scale it by / d(t) In(d); de ex)(d(())
f(a, b,t) = de
(17)
A.3 GAUSSIAN DISTRIBUTION
Consider Lie group G and its Lie algebra vector space with & DoF. A mean transformation / E G and covariance matrix Σ e kk We can sample an element from the Gaussian distribution on G
8 ~ N(0; () E g) I exp(o) 4
(18)
c.f. Gaussian in R
1 (x
T mm}
N(x 14, Σ)
exp
(19)
D/2 1/1
2
14

---

Published as a conference paper at ICLR 2025
B PROOF OF PROPOSITION 1
1( = / plat 1c)p(x" - 1 c)da
K = f p(a* (c)
II p(a c)dx
Am1
= -f /D(a* (c) p 1x"h cc)pp(=" c)( II p(x 1x, ,c)ddx
k=
(20)
= =s p(a (Tc)]
Pr (x 12 Tc))pp Tc)
kmn n-2
Pa (T 1T Te)]
= (Te))
1 p(T 2=" 2 Tc))
= (Te))
C PROOF PROPOSITION 2
Here, for simplicity, we define two operations. 1) For matrix T and a list of matrix A [H,,... H the notation TA means multiplying on each H,, i.e., TA i.e.TA , THN]: 2) For two lists of matrix A = [ ..... Hy], B [G,,... Gy., the notation AB means multiplying the matrices at the corresponding positions. i.e., = H, GI...,,HyC. HyG
As shown in 4 and 5 our backbones are theoretically SE) equivariant and invariant, i.c.
(0, k) = E AR;
(21)
T'equio Ah; k) Eeguiv AT0,
We firstly use Eiv to denoise K - 1 steps, so for any 1 < k S K, the denoise iteration satisfies Å A (O, 1b;
(22)
When the input observation is transformed by SE(() element, for the property of Einns have VTESE(3) A En (TO,A*;))
(23)
means the predicted noise Á keeps invariant no matter what SF:(3) transformation is applied on the input observation. And then we carry each element of the predicted noise sequence into I0
it's obvious that H is also SE(3) invariant for any < i < Tр. So we can infer that H is S SE3) invariant to input observation. In terms of Markov transition, the first K - transitions are P1 -ike..
(H H O) p(H H 1 < K
(24)
For the last denoising iteration, we use SE(3) equivariant model predict noise, so when the input observation is transformed, we have
VTESES)) I- = Enguin (TO,
(25)
Carry each element the result into the last denoise step H = A H. will discover the final denoised action sequence o is SE(3) Equivariant. another word, the last Markov transition is Pi-like.
p H H. O) = pTH H TO)
(26)
Additionally, as the initial noisy action H is sampled from Gaussian distribution, it's not condi- tioned on the observation.
p pH k [0) = p( (TO)
(27)
Combine 24
26
27
together and put them back into 20 we find the whole diffusion process for (E)3 equivariant. Joint all the H O < < Tp) into sequence A it's easy
single action to verify proposition 2 holds. In other word, our predicted action sequence is theoretically (3) equivariant to input observations.
15

---

Published as a conference paper at ICLR 2025
D EXPERIMENTS SHOWING OUR PROPOSED MARKOV PROCESS IS EASIER TO LEARN
The properties of three different Markov transitions can be descrihed
plyz, c) pi (v/r, Tc)
plyz, c) = p2(Tylz,Te) ply]z,c) Ps (T,
(28)
In practice, the SE(3) Transformer (Fuchs et al. 2020) with different input and output feature types to approximate the three types of transitions Denoted as PINet, P2Net and P3Net).
D.1 SINGLE- STEP EVALUATION
In this validation experiment, we take a point cloud P with random orientation as observation (fo- cusing solely rotation for simplicity). The detailed input and output feature types are shown table 5. According to the features of SE(33 Transformer, it's easy to verify the networks satisfy the corresponding equivariant properties.
Table 5: Input and Output Feature Types
Input Feature
Output Feature
Supervision
Final Loss
PINet P2Net
type-0
9 type-0
identity matrix
0.0002
3 type-0
3 type-1
Pose of input pts
0.25
P3Net
1 type-1
3 type-1
Pose input pts
0.27
For all three networks, the input feature consists of 3 scalar values attached to each point, and the output feature consists of 9 scalar values (after pooling across all points). For PINet, the output is set as nine type-0 features, meaning the output remains invariant to the rotation of the input point cloud. We supervise the output by computing the L2 loss between it and fixed rotation matrix. In contrast, for P2Net and P3Net, the output is treated as three type-1 features, which are supervised using the pose of the input point cloud. The only difference among the three networks is the input and output feature types, while all other hyperparameters remain the same.
Comparison of Three Models
PlNet
1.2
P2Net
P3Net
Loss
0.8
0.6
0.4
0.2
Train Epochs
Figure 5: Loss curve of PINet, P2Net and P3Net. After only several gradient descent, the loss of PINet converges almost to 0, while the losses of P2Net and P3Net do not decrease obviously.
After training for the same number of epochs, the loss curve of the three networks is shown in fig. 5 The experiments demonstrate that the invariant model (P (PINet) is significantly easier to train compared to the equivariant models (P2Net and P3Net), as it is expected to output the same values regardless of the transformation applied to the input point cloud. Additionally, observe that the use of higher-type features in P2Net and P3Net results in increased memory requirements and longer inference times.
16

---

Published as a conference paper at ICLR 2025
D.2 MULTI-STEP EVALUATION
Moreover, to further demonstrate the superiority of our proposed diffusion process, we compose PINet, P2Net and P3Net different ways and evaluate the performance. In this experiment, we compare two diffusion processes, both are end-to-end equívariant to the condition c, but have differ- ent kinds of transitions. Same as the single-step experiment, we take a point cloud P with random orientation observation (fo- cusing solely rotation for simplicity.
For the first process, all transitions are pı-like. So we use one P3Net to approximate the transition function. The input features of this P3Net consists of 3 type-1 features size with rotation matrix), and one type-O scaler indicating the index of current denoising step. And the output of the P3Net is also 3 type-1 features. So the first diffusion process be written
Fk-1 P3Net(P, Iy, for S K
(29)
And for the second process, the first K denoising steps are P1 Pı-like, and the last is p2-like. So here we use one PINet and one P2Net to approximate the transitions. The input of the PINet is 10 type-O features, representing the rotation matrix, representing the index of denoising step, and the output is 9 type-0 features representing the rotation matrix. The input of the P2Net is 9 type-0 features representing the rotation matrix, as we only use it in the last step, we don't 't input the step index into it. And the P2Net outputs 3 type- features. Formally, the diffusion process can be written as
Lk PINet(P, vk, for 2 SkSS
(30)
(2,
(31)
According the properties of (3) Transformer, it's easy to verify the two diffusion processes are both equivariant to the input point cloud P. We generate same number of diffusion data pieces {P, K::) } and separately train the models in two diffusion process. The supervision is added on the
output of each transition step. To compare the performance of the two diffusion process, we show the final loss [(xbred: age)) of the two process with respect to the training epoch.
Comparison of Two diffusion process
120
Inv+Eqv
100
Pure Eqv
80
Loss
80
20
2
4
8
Train Epochs
Figure 6: Loss curve of two diffusion process. After only several gradient descent, the loss of Inv, process onverges much faster than the Pure Eqv. process
As shown in fig. 6 the Inv.+Eqv. process converges much faster than the Pure Eqv. process.
With the two experiments, we demonstrate our choice of using K P1 pi-kike denoising steps and only one like step is better than using K Pi-like steps.
E IMPLEMENTATION OF SE(3) EQUIVARIANT AND INVARIANT BACKBONES
Here we introduce the implementation of our true SE backbone Ein (F3 Transformer (Fuchs et al. 2020
equivariant backbone Eeguiv and invariant
17

---

Published as a conference paper at ICLR 2025
In general, each module consists of SE(3) Transformers, called pos_ner and ori.net, outputing translation and rotation separately. As the output SE(3) Transformer is per-point features, we implement mean pooling all points to get global features.
E. INVARIANT MODULE
We set the output of ori_ett 6 Tр typeO features, and then implement Schimidt orthogonalization to get rotation matrix. As the 6 x Tp type0 features are (3) invariant, the rotation matrix is also invariant.
We set the output of pos_ner as 3 x T type0 features, which is naturally invariant to any SE(3) transformation of the point cloud, guaranteed by the translation invariance of SE(3) Transformer.
Finally we combine each translation and rotation parts to a 4 x 4 matrix. So we get Tp of 4 x matrices, and all of them are invariant any SE(3) transformations of input point cloud.
E.2 EQUIVARIANT MODULE
We set the output of ori net 2 x T, typel features, and then implement Schimidt orthogonalization on each typel features to get Tp of rotation matrices. As the x Tp type1 features are SE(3) equivariant, the rotation matrices
also equivariant.
We set the output of poer as 2 x T, typel feature and x T, typeO feature(denoted as offsser). First each 2 typel features, get rotation matrix (denoted
implement Schimidt orthogonalization as R?). Additionally, we denote the mass center of the input point cloud as M := N D.N the coordinate of the i-th point. Then we can write each the predicted translations' as
-i=1 æ is
t(X) = M R- offiet
(32)
We can prove this translation vector is equivariant any SF(( transformation of the input point- cloud X. When the input point cloud is transformed, X' = Ratata + Idata-
1(x') = (RdataM + (data) - Raara R offset
RRtata (M R offset) Edata
(33)
R(X) + Idata
Finally we combine each translation and rotation parts to a 4 x 4 matrix. So we get Tp of 4 x matrices, and all of them are equivariant any SE(3) transformations of input point cloud.
F EQUIVARIANT DIFFUSION ON SE(3) MANIFOLD
The noisy action of step &s can represented as
H = Exp (vv1 (vvI-aa=)
F
H.
(34)
Perturbation
Interpolation
The first term, Exp(7/I- ā is a random noise on SE(3) manifold, aiming to randomize the dif- fusion process. According to the Gaussian distribution SE(3) (appendix(A.3) SE(3) Gaussian variable can be written as the Exp of a Gaussian variable on se(3) So we first randomly sample a 6D noise e E g from unit Gaussian distribution, then scale by a scheduler factor > âe to control the magnitude of the perturbation at different steps. Finally we use the Exp map to convert the variable back to SE(3).
The second term is interpolation SE(3) manifold between H and H. The idea behind this function is, first project the SE(3) transformation to sc(3). perform linear interpolation in this tangent space, and then convert the interpolated vector back to SFF(3 obtain the interpolated transformation. One can refer to Jiang et al. (2024) for more details. Formally, the interpolation funcion J can be expressed
F (Vāi Ho, HH]
Exp ((1 Vor) -log (HH²')) Ho
(35)
18

---

Published as a conference paper at ICLR 2025
G SIMULATION TASKS- FURTHER DETAILS
Rotate Triangle: A robotic arm with 2D anchor pushes the triangle to a target 6D pose. The task reward is computed as the percentage of the Triangle shape that overlaps with the target Triangle pose.
Open Bottle Cap: A bottle with a cap is placed at a random position in Workspace, and robot arm is tasked with opening the cap. In this task, the demonstrations show robots Unserewing bottle cap with parallel gripper. Success in this task depends on whether the bottle cap is successfully opened. Note that, due to simulator constraints, opening the bottle cap simply involves lifting it upward without the need to twist it first.
This task evaluates the manipulation of articulated objects. The model is re- quired to generate trajectories to open doors positioned at various orientations. The demon- stration is given as: We initialize the gripper at point p sampled on the handle of the door and set the forward orientation along the negative direction of the surface normal p. And then we pull the door by a degree. Different from door pushing, we perform grasping at contact point p for pulling. Success in this task is determined by the opening angle of the door.
Open Door:
Robot Calligraphy: This long-horizon task involves using a robot arm to write complex Chinese characters paper, accounting for different orientations. Success in this task is determined by the aesthetic quality and accuracy of the characters or patterns formed, which should closely resemble the target trajectory.
Fold Garment: A long-horizon task involving deformable object manipulation, where robot folds long-sleeved garment. The robot folds the sleeves inward along the garment's central axis, then gathers the lower edge of the garment and folds it upward, aligning it with the underarm region. A folding succeeds when the Intersection over- -over-Uiion (IOU) between
the target and the folded garments exceeds bar (Xue et al. 2023a Canberk et al. 2022
Fling Garment: A dual-arm task for manipulating deformable objects. The robot grasps the two shoulder sections of a wrinkled dress, lifting to allow the fabric clear of the surface. Then flings the garment it to flatten the fabric, and then places it back onto flat surface. Success is determined by the projection area the flattened garment.
Some of the six tasks are exactly SE) equivariant, and some are partially. In the Open Door task, the manipulation trajectory is exactly equivariant with the point cloud of the door. In the Robot Calligraphy task, the manipulation trajectory is exactly equivariant with the point cloud of the paper and the handwriting that has been written. In the Open Bottle Cap task, the manipulation trajectory is exactly equivariant with the point cloud of the bottle. In the Rotate Triangle task, as we always add same transformation on initial pose and target pose of the triangle, the manipulation trajectory is exactly equivariant with the point cloud of the triangle. In the Fling Garment and Fold Garment tasks, the trajectories are not exactly equivariant with the initial observation of the garment, as the deformation differs in each initialization. But even so, our method can also outperform baselines.
The intuition is, by ensuring end-to-end SE(3) equivariance, our model treats point cloud in the observation space and all the point clouds possible by S3( transformation as an equivalence class, so the observation space is reduced the quotient space of the observation space over the S SF()) group. Once the S3()) equivariance is naturally ensured, the network can focus on the geo- metric features of the object, so still has the ability to generalize the deformation and geometry of objects.
H REAL TASKS- FURTHER DETAILS
In the task of Open Bottle Cap, unlike in simulator, the process in the real world involves first twisting the cap and then lifting off to open. The bottle is initially placed at a random position on the table. For Opening Door, the initial position of the cabinet is randomly determined. In Calligraphy, we use bristldd brushes and watercolor paper, which is randomly positioned on the table. In Cloth Folding, the limited workspace of the Franka robot means it cannot reach every possible location. Therefore, the placement of clothes is not random but is instead based on locations accessible the robot.
19


Reviews:
[{"review_id": "5CxfqLpmAI", "reviewer": "Reviewer_fnNr", "summary": "This paper presents a framework for learning SE(3) equivariant policies with diffusion models. The training and inference of the diffusion model is based on invariance and the SE(3) equivariance properties. Both simulation and real-world experiments are conducted and show promising results on four robotic tasks.", "questions": "- What is exactly the format of observation? Is it RGB images? Or colored 3D point clouds? How did the authors guarantee the transformation of input observation T is a simple transformation so that they can verify the equivariance of the policy with T? \n- In Algorithm 1, I didn't see how the equivariance and invariance are included anywhere. Does it mean that equivariance are only guaranteed because of inference?\n- How are $E_{equiv}$ and $E_{inv}$ defined?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The framework presented in this paper is interesting. Incorporating equivariance property in diffusion policy makes sense.\n- The presentation of the paper is good.\n- The experiments conducted in this paper is thorough and the results are convincing.", "weaknesses": "- Not sure if including too much math background in the appendix is appropriate. Most content in Sections A and B, if not all of, is unrelated to this paper and should be deleted. Please don't expect that including seemingly sophisticated math in the paper can make the paper look awesome superficially and thus get better rating scores from reviewers. I actually lowered my score because of these unrelated math being included.\n- Some math notations are not defined. For example, $E_{equiv}$ and $E_{inv}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HJJNXwfR3X", "reviewer": "Reviewer_t1h8", "summary": "This paper combines score-based modeling and geometric deep learning for robot policy learning. The results are evaluated in both simulated and real-world robot environments. The authors claim that the proposed approach attains stronger performance than existing baselines.", "questions": "Please see my comments in the weakness section. Thanks.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Real-world experiments are valuable. I appreciate the authors' efforts to conduct real-world validations of their approach\n- Baseline comparisons add value. I appreciate the authors' efforts to compare against 3D representation policy learning works and probabilistic modeling of multi-modal trajectories such as EquiBot.", "weaknesses": "**A. Presentation**.\n\nThe presentation of the paper needs a lot of care in both writing and figures.\n\nThere are currently too many display items in the teaser figure, some of which distract the readers from the fruit of the proposed technique. If the authors aim to convince the audience of probabilistic modeling of trajectories and geometric learning, then the figure should focus on those two things. Showing the performance graph directly in the first figure does not clarify that point. I would refer the authors to the teaser figures in diffusion policy and vector neurons, both of which are cited by the authors. Currently, the teaser figure does not strongly convey how trajectories are equivariant and how multi-modality is captured by score-based modeling. Perhaps part of the Figure 2 design can be used to improve this.\n\nA few minor points: I am also unsure if you need almost ten lines of text to clarify the definitions of equivariant and invariant functions (L162-172). I'm not sure whether Algorithm and Figure 3 need to be taking that much space. Perhaps you can make more space there and add my suggested experiments.\n\n**B. Missing critical experiments, novelty concerns**. \n\nI am currently unconvinced and cannot fully assess the performance of the score-based modeling and geometric modeling. If the goal of this work is to showcase the capabilities of combining score-based probabilistic modeling and geometric learning, then I think the two strongest values would be (1) handling multi-modality in the supervision signals for imitation learning, as shown in many works such as diffusion policy, and (2) equivariant representations. If these are the authors' aims, these two points must be substantiated more deeply with visualizations. \n\nI cannot find a single visualization of the generated trajectory in the main manuscript that showcases multi-modality. I cannot find a visualization that showcases equivariant representation either. These two points are not substantiated. \n\nThis is deeply related to the novelty of the work. By just reporting on the criterion of success rate, it is truly difficult to understand what caused the performance gaps between the prior works and this one. The dynamical process is abstracted away into a single scalar evaluation criterion. I need more convincing visualizations and results to understand and assess this work. The leading performance reported by the authors could be due to a bunch of things, and whether it is related to the incorporation of probabilistic and geometric modeling is hard to assess.\n\n\n**C. Implicit Assumptions, evaluation criteria**.\n\nThere seem to be several assumptions in this work that I encourage the authors to clarify, perhaps in a limitation section. For example, there seem to be assumptions about perception and policy representation. The perception part needs to be a point cloud and a specified coordinate system by experts. The policy part assumes 6 DOF end-effector representation and the availability of an inverse kinematics/dynamics controller. These assumptions are non-trivial when it comes to more dexterous tasks. Consider spinning a pen with a robot hand. Do we expect the commands of the robot hand to be specified in task space, which would be highly difficult for high-frequency control, and what is the notion of equivariance there? \n\nThe presented tasks seem to be primarily trajectory generations in a single Euclidean frame. If this is true, it needs to be specified in the paper, perhaps a limitation section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G73raTwjK8", "reviewer": "Reviewer_eYAU", "summary": "This paper proposes a new method for SE(3) equivariant diffusion policy. The key is a proved theorem that in the denoising process, only the last denoising step needs to be equivariant, and all previous denoising steps can be invariant. A new diffusion policy is designed based on this results, where the first K-1 denoising steps employ a invariant transformer, and the last one employs a equivariant transformer. Results on 6 simulation tasks show the proposed method, ET-SEED, performs better than existing baselines, especially on new poses not seen during training.", "questions": "see weakness section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is overall very clearly written.\n- The proposed method is straightforward in implementation yet very effective. \n- The experiments results show the strong performance of the proposed method.", "weaknesses": "- Overall I think this is a good paper and do not have major weaknesses. One minor comment: for figure 3, the block for Inv. SE(3) transformer and Eqv. SE(3) transformer are identical, which seems like a typo?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vEDtbujdsT", "reviewer": "Reviewer_kmzf", "summary": "The paper proposes a novel decomposition of an SE(3)-equivariant Markovian denoising process, classifying each denoising step into $p_1$, $p_2$, and $p_3$-type equivariance. In contrast to previous approaches that utilize $p_3$-type equivariance in which the denoising is equivariant to both the context input $c$ and the noised sample $x^{k}$, the authors demonstrate that equivariant denoising can be achieved through a combination of $p_1$ and $p_2$-type equivariance. Furthermore, the authors argue that training $p_1$ models is significantly easier than training $p_2$ and $p_3$ models, as $p_1$ denoising is invariant to $c$, whereas $p_2$ and $p_3$ denoising are equivariant to it. Consequently, the proposed ET-SEED model employs the invariant $p_1$ for all denoising steps except the final step, which uses $p_2$.", "questions": "Suggestion: Why did you use the SE(3)-transformer? More advanced models, like Equiformer v2, with improved efficiency and scalability are available nowadays.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "### **Strength 1. Novelty**\nThe paper introduces a novel approach to the SE(3)-equivariant pose/trajectory generation problem. Unlike previous works, ET-SEED achieves full equivariance by first generating the trajectory independently of the reference frame in which the context input (e.g., point cloud) is observed, then subsequently transforming it equivariantly with respective to the context frame. Results in Appendix E indicate potential advantages of this approach, though this conclusion should be interpreted with caution, as the experiment in Appendix E is limited to a single timestep and does not account for potential interactions across different timesteps.\n\nOverall, the paper offers valuable new insights into defining equivariance condition for diffusion models, making it worth sharing with the community regardless of its limitations.\n\n### **Strength 2. Experimental Results**\nThe authors present a benchmark comparison against two state-of-the-art methods (DP3 and EquiBot) across six simulated and four real-world tasks. ET-SEED outperforms these methods by a substantial margin in most settings, clearly demonstrating the benefits of this approach. It is particularly exciting to observe robots learning to perform complex, trajectory-level real-world tasks with only 20 demonstrations.", "weaknesses": "### **Weakness 1. No interdependence between each keypose**\nThe primary limitation of the proposed ET-SEED is that each keypose is denoised independently, with no interaction term between them. Consequently, this is not a true trajectory-level generative model. Joint modeling of all keyposes within a trajectory is indispensable to recent successes in trajectory-level generative models such as diffusion policy and action chunking transformer. Assuming each keypose is independent from the others is unrealistic. \n\nFor example, if the robot needs to open a bottle cap and a drawer, it could start by grasping either the bottle cap or the drawer handle. However, once it has selected one action, say, grasping the bottle cap, the subsequent step should be opening the bottle cap rather than the drawer. While this issue could be addressed in a closed-loop manner by putting the current state as input, this approach is less suitable from an open-loop planning perspective.\n\nI am willing to raise my score if this is my misunderstanding. Otherwise, I cannot give a very high score to the soundness of the method.\n\n\n### **Weakness 2. Experiment in Appendix E**\nThe experiments in Appendix E provide a compelling reason for preferring $p_1$-type over $p_3$-type denoising. However, the comparison is limited, as each model is evaluated only with a single-timestep kernel, ignoring potential positive interactions that might arise across timesteps with $p_3$-type denoising.\n\n\n### **Weakness 3. Requirement for Segmentation**\nMany recent equivariant robotic manipulation models exhibit strong robustness to unsegmented inputs. In contrast, ET-SEED relies on object segmentation. While segmenting target objects is not particularly difficult these days due to open-vocab segmentation models like SAM, this requirement restricts ET-SEED’s applicability to object-centric tasks. With segmentation, the model is unable to manipulate non-object entities (e.g., a pile of tiny objects as a whole) or understand global scene context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)