ET-SEED: EFFICIENT TRAJECTORY-LEVEL SE(3) EQUIVARIANT DIFFUSION POLICY

## Abstract

Abstract Imitation learning, e.g. , diffusion policy, has been proven effective in various robotic manipulation tasks.
However, extensive demonstrations are required for policy robustness and generalization.
To reduce the demonstration reliance, we leverage spatial symmetry and propose ET-SEED, an efficient trajectory-level S ⁢ E ⁢ ( 3 ) 𝑆 𝐸 3 SE(3)\ italic_S italic_E ( 3 ) equivariant diffusion model for generating action sequences in complex robot manipulation tasks.
Further, previous equivariant diffusion models require the per-step equivariance in the Markov process, making it difficult to learn policy under such strong constraints.
We theoretically extend equivariant Markov kernels and simplify the condition of equivariant diffusion process, thereby significantly improving training efficiency for trajectory-level S ⁢ E ⁢ ( 3 ) 𝑆 𝐸 3 SE(3)\ italic_S italic_E ( 3 ) equivariant diffusion policy in an end-to-end manner.
We evaluate ET-SEED on representative robotic manipulation tasks, involving rigid body, articulated and deformable object.
Experiments demonstrate superior data efficiency and manipulation proficiency of our proposed method,
as well as its ability to generalize to unseen configurations with only a few demonstrations. Website: https://et-seed.github.io/

## 1 Introduction

Imitation learning has achieved promising results for acquiring robot manipulation skills [, , ].
Though, one of the main challenges of imitation learning is that it requires extensive demonstrations to learn a robust manipulation policy [, , ].
Especially once the spatial pose of the object to be manipulated runs out of the demonstration distribution, the policy performance will easily decrease.
Although some works seek to tackle these issues through data augmentation [] or contrastive learning [], they usually require task-specific knowledge or extra training, and without theoretical guarantee of spatial generalization ability.

Another promising idea is to leverage symmetry.
Symmetry is ubiquitous in the physical world, and many manipulation tasks exhibit a specific type of symmetry known as SE(3) Equivariance.
$SE(3)\$is a group consisting of 3D rigid transformations.
For example, as shown in fig. 1(b), a real robot arm is required to write characters “ICLR” on a paper or fold a garment, when the pose of the paper or the garment changes, the manipulation trajectories of the end-effector should transform equivalently.
Employing such symmetries into policy learning can not only improve the data efficiency but also increase the spatial generalization ability.
Recent works on 3D manipulation have explored using SE(3) equivariance in the imitation learning process.
Most of these works focus on equivariant pose estimation of the target object or end-effector [, , ].
Trajectory-level imitation learning has achieved state-of-the-art performances on diverse manipulation tasks [, ].
By generating a whole manipulation trajectory, this kind of method is capable to tackle more complex manipulation task beyond pick-and-place.
For trajectory-level equivariance, Equivariant Diffusion Policy [] and Equibot [] propose equivariant diffusion process for robotic manipulation tasks.

However, previous trajectory-level diffusion models for robotic manipulation have two key limitations.
First, to maintain equivariance throughout the diffusion process, these models assume that every transition step must preserve equivariance.
As we will show in section 3.1, training neural networks with equivariance is more challenging than neural networks with invariance, requiring additional computational resources and leading to slower convergence.
This design constrains the model’s efficiency, making it hard for tackling complex long-horizon manipulation tasks.
Second, these models define the diffusion process in Euclidean space, which is not a natural definition, and limits the expressiveness.
Since the focus is on equivariant diffusion processes within the $SE(3)\$group, it is more natural to define both the diffused variables and the noise as elements of the $SE(3)\$group, which will lead to better convergence and multimodal distributions representation [].

In this work, we propose ET-SEED, a new trajectory-level $SE(3)\$equivariant diffusion model for manipulation tasks.
ET-SEED improves the sample efficiency and decreases the training difficulty by restricting the equivariant operations during the diffusion denoising process.
We extend the equivaraint Markov kernels theory and prove that during the full denoising process, at least only one equivariant transition is required.
Then, we integrate the diffusion process on $SE(3)\$manifold [] and $SE(3)\$transformers [] to design a new trajectory-level equivariant diffusion model on $SE(3)\$space.
In experiment, we evaluate our method on several common and representative manipulation tasks, as shown in fig. 1(c), including rigid body manipulation (rotate triangle, open bottle cap), articulated object manipulation (open door), long-horizon tasks (robot calligraphy), and deformable object manipulation (fold and fling garment).
Experiments show our method outperforms SOTA methods in terms of data efficiency, manipulation proficiency and spatial generalization ability (fig. 1(a)).
Further, in real-world experiments, with only 20 demonstration trajectories, our method is able to generalize to unseen scenarios.

In summary, our contributions are mainly as followed:

- •
We propose ET-SEED, an efficient trajectory-level $SE(3)\$equivariant diffusion policy defined on $SE(3)\$manifold, which achieves a proficient and generalizable manipulation policy with only a few demonstrations.
- •
We extend the theory of equivariant diffusion processes and derive a novel $SE(3)\$equivariant diffusion process, for simplified modeling and inference.
- •
We extensively evaluate our method on standard robot manipulation tasks in both simulation and real-world settings, demonstrating its data efficiency, manipulation proficiency, and spatial generalization ability, significantly outperforming baseline methods.

## 2 Related work

### 2.1 Leveraging Equivariance for Robotic Manipulation

Previous research has demonstrated that leveraging symmetry or equivariance in 3D Euclidean space can improve spatial generalization in a variety of robotic manipulation tasks.
Lim et al. [], Hu et al. [], Simeonov et al. [], Xue et al. [], Gao et al. [] proposed $SE(3)\$equivariant model for grasp pose prediction.
Other works have also leveraged this symmetry in tasks such as part assembly [, ], object manipulation on desktop  [], articulated and deformable object manipulation  [] and affordance learning  [].
Most of these studies either focus solely on generating a single 6D pose or fail to guarantee end-to-end equivariance across the entire $SE(3)\$space.
In this paper, our proposed method is capable of generating manipulation trajectories while theoretically maintaining end-to-end equivariance over the entire $SE(3)\$group.

### 2.2 Equivariant Diffusion Models

Diffusion models  [, ] compose a powerful family of generative models that have proven effective in robotic manipulation tasks  [].
Previous studies [, ] have investigated the effectiveness of combining spatial equivariance in the diffusion process to increase data efficiency and improve the spatial generalization ability of the model.
GeoDiff  [] gave a theoretical proof of $SE(3)\$equivariant Markov process.
Diffusion-EDFs  [] and Orbitgrasp  [] introduced $SE(3)\$equivariant diffusion processes for target grasp pose prediction, but lack the capability to generate entire manipulation trajectories.
Wang et al. [] proposed an equivariant diffusion policy capable of addressing $SO(2)\$equivariant tasks.
EquiBot  [] extended equivariant diffusion policies to $SIM(3)$ transformations, with the assumption that every transition step in the diffusion process is equivariant, which demands a high training cost.
We further discuss the conditions of $SE(3)\$equivariant diffusion process and prove that not each, but at least one equivariant step is required.
Based on this condition, we propose a novel $SE(3)\$equivariant diffusion model achieving better performance than previous works.

### 2.3 Diffusion on S ⁢ E ⁢ ( 3 ) 𝑆 𝐸 3 SE(3)\ italic_S italic_E ( 3 ) manifold

Most diffusion models define the diffusion process on pixel space [] or 3D Euclidean space [].
Leach et al. [] introduces a denoising diffusion model on $SO(3)$ group.
$SE(3)$-Diffusion Fields [] suggests that in 6-DoF grasp pose generation scenarios, formulating the diffusion process in $SE(3)\$manifold provides better coverage and representation of multimodal distributions, resulting in improved sample efficiency and performance.
Jiang et al. [] proposes a $SE(3)\$diffusion model for robust 6D object pose estimation.
In this work, we introduce an equivariant diffusion model on $SE(3)\$manifold for robot manipulation, revealing the superiority of defining equivariant diffusion process on $SE(3)\$over Euclidean space.

## 3 Method

Problem Formulation.
We formulate the problem as an imitation learning setting, aiming to learn a mapping from observation O to action sequence $A$, with some demonstrations from an expert policy. In our setting, the observation O is colored point clouds $\mathbf{P}=\{(x_{1},c_{1}),\cdot\cdot\cdot,(x_{N},c_{N})\}\in\mathbb{R}^{N
\times 6}$.
The action is defined directly as the desired 6D pose $\textbf{H}\in SE(3)$ of the end-effectors.
So in our setting, the action sequence means to the trajectory of end-effectors.
This experimental setup does not require additional input information, and the action definition is both intuitive and consistent with real robot control, making it applicable to a wide range of robotic manipulation tasks.

In this paper, we propose ET-SEED, a trajectory-level end-to-end $SE(3)\$equivariant diffusion model for robotic manipulation.
ET-SEED  can theoretically guarantee the output action is equivariant to any $SE(3)\$transformation applied on the input observation, while only involving one equivariant denoising step.
As shown in fig. 3 , given an observation and a noisy action sequence, our model first implement $K-1$ invariant denoising steps, and pass the result into the last equivariant denoising step to generate a $SE(3)\$equivariant denoised trajectory.

We will discuss equivariant Markov processes further to explain the correctness and advantages of our proposed diffusion process in section 3.1 , with only one denoising step $SE(3)\$equivariant and the rest $SE(3)\$invariant.
Then introduce our modified $SE(3)\$invariant and equivariant backbones in section 3.2 , and illustrate our $SE(3)\$equivariant diffusion process in section 3.3 .
Finally we prove end-to-end $SE(3)\$equivariance of our pipeline in section 3.4 .

### 3.1 Equivariant Markov Process

For a Markov process $x^{K:0}$, and any roto-translational transformation $T\in SE(3)$.
Geodiff [] shows that if the initial probabilistic distribution is $SE(3)\$invariant, $i.e.,p(x^{K})=p(Tx^{K})$, and the Markov transitions $p(x^{k-1}|x^{k})$ are $SE(3)\$equivariant for any $1\leq k\leq K$, $i.e.,p(x^{k-1}|x^{k})=p(Tx^{k-1}|Tx^{k})$, then the density of $x_{0}$ satisfies $p(x_{0})=p(Tx_{0})$.
Equibot [] adapts the theory and makes it more consistent with the robotics setting.
They involve an additional condition $c$ (can be seen as an observation) and show that if the initial distribution $p(x^{K}|c)$ and transitions are all equivariant, $i.e.,p(x^{K}|c)=p(Tx^{K}|Tc),p(x^{k-1}|x^{k},c)=p(Tx^{k-1}|Tx^{k},Tc)$
then the marginal distribution satisfies $p(x^{0}|c)=p(Tx^{0}|Tc)$.

In this paper, we discover that the condition of getting an equivariant marginal distribution $p(x^{0}|c)$ can be weaker.
Formally, we first define three Markov transitions with different properties.

$$ $\displaystyle p_{1}(x^{k-1}|x^{k},c)$ $\displaystyle=p_{1}(x^{k-1}|x^{k},Tc)$ (1) $\displaystyle p_{2}(x^{k-1}|x^{k},c)$ $\displaystyle=p_{2}(Tx^{k-1}|x^{k},Tc)$ $\displaystyle p_{3}(x^{k-1}|x^{k},c)$ $\displaystyle=p_{3}(Tx^{k-1}|Tx^{k},Tc)$ $$

Then we derive the marginal distribution using the three types of Markov transitions.
We have the following statement.
See appendix C for the detailed proof.

###### Proposition 1

Previous works  [, ] make all transitions $p_{3}$-like, which is a special case of proposition 1 .
In practice, we observe that training neural networks to approximate the properties of $p_{2}$ and $p_{3}$ is much more challenging compared to $p_{1}$, both in terms of final performance and training cost.
When the condition $c$ is transformed by a $SE(3)\$element, the distributions in $p_{2}$ and $p_{3}$ change equivalently, while the distribution in $p_{1}$ remains unchanged.
Learning to output an equivariant feature is clearly more challenging for neural networks than producing an invariant feature.
Additionally, in most of implementations of equivariant networks, building and training a model whose output is $SE(3)\$equivariant to the input takes up more computing resources than a $SE(3)\$invariant version.
We design experiments to validate these facts, with details in  appendix E .

In ET-SEED, we set the parameter $n=2$, meaning there are $K-1$ $p_{1}$-like transitions (referred to as “$SE(3)\$Invariant Denoising Steps”) and one $p_{2}$-like transition (referred to as the “$SE(3)\$Equivariant Denoising Step”). This key design choice significantly reduces the training complexity, thereby enhancing the overall performance of our method.

###### Proposition 1

### 3.2 S ⁢ E ⁢ ( 3 ) 𝑆 𝐸 3 SE(3)\ italic_S italic_E ( 3 ) equivariant backbone

In order to generate whole manipulation trajectories, it’s necessary that the network has the ability to output a translation vector at anywhere in the 3D space (even beyond the convex hull of the object), which can not be achieved by directly using existing equivariant backbones [, , ].
In this paper, based on $SE(3)\$Transformer [], we propose $SE(3)\$equivariant and invariant backbones suitable for predicting $SE(3)\$action sequences, which theoretically satisfy definition 1 and 2 .
The implementation details can be found at  appendix F .
The input of backbone is a set of points coordinates $\mathcal{X}\in\mathbb{R}^{N\times 3}$, with some type-0 features $D_{0}^{in}$ and type-1 features $D_{1}^{in}$ attached on each point.
Type-0 vectors are invariant under roto-translation transformations
and type-1 vectors rotate and translate according to $SE(3)\$transformation of point cloud.
The output is an element of $SE(3)\$, represented as a $4\times 4$ matrix.
For the $SE(3)\$equivariant model (denoted as $\mathcal{E}_{equiv}$), we have

$$ $\forall_{T\in SE(3)}:T\mathcal{E}_{equiv}(\mathcal{X};D_{0}^{in},D_{1}^{in})= \mathcal{E}_{equiv}(T\mathcal{X};D_{0}^{in},TD_{1}^{in})$ (2) $$

And for the $SE(3)\$invariant model (denoted as $\mathcal{E}_{inv}$), we have

$$ $\forall_{T\in SE(3)}:\mathcal{E}_{equiv}(\mathcal{X};D_{0}^{in},D_{1}^{in})= \mathcal{E}_{inv}(T\mathcal{X};D_{0}^{in},TD_{1}^{in})$ (3) $$

Figure: Algorithm 1 Training phase

Figure: Algorithm 2 Inference phase

### 3.3 S ⁢ E ⁢ ( 3 ) 𝑆 𝐸 3 SE(3)\ italic_S italic_E ( 3 ) Diffusion Models

Figure: Figure 2: Overview of our pipeline. A colored point cloud and a random sampled action sequence are first passed through $K-1$ $SE(3)\$invariant denoising steps and then a $SE(3)\$equivariant denoising step to generate a noise free action sequence.
Refer to caption: x2.png

Inspired by standard diffusion model, ET-SEED progressively disturbs the action $\textbf{H}^{0}$ into a noisy action $\textbf{H}^{K}$.
As standard diffusion process assume the final noisy variable $x_{T}$ follows the standard Gaussian distribution $\mathcal{N}(\textbf{0,\ I})$, we assume the noisy action $\textbf{H}^{K}$ follow a Gaussian distribution on $SE(3)\$, centered at the identity transformation $\mathbb{H}$. So we use an interpolation-based $SE(3)\$diffusion formula, which represent the $\textbf{H}^{k}\sim q(\textbf{H}^{k}|\textbf{H}^{0})$ at noise step $k(1\leq k\leq K)$ as

$$ $\mathbf{H}^{k}=\underbrace{\operatorname{Exp}\left(\gamma\sqrt{1-\bar{\alpha}_ {t}}\varepsilon\right)}_{\text{Perturbation }}\underbrace{\mathcal{F}\left( \sqrt{\bar{\alpha}_{t}};\mathbf{H}^{0},\mathbb{H}\right)}_{\text{Interpolation }},\varepsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ (4) $$

The interpolation function $\mathcal{F}\left(\sqrt{\bar{\alpha}_{t}};\mathbf{H}^{0},\mathbb{H}\right)$ is an intermediate transformation between the origin action $\mathbf{H}^{0}$ and the identity transformation $\mathbb{H}$. By adding a perturbation noise $\operatorname{Exp}\left(\gamma\sqrt{1-\bar{\alpha}_{t}}\varepsilon\right)$ on the intermediate transformation, we get a diffused action $\textbf{H}^{k}$.
The training process of ET-SEED  is shown in algorithm 1 .
More explanation about 4 can be found in Jiang et al. [] or appendix G.

This formulation is an analogy of DDPM [], which represent the noisy image as

$$ $x_{t}=\bar{\alpha}_{t}x_{0}+\bar{\beta}_{t}\bar{\varepsilon},\bar{\varepsilon} \sim\mathcal{N}(\mathbf{0},\mathbf{I})$ (5) $$

We can treat the first term of 5 as interpolation between $x_{0}$ and 0 0, second term as external noise.

The goal of $SE(3)\$reverse process is to train a denoising network, gradually refine the noisy action to the optimal ones.
In this paper, we propose a novel $SE(3)\$equivariant denoising process, which theoretically keeps the equivariance property while reducing the training difficulty.
As illustrated in fig. 2 , the input of reverse process is an observation $O$, an noisy action sequence $A^{K}=[\bf{H}_{0}^{K},\bf{H}_{1}^{K},...,\bf{H}_{T_{p}}^{K}]$, where $T_{p}$ is the action prediction horizon, and each $\bf{H}_{i}^{K}$ is drawn from a $SE(3)\$Gaussian distribution centered at identity transformation $\mathbb{H}$.
The denoising process forms a reverse Markov chain $A^{K}\rightarrow A^{K-1}\rightarrow\cdot\cdot\cdot\rightarrow A^{0}$. In our setting, we design a model to predict a probability distribution of the action sequence $A^{k-1}$ given current observation $O$ and the action sequence $A^{k}$ of the last denoising step. So the input of our denoising network $s_{\theta}$
consists of observation $O$, noisy action $\bf{H}_{i}^{k}$, and scalar condition $k,i$, outputs the predicted relative transformation between $\bf{H}_{i}^{k}$ and ground truth.
Formally, we have

$$ $\bf{\hat{H}}_{i}^{k\rightarrow 0}=s_{\theta}(O,\bf{H}_{i}^{k};k,i)$ (6) $$

To ensure the overall $SE(3)\$equivariance of our pipeline, we propose a novel design of denoising network $s_{\theta}$.
It is consists of one $SE(3)\$invariant backbone $\mathcal{E}_{inv}$ and one $SE(3)\$equivariant backbone $\mathcal{E}_{equiv}$. We use $\mathcal{E}_{inv}$ to denoise in the first $K-1$ iterations and use $\mathcal{E}_{equiv}$ to denoise in the last iteration.
Formally, our denoising network $s_{\theta}$ is

$$ $\begin{split}s_{\theta}(O,\bf{H}_{i}^{k};k,i)=\left\{\begin{matrix}\mathcal{E} _{inv}(O,\bf{H}_{i}^{k};k,i),k>1\\ \mathcal{E}_{equiv}(O,\bf{H}_{i}^{k};k,i),k=1\end{matrix}\right.\end{split}$ (7) $$

As illustrated in algorithm 2 , in the first $K-1$ denoising iteration, we use $SE(3)\$invariant backbone $\mathcal{E}_{inv}$ to predict noise, and implement a denoise step by

$$ $\mathbf{H}_{i}^{k-1}=\operatorname{Exp}(\lambda_{0}\operatorname{Log}(\hat{ \mathbf{H}}_{i}^{k\rightarrow 0}\mathbf{H}_{i}^{k})+\lambda_{1}\operatorname{ Log}(\mathbf{H}_{i}^{k})$ (8) $$

This formulation, by minimizing the KL divergence of the posterior distribution and prior distribution of $\bf{H}_{i}^{k-1}$, is able to infer a more reliable distribution for $\bf{H}_{i}^{k-1}$ [].

In the last denoising iteration, we use a $SE(3)\$equivariant backbone $\mathcal{E}_{equiv}$ to predict noise and directly apply the predicted transformation on current action.

$$ $\mathbf{H}_{i}^{0}=\mathbf{\hat{H}}_{i}^{1\rightarrow 0}\mathbf{H}_{i}^{k}$ (9) $$

### 3.4 Proof of end-to-end Equivariance

In this paper, we derive equivariant Markov process from a weaker condition, and build up a practically feasible framework.
In this part, we will prove that our proposed denoising process is theoretically $SE(3)\$equivariant.
When the input observation $O$ is transformed by any $SE(3)\$element $T$, the output denoised action sequence $A^{0}$ will be equivariantly transformed. Denote the denoising process as $A^{0}=ETSEED(A^{K};O)$, we have the following statement

###### Proposition 2

A detailed proof is attached in appendix D .
Here we briefly introduce the intuition.
In the first $K-1$ denoising steps, as we use $SE(3)\$invariant backbone to predict noise, the predicted noise $\hat{H}_{i}^{k\rightarrow 0}$ is $SE(3)\$invariant, thus the updated $H_{i}^{k-1}$ is also invariant. After the first $K-1$ steps, we get an invariant $H_{i}^{1}$. Because of the equivariance property of our last noise prediction backbone, the predicted noise of last step $\hat{H}_{i}^{1\rightarrow 0}$ is $SE(3)\$equivariant.
Carried into 9 , we get a $SE(3)\$equivariant $H^{0}_{i}$ for all $0\leq i\leq T_{p}$.
It means the final denoised action sequence $A^{0}$ is $SE(3)\$equivariant to the input observation $O$.

###### Proposition 2

## 4 Experiments

We systematically evaluate ET-SEED through both simulation and real-world experiments, aiming to address the following research questions:
(1) Does our method demonstrate superior spatial generalization compared to existing imitation learning approaches?
(2) Can our method achieve comparable performance with fewer demonstrations?
(3) Is our method applicable to real-world robotic manipulation tasks?

### 4.1 Simulation Experiments

Tasks. We design six representative robot manipulation tasks: Open Bottle Cap, Open Door, Rotate Triangle, Calligraphy, Cloth Folding, and Cloth Fling. These tasks encompass manipulation of rigid bodies, articulated bodies, and deformable objects, as well as dual-arm collaboration, long-horizon tasks, and complex manipulation scenarios. A brief overview is illustrated in  fig. 1.
For each task, we set up multiple cameras to capture full point clouds of the objects to be manipulated. We assume each robot manipulator operates within a complete 6DoF $SE(3)\$action space. Further expeirments details and discussions of their equivariant properties can be found in  appendix H .

Baselines. We compare our method against the following baselines:

- •
3D Diffusion Policy (DP3) []: A diffusion-based 3D visuomotor policy.
- •
3D Diffusion Policy with Data Augmentations (DP3+Aug): Same architecture as DP3, with $SE(3)\$data augmentation added.
- •
EquiBot []: A baseline combines SIM(3)-equivariant neural network architectures with diffusion policy.

DP3 and DP3+Aug are used to compare ET-SEED with baseline methods that utilize data augmentation to achieve spatial generalization,
while EquiBot allows for a comparison between different architectures of equivariant diffusion process.

Augmentations.
The DP3+Aug baseline utilizes augmentations during training. In all environments, training data is augmented by (1) rotating the observation along all three axes by random angles between $0^{\circ}$ and $90^{\circ}$, and (2) applying a random Gaussian offset to the observation. The standard deviation of the Gaussian noise is set to 10% of the workspace size.

Evaluation. Following the setup of Gao et al. [], we collect demonstrations and train our policy under the Training setting (T), subsequently testing the trained policy on both T and New Poses (NP), where target object poses undergo random $SE(3)\$transformations.
We evaluate all methods using two metrics, based on 20 evaluation rollouts, averaged over 5 random seeds.
Since we generate complete manipulation trajectories, the final success rate alone is inadequate for fully assessing the trajectory’s quality. We calculate the geodesic distance between each step of the predicted trajectory and the ground truth trajectory, providing a more comprehensive reflection of the trajectory’s overall quality. The geodesic distance between each step of the predicted trajectory and the ground truth trajectory, we can obtain a more accurate reflection of the trajectory’s overall quality. The definition of geodesic distance between $T,\hat{T}\in SE(3)$ is

$$ $\mathcal{D}_{geo}(\mathbf{T},\mathbf{\hat{T}})=\sqrt{\left\|\operatorname{Log} (\mathbf{R}^{\top}\mathbf{\hat{R}})\right\|^{2}+\left\|\mathbf{\hat{t}}- \mathbf{t}\right\|^{2}},$ (10) $$

where $R$ and $t$ are the rotation and translation parts of $T$.
We report $\mathcal{D}_{geo}$ in the same manner as success rates.

Results.
Table  2 and  3 provide a quantitative comparison between our method and the baseline. Both DP3 and its augmented variant demonstrate strong performance in the training setting (T), but they exhibit a significant drop in performance when faced with New Poses (NP) scenarios. This highlights that merely incorporating data augmentation is insufficient for the model to generalize effectively to unseen poses. Instead, leveraging equivariance proves essential for enhancing spatial generalization.

While EquiBot achieves commendable results in both success rate and $\mathcal{D}_{geo}$, it struggles with more complex, long-horizon tasks such as Calligraphy and Fold Garment.
Also, when less demonstrations are given, the performance is not satisfactory.
These challenges stem from the inherent complexity of its diffusion process design, where maintaining equivariance in each Markov transition adds substantial difficulty to the learning task.

In contrast, ET-SEED consistently outperforms across all six tasks, with minimal performance drop when facing unseen object poses.
This advantage is especially pronounced when using a limited number of demonstrations, showcasing T-SEED’s superior data efficiency, manipulation proficiency, and spatial generalization ability.

### 4.2 Real-Robot Experiment

Setup.
We test the performance of our model on four tasks on real scenarios. All the tasks are visualized in Figure 5 .
We use Segment Anything Model 2 (SAM2) [] to segment the object from the scene and project the segmented image with depth to point cloud.
Please refer to appendix I and our website for more details and videos of real-world manipulations.

Expert demonstrations are collected by human tele-operation. The Franka arm and the gripper are teleoperated by
the keyboard. Since our tasks contain more than one stage and include two robots and various objects, making the process of demonstration collection very time-consuming, we only provide 20 demonstrations for each task.

In test setting, We place the object at 10 different positions with different poses that are unseen in the training data. Each position is evaluated with one trial.

Results.
Results for our real robot tasks are given in Table 4.
Consistent with our simulation findings, in real world experiments, ET-SEED performs better than baselines in all the four tasks, given only 20 demonstrations. The evaluation shows the effectiveness and spatial generalization ability of our method.

## 5 Conclusion

In this paper, we propose ET-SEED, an efficient trajectory-level $SE(3)\$equivariant diffusion policy.
By leveraging $SE(3)\$symmetries, our method enhances both data efficiency and spatial generalization while reducing the training complexity typically encountered in diffusion-based methods.
Through theoretical extensions of equivariant Markov kernels, we demonstrated that the $SE(3)\$equivariant diffusion process can be achieved given a weaker condition, significantly simplifying the learning task.
Experimental results on diverse robotic manipulation tasks show that ET-SEED performs better than SOTA methods.
Real-world experiments further validate the generalization ability of our model to unseen object poses with only 20 demonstrations.
Our work presents a novel approach for data efficient and generalizable imitation learning, paving the way for more capable and adaptive robots in real-world applications.

#### Acknowledgments

If a paper is accepted, the final camera-ready version will (and probably should) include acknowledgments. All acknowledgments go at the end of the paper, including thanks to reviewers who gave useful comments, to colleagues who contributed to the ideas, and to funding agencies and corporate sponsors that provided financial support.

## Appendix A Preliminary Background

Figure: Figure 3: Left: Given a point cloud observation, the model outputs a trajectory of 6D pose. Right: For a noisy trajectory, an $SE(3)\$invariant denoise module generates $SE(3)\$invariant trajectories, followed by an $SE(3)\$equivariant module to produce noise free equivariant trajectories on the $SE(3)\$manifold. The model can generate corresponding trajectories even for unseen poses.
Refer to caption: x3.png

SE(3) Group and its Lie Algebra.
$SE(3)\$(Special Euclidian Group) is a group consisting of 3D rigid transformations.
Each $SE(3)\$transformation can be represented as a $4\times 4$ matrix (denoted as $T$), indicating linear transformation on homogeneous 4-vectors. Formally,
$T=\left[\begin{smallmatrix}R&\mathbf{t}\\
\mathbf{0}&1\end{smallmatrix}\right]$
, where $R\in\mathbb{R}^{3\times 3}$ is a rotation matrix, $\textbf{t}\in\mathbb{R}^{3}$ is a translation vector.
The Lie algebra $\mathfrak{se}(3)\$is a linear 6D vector space corresponding to the tangent space of $SE(3)\$.
Each element of $\mathfrak{se}(3)\$is a 6D vector
$\mathbf{\delta}\in\mathbb{R}^{6}$. The mutual mapping between $SE(3)\$and $\mathfrak{se}(3)\$is achieved by the logarithm map $\operatorname{Log}:SE(3)\rightarrow\mathbb{R}^{6}$ and the exponential map $\operatorname{Exp}:\mathbb{R}^{6}\rightarrow SE(3)$.
More information about $SE(3)\$and $\mathfrak{se}(3)\$can be found in appendix B .

$SE(3)\$ Equivariant Function.
Generally, we call a function $f:\mathcal{X}\rightarrow\mathcal{Y}$ that maps elements from input space $\mathcal{X}$ to output space $\mathcal{Y}$ is equivariant to a group $G$
if there are group representations of $G$ on $\mathcal{X}$ and $\mathcal{Y}$ respectively denoted by $\rho^{\mathcal{X}}$ and $\rho^{\mathcal{Y}}$ such that
$\forall_{g\in G}:\rho^{\mathcal{Y}}(g)\circ f=f\circ\rho^{\mathcal{X}}(g)$

In other words, the function $f$ commutes with representations of the group $G$. In this paper, we focus on $SE(3)\$group, and represent elements of $SE(3)\$as $4\times 4$ matrixes.
As special cases of general equivariance, we can define $SE(3)\$equivariant and invariant functions as:

###### Definition 1

###### Definition 2

$SE(3)\$Equivariant Trajectory.
In many robotic manipulation tasks, the trajectories of the manipulator show a certain symmetry.
If the representation of a trajectory under certain coordinate frame is $SE(3)\$invariant, we call the trajectory as $SE(3)\$Equivariant.
Formally, it can be defined as

###### Definition 3

This property means the trajectory is “attached” on a certain frame, and when the frame transforms, the trajectory transforms accordingly.
To demonstrate the universality of such symmetrical trajectories, we use a general example in fig. 3 .
For planes at different position and orientation, the representations of “turning right” trajectories appear identical in each plane’s body coordinate frame.
In our experiments, we select 6 representative manipulation tasks showing this symmetry.
Further discussion can be found in appendix H .

###### Definition 1

###### Definition 2

###### Definition 3

## Appendix B S ⁢ E ⁢ ( 3 ) 𝑆 𝐸 3 SE(3)\ italic_S italic_E ( 3 ) group and 𝔰 ⁢ 𝔢 ⁢ ( 3 ) 𝔰 𝔢 3 \mathfrak{se}(3)\ fraktur_s fraktur_e ( 3 ) algebra

### B.1 Basic Terms

An element in $SE(3)$ can be represented as a $4\times 4$ matrix

$$ $\begin{split}\mathbf{R}\in SO(3),t\in\mathbb{R}^{3}\\ C=\begin{pmatrix}\mathbf{R}&t\\ 0&1\end{pmatrix}\end{split}$ (13) $$

This representation means we can compute the composition and inversion of elements in $SE(3)$ by matrix multiplication and inversion.

An element $\delta=se(3)$ can be represented by multiples of the generators

$$ $\delta=(\mathbf{u,\omega})^{T}\in\mathbb{R}^{6}$ (14) $$

$\mathbf{u}$ is the translation, $\mathbf{u}\in\mathbb{R}^{3}$

$\omega$ is the rotation (exactly the axis-angle representation, its normal is the rotation angle, and its direction is the rotation axis), $\omega\in\mathbb{R}^{3}$

There’s a 1-1 map between $SE(3)$ and $se(3)$

$$ $\begin{split}\delta=ln(C)\\ C=exp(\delta)\end{split}$ (15) $$

### B.2 Adjoint

consider left-multiplication and right-multiplication in SE(3) group

$$ $C\cdot exp(\delta)=exp(Adj_{C}\cdot\delta)C$ (16) $$

$$ $Adj_{C}=\begin{pmatrix}\mathbf{R}&t_{\times}\mathbf{R}\\ \mathbf{0}&\mathbf{R}\end{pmatrix}$ (17) $$

c.f. Skew-symmetric matrix of a vector
$\alpha:=[a,b,c]^{T},\beta:=[l,m,n]^{T}$

$$ $\begin{split}\alpha\times\beta=\begin{bmatrix}bn-cm\\ cl-an\\ am-bl\end{bmatrix}=\begin{bmatrix}0&-c&b\\ c&0&-a\\ -b&a&0\end{bmatrix}\begin{bmatrix}l\\ m\\ n\end{bmatrix}\end{split}$ (18) $$

So we define $\alpha_{\times}([\alpha]_{\times})$ as the skew-symmetric matrix in 18

### B.3 Interpolation

two elements $a,b\in G$, we would like to interpolate between the two elements according to a parameter $t\in[0,1]$, define an interpolation function

$$ $f:G\times G\times\mathbb{R}\rightarrow G$ (19) $$

First define a group element that tasks $a$ to $b$

$$ $d:=b\cdot a^{-1}\\ d\cdot a=b$ (20) $$

Compute the corresponding Lie algebra vector and scale it by $t$

$\mathbf{d}(t)=t\cdot ln(d);d_{t}=exp(\mathbf{d}(t))$

$$ $f(a,b,t)=d_{t}\cdot a$ (21) $$

### B.4 Gaussian Distribution

Consider a Lie group $G$ and its Lie algebra vector space $\mathfrak{g}$, with $k$ DoF. A mean transformation $\mu\in G$ and a covariance matrix $\Sigma\in\mathbb{R}^{k\times k}$. We can sample an element from the Gaussian distribution on $G$

$$ $\begin{split}\delta\sim\mathcal{N}(0;\Sigma)(\delta\in\mathfrak{g})\\ x=exp(\delta)\cdot\mu\end{split}$ (22) $$

c.f. Gaussian in $\mathbb{R}^{D}$

$$ $N(\mathbf{x}\mid\mu,\boldsymbol{\Sigma})=\frac{1}{(2\pi)^{D/2}}\frac{1}{| \boldsymbol{\Sigma}|^{1/2}}\exp\left\{-\frac{1}{2}(\mathbf{x}-\mu)^{T} \boldsymbol{\Sigma}^{-1}(\mathbf{x}-\mu)\right\}$ (23) $$

## Appendix C Proof of proposition 1

$$ $\displaystyle p(x^{0}|c)$ $\displaystyle=\int p(x^{K}|c)p(x^{0:K-1}|x^{K},c)dx^{1:K}$ (24) $\displaystyle=\int p(x^{K}|c)\prod_{k=1}^{K}p(x^{k-1}|x^{k},c)dx^{1:K}$ $\displaystyle=\int p(x^{K}|c)(\prod_{k=n}^{K}p(x^{k-1}|x^{k},c))p(x^{n-2}|x^{n -1},c)(\prod_{i=1}^{n-2}p(x^{i-1}|x^{i},c))dx^{1:K}$ $\displaystyle=\int p(x^{K}|Tc)[\prod_{k=n}^{K}p_{1}(x^{k-1}|x^{k},Tc)]p_{2}(Tx ^{n-2}|x^{n-1},Tc)$ $\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ [\prod_{i=1}^{n -2}p_{3}(Tx^{i-1}|Tx^{i},Tc)]dx^{1:K}$ $\displaystyle=p(x^{n-1}|Tc)\ p(Tx^{n-2}|x^{n-1},Tc)\ p(Tx^{0}|Tx^{n-2},Tc)$ $\displaystyle=p(Tx^{0}|Tc)$ $$

## Appendix D Proof proposition 2

As shown in 2 and 3, our backbones are theoretically $SE(3)\$equivariant and invariant, i.e.

$$ $\begin{split}\mathcal{E}_{inv}(O,\mathbf{H}_{i}^{k};k,i)=\mathcal{E}_{inv}(TO, \mathbf{H}_{i}^{k};k,i)\\ T\mathcal{E}_{equiv}(O,\mathbf{H}_{i}^{k};k,i)=\mathcal{E}_{equiv}(TO,\mathbf{ H}_{i}^{k};k,i)\end{split}$ (25) $$

For each element $\mathbf{H}_{i}^{K}$ in $A^{K}$, we firstly use $\mathcal{E}_{inv}$ to denoise $K-1$ steps, so for any $1<k\leq K$, the denoise iteration satisfies

$$ $\mathbf{\hat{H}}_{i}^{k\rightarrow 0}=\mathcal{E}_{inv}(O,\mathbf{H}_{i}^{k};k ,i)$ (26) $$

When the input observation is transformed by $SE(3)\$element, for the property of $\mathcal{E}_{inv}$, we have

$$ $\forall_{T\in SE(3)}:\mathbf{\hat{H}}_{i}^{k\rightarrow 0}=\mathcal{E}_{inv}( TO,\mathbf{H}_{i}^{k};k,i)$ (27) $$

It means the predicted noise $\mathbf{\hat{H}}_{i}^{k\rightarrow 0}$ keeps invariant no matter what $SE(3)\$transformation is applied on the input observation. And then we carry the predicted noise into 8 , it’s obvious that $\mathbf{H}_{i}^{k-1}$ is also $SE(3)\$invariant.
So we can infer that $\mathbf{H}_{i}^{1}$ is $SE(3)\$invariant to input observation. In terms of Markov transition, the first $K-1$ transitions are $p_{1}$-like.

$$ $p(\mathbf{H}_{i}^{k-1}|\mathbf{H}_{i}^{k},O)=p(\mathbf{H}_{i}^{k-1}|\mathbf{H} _{i}^{k},TO),1<k\leq K$ (28) $$

For the last denoising iteration, we use a $SE(3)\$equivariant model to predict noise, so when the input observation is transformed, we have

$$ $\forall_{T\in SE(3)}:T\mathbf{\hat{H}}_{i}^{1\rightarrow 0}=\mathcal{E}_{equiv }(TO,\mathbf{H}_{i}^{1};1,i)$ (29) $$

Carry the result into 9, we will discover the final denoised action $\mathbf{\hat{H}}_{i}^{0}$ is $SE(3)\$Equivariant.
In another word, the last Markov transition is $p_{2}$-like.

$$ $p(\mathbf{H}_{i}^{0}|\mathbf{H}_{i}^{1},O)=p(T\mathbf{H}_{i}^{0}|\mathbf{H}_{i }^{1},TO)$ (30) $$

Additionally, as the initial noisy action $\mathbf{H}_{i}^{K}$ is sampled from Gaussian distribution, it’s not conditioned on the observation.

$$ $p(\mathbf{H}_{i}^{K}|O)=p(\mathbf{H}_{i}^{K}|TO)$ (31) $$

Combine 28, 30, 31 together and put them back into 24, we find the whole diffusion process for single action is $SE(3)\$equivariant.
Joint all the $\mathbf{H}_{i}^{0}(0\leq i\leq T_{p})$ into a sequence $A^{0}$, it’s easy to verify proposition 2 holds. In other word, our predicted action sequence is theoretically $SE(3)\$equivariant to input observations.

## Appendix E Experiments showing p 1 subscript 𝑝 1 p_{1} italic_p start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT -like transition is easier to learn

The properties of three different Markov transitions can be described as

$$ $\displaystyle p(y|x,c)$ $\displaystyle=p_{1}(y|x,Tc)$ (32) $\displaystyle p(y|x,c)$ $\displaystyle=p_{2}(Ty|x,Tc)$ $\displaystyle p(y|x,c)$ $\displaystyle=p_{3}(Ty|Tx,Tc)$ $$

In practice, we use the $SE(3)\$Transformer [] with different input and output feature types to approximate the three types of transitions(Denoted as P1Net, P2Net and P3Net). In this validation experiment, we take a point cloud $P$ with random orientation as observation (focusing solely on rotation for simplicity). The detailed input and output feature types are shown in table 1 .
According to the features of $SE(3)\$Transformer, it’s easy to verify the networks satisfy the corresponding equivariant properties.

**Table 1: Input and Output Feature Types**
|  | Input Feature | Output Feature | Supervision | Final Loss |
| --- | --- | --- | --- | --- |
| P1Net | 3 type-0 | 9 type-0 | identity matrix | 0.0002 |
| P2Net | 3 type-0 | 3 type-1 | Pose of input pts | 0.25 |
| P3Net | 1 type-1 | 3 type-1 | Pose of input pts | 0.27 |

For all three networks, the input feature consists of 3 scalar values attached to each point, and the output feature consists of 9 scalar values (after pooling across all points).
For P1Net, the output is set as nine type-0 features, meaning the output remains invariant to the rotation of the input point cloud. We supervise the output by computing the L2 loss between it and a fixed rotation matrix.
In contrast, for P2Net and P3Net, the output is treated as three type-1 features, which are supervised using the pose of the input point cloud.
The only difference among the three networks is the input and output feature types, while all other hyperparameters remain the same.

Figure: Figure 4: Loss curve of P1Net, P2Net and P3Net. After only several gradient descent, the loss of P1Net converges almost to 0, while the losses of P2Net and P3Net do not decrease obviously.
Refer to caption: x4.png

After training for the same number of epochs, the loss curve of the three networks is shown in  fig. 4.
The experiments demonstrate that the invariant model (P1Net) is significantly easier to train compared to the equivariant models (P2Net and P3Net), as it is expected to output the same values regardless of the transformation applied to the input point cloud.
Additionally, we observe that the use of higher-type features in P2Net and P3Net results in increased memory requirements and longer inference times.

Therefore, in ET-SEED, the diffusion process consists of $K-1$ $p_{1}$-like denoising steps and only one $p_{2}$-like step.

## Appendix F Implementation of S ⁢ E ⁢ ( 3 ) 𝑆 𝐸 3 SE(3)\ italic_S italic_E ( 3 ) equivariant and invariant backbones

Here we introduce the implementation of our true $SE(3)\$equivariant backbone $\mathcal{E}_{equiv}$ and invariant backbone $\mathcal{E}_{inv}$ $SE(3)\$Transformer [].

In general, each module consists of 2 $SE(3)\$Transformers, called as pos_net and ori_net, outputing translation and rotation separately. As the output of $SE(3)\$Transformer is per-point features, we implement a mean pooling over all points to get global features.

### F.1 Invariant module

We set the output of ori_net as 6 type0 features, and then implement Schimidt orthogonalization to get rotation matrix. As the 6 type0 features are SE(3) invariant, the rotation matrix is also invariant.

We set the output of pos_net as 3 type0 features, which is naturally invariant to any SE(3) transformation of the point cloud, guaranteed by the translation invariance of $SE(3)\$Transformer.

Finally we combine the translation and rotation parts to a $4\times 4$ matrix, and it is invariant to any $SE(3)\$transformations of input point cloud.

### F.2 Equivariant module

We set the output of ori_net as 2 type1 features, and then implement Schimidt orthogonalization to get rotation matrix. As the 2 type1 features are SE(3) equivariant, the rotation matrix is also equivariant.

We set the output of pos_net as 2 type1 feature and 3 type0 feature(denoted as offset). First we implement Schimidt orthogonalization on the 2 type1 feature, get a rotation matrix (denoted as $R$). Additionally, we denote the mass center of the input point cloud as $\mathcal{M}:=\frac{1}{N}\Sigma_{i=1}^{N}x_{i}$, $x_{i}$ is the coordinate of the $i-th$ point. Then we can write the predicted translation $t$ as

$$ $t(\mathcal{X})=\mathcal{M}+R\cdot\textit{offset}$ (33) $$

We can prove this translation vector is equivariant to any $SE(3)\$transformation of the input pointcloud $\mathcal{X}$. When the input point cloud is transformed, $\mathcal{X}^{\prime}=R_{data}\mathcal{X}+t_{data}$.

$$ $\begin{split}t^{\prime}(\mathcal{X})=(R_{data}\mathcal{M}+t_{data})+R_{data}R \cdot\textit{offset}\\ =R_{data}(\mathcal{M}+R\cdot\textit{offset})+t_{data}\\ =R_{data}t(\mathcal{X})+t_{data}\end{split}$ (34) $$

Finally we combine the translation and rotation parts to a $4\times 4$ matrix, and it is equivariant to any $SE(3)\$transformations of input point cloud.

## Appendix G Equivariant Diffusion on SE(3) manifold

The noisy action of step $k$ can be represented as

$$ $\mathbf{H}^{k}=\underbrace{\operatorname{Exp}\left(\gamma\sqrt{1-\bar{\alpha}_ {t}}\varepsilon\right)}_{\text{Perturbation }}\underbrace{\mathcal{F}\left( \sqrt{\bar{\alpha}_{t}};\mathbf{H}^{0},\mathbb{H}\right)}_{\text{Interpolation }},\varepsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ (35) $$

The first term, $\operatorname{Exp}(\gamma\sqrt{1-\bar{\alpha}_{t}}\varepsilon)$ is a random noise on $SE(3)\$manifold, aiming to randomize the diffusion process. According to the Gaussian distribution on $SE(3)\$(section B.4), a $SE(3)\$Gaussian variable can be written as the Exp of a Gaussian variable on $\mathfrak{se}(3)\$. So we first randomly sample a 6D noise $\varepsilon\in\mathbb{R}^{6}$ from unit Gaussian distribution, then scale it by a scheduler factor $\gamma\sqrt{1-\hat{\alpha}_{t}}$ to control the magnitude of the perturbation at different steps. Finally we use the Exp map to convert the variable back to $SE(3)\$.

The second term is an interpolation on $SE(3)\$manifold between $\mathbf{H}^{0}$ and $\mathbb{H}$. The idea behind this function is, first project the $SE(3)\$transformation to $\mathfrak{se}(3)\$, perform linear interpolation in this tangent space, and then convert the interpolated vector back to $SE(3)\$to obtain the interpolated transformation. One can refer to Jiang et al. [] for more details. Formally, the interpolation funcion $\mathcal{F}$ can be expressed as

$$ $\mathcal{F}\left(\sqrt{\bar{\alpha}_{t}};\mathbf{H}_{0},\mathbb{H}\right)= \operatorname{Exp}\left(\left(1-\sqrt{\bar{\alpha}_{t}}\right)\cdot\log\left( \mathbb{H}\mathbf{H}_{0}^{-1}\right)\right)\mathbf{H}_{0}$ (36) $$

## Appendix H Simulation Experiments– Further Details

**Table 2: Success rates ($\uparrow$) and standard deviation of different tasks in simulation.**
|  | Open Bottle Cap | Open Door | Rotate Triangle |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | T | NP | T | NP | T | NP |  |  |  |  |  |  |
| Method | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 |
| DP3  [] | 65±4.5 | 76±5.5 | 11±4.2 | 14±6.5 | 61±2.24 | 72±2.74 | 9±3.54 | 16±5.48 | 67±2.74 | 89±2.24 | 5±2.24 | 10±2.74 |
| DP3+Aug | 35.0±5.0 | 44±4.2 | 38±4.47 | 46±7.42 | 43±8.37 | 54±6.52 | 30±4.18 | 40±8.22 | 35±3.54 | 42±4.47 | 32±5.70 | 41±4.18 |
| EquiBot  [] | 63±2.74 | 73±2.74 | 63±5.70 | 77±7.58 | 56±2.24 | 72±2.24 | 58±7.58 | 77±7.58 | 67±2.74 | 84±2.24 | 64±8.66 | 86±5.48 |
| ET-SEED(Ours) | 67±2.74 | 81±2.24 | 74±6.52 | 82±2.74 | 66±2.24 | 75±2.74 | 66±2.74 | 76±2.24 | 83±2.24 | 93±2.74 | 85±2.24 | 89±4.18 |
|  | Calligraphy | Fold Garment | Fling Garment |  |  |  |  |  |  |  |  |  |
|  | T | NP | T | NP | T | NP |  |  |  |  |  |  |
| Method | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 |
| DP3  [] | 28±2.74 | 50±3.54 | 0±0.00 | 3±2.74 | 44±2.24 | 60±4.18 | 4±5.48 | 8±4.48 | 36±5.48 | 67±4.48 | 4±5.48 | 9±8.22 |
| DP3+Aug | 8±2.74 | 21±4.18 | 3±2.24 | 12±11.51 | 13±5.70 | 27±7.58 | 17±10.37 | 31±9.62 | 28±7.58 | 38±4.48 | 11±4.18 | 31±2.24 |
| EquiBot  [] | 24±5.48 | 43±8.37 | 14±10.84 | 40±10.61 | 34±4.18 | 58±2.74 | 33±2.74 | 60±7.90 | 35±6.12 | 61±6.52 | 36±6.52 | 64±8.22 |
| ET-SEED(Ours) | 38±2.74 | 55±3.54 | 36±6.52 | 54±8.22 | 47±2.74 | 67±2.74 | 49±2.24 | 69±4.18 | 50±5.00 | 67±4.48 | 48±4.47 | 62±5.70 |

**Table 3: SE(3) Geodesic distances ($\downarrow$) of different tasks in simulation.**
|  | Open bottle cap | Open Door | Rotate Triangle |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | T | NP | T | NP | T | NP |  |  |  |  |  |  |
| Method | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 |
| DP3  [] | 0.257 | 0.197 | 1.413 | 1.785 | 0.384 | 0.354 | 0.478 | 0.442 | 0.265 | 0.192 | 1.812 | 1.627 |
| DP3+Aug | 0.283 | 0.234 | 0.276 | 0.218 | 0.391 | 0.315 | 0.442 | 0.329 | 0.247 | 0.187 | 0.578 | 0.447 |
| EquiBot  [] | 0.194 | 0.151 | 0.197 | 0.170 | 0.241 | 0.224 | 0.266 | 0.228 | 0.197 | 0.107 | 0.214 | 0.099 |
| ET-SEED (Ours) | 0.133 | 0.114 | 0.127 | 0.124 | 0.127 | 0.101 | 0.121 | 0.128 | 0.098 | 0.082 | 0.104 | 0.087 |
|  | Calligraphy | Fold Garment | Fling Garment |  |  |  |  |  |  |  |  |  |
|  | T | NP | T | NP | T | NP |  |  |  |  |  |  |
| Method | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 | 25 | 50 |
| DP3  [] | 0.305 | 0.241 | 4.988 | 4.662 | 0.479 | 0.298 | 4.466 | 4.179 | 0.529 | 0.348 | 4.993 | 4.365 |
| DP3+Aug | 0.354 | 0.337 | 4.752 | 4.365 | 1.318 | 0.976 | 1.524 | 1.219 | 1.318 | 0.976 | 1.524 | 1.219 |
| EquiBot  [] | 0.291 | 0.117 | 0.282 | 0.129 | 0.368 | 0.293 | 0.387 | 0.288 | 0.418 | 0.343 | 0.437 | 0.338 |
| ET-SEED (Ours) | 0.124 | 0.083 | 0.121 | 0.089 | 0.299 | 0.149 | 0.287 | 0.136 | 0.349 | 0.179 | 0.337 | 0.186 |

### H.1 Task Setting:

- •
Rotate Triangle:
A robotic arm with 2D anchor pushes the triangle to a target 6D pose.
The task reward is computed as the percentage of the Triangle shape that overlaps with the target Triangle pose.
- •
Open Bottle Cap: A bottle with a cap is placed at a random position in Workspace, and a robot arm is tasked with opening the cap.
In this task, the demonstrations show robots Unscrewing bottle cap with parallel gripper. Success in this task depends on whether the bottle cap is successfully opened. Note that, due to simulator constraints, opening the bottle cap simply involves lifting it upward without the need to twist it first.
- •
Open Door: This task evaluates the manipulation of articulated objects.
The model is required to generate trajectories to open doors positioned at various orientations.
The demonstration is given as:
We initialize the gripper at a point $p$ sampled on the handle of the door and set the forward orientation along the negative direction of the surface normal at $p$. And then we pull the door by a degree. Different from door pushing, we perform a grasping at contact point $p$ for pulling.
Success in this task is determined by the opening angle of the door.
- •
Robot Calligraphy: This long-horizon task involves using a robot arm to write complex Chinese characters on paper, accounting for different orientations.
Success in this task is determined by the aesthetic quality and accuracy of the characters or patterns formed, which should closely resemble the target trajectory.
- •
Fold Garment: A long-horizon task involving deformable object manipulation, where a robot folds a long-sleeved garment.
The robot folds the sleeves inward along the garment’s central axis, then gathers the lower edge of the garment and folds it upward, aligning it with the underarm region.
A folding succeeds when the Intersection-over-Union (IOU) between the target and the folded garments exceeds a bar [, , , ].
- •
Fling Garment: A dual-arm task for manipulating deformable objects.
The robot grasps the two shoulder sections of a wrinkled dress, lifting it to allow the fabric is clear of the surface. Then flings the garment it to flatten the fabric, and then places it back onto a flat surface.
Success is determined by the projection area of the flattened garment.

Some of the six tasks are exactly $SE(3)\$equivariant, and some are partially.
In the Open Door task, the manipulation trajectory is exactly equivariant with the point cloud of the door.
In the Robot Calligraphy task, the manipulation trajectory is exactly equivariant with the point cloud of the paper and the handwriting that has been written.
In the Open Bottle Cap task, the manipulation trajectory is exactly equivariant with the point cloud of the bottle.
In the Rotate Triangle task, as we always add same transformation on initial pose and target pose of the triangle, the manipulation trajectory is exactly equivariant with the point cloud of the triangle. In the Fling Garment and Fold Garment tasks, the trajectories are not exactly equivariant with the initial observation of the garment, as the deformation differs in each initialization. But even so, our method can also outperform baselines.

The intuition is, by ensuring end-to-end $SE(3)\$equivariance, our model treats a point cloud in the observation space and all the point clouds possible by $SE(3)\$transformation as an equivalence class, so the observation space is reduced to the quotient space of the observation space over the $SE(3)\$group.
Once the $SE(3)\$equivariance is naturally ensured, the network can focus on the geometric features of the object, so it still has the ability to generalize to the deformation and geometry of objects.

### H.2 Qualitative Results.

Table  2 and  3 provide a quantitative comparison between our method and the baseline. Both DP3 and its augmented variant demonstrate strong performance in the training setting (T), but they exhibit a significant drop in performance when faced with New Poses (NP) scenarios. This highlights that merely incorporating data augmentation is insufficient for the model to generalize effectively to unseen poses. Instead, leveraging equivariance proves essential for enhancing spatial generalization.

While EquiBot achieves commendable results in both success rate and $\mathcal{D}_{geo}$, it struggles with more complex, long-horizon tasks such as Calligraphy and Fold Garment.
Also, when less demonstrations are given, the performance is not satisfactory.
These challenges stem from the inherent complexity of its diffusion process design, where maintaining equivariance in each Markov transition adds substantial difficulty to the learning task.

In contrast, ET-SEED consistently outperforms across all six tasks, with minimal performance drop when facing unseen object poses.
This advantage is especially pronounced when using a limited number of demonstrations, showcasing T-SEED’s superior data efficiency, manipulation proficiency, and spatial generalization ability.

## Appendix I Real World Experiments– Further Details

Figure: Figure 5: Visualizations of the real-world environments used in our experiments. The tasks are performed using multiple Microsoft Azure Kinect cameras and Intel® RealSense for point cloud fusion and a Panda robotic arm for execution.
Refer to caption: x5.png

**Table 4: Success rates in real-world robot experiments.**
| Method | Open Bottle Cap | Open Door | Calligraphy | Fold Garment |
| --- | --- | --- | --- | --- |
| DP3 | 0.2 | 0.2 | 0.0 | 0.1 |
| DP3+Aug | 0.2 | 0.3 | 0.0 | 0.2 |
| EquiBot | 0.6 | 0.5 | 0.0 | 0.3 |
| ET-SEED (Ours) | 0.8 | 0.6 | 0.4 | 0.6 |

Task Description. In the task of Open Bottle Cap, unlike in a simulator, the process in the real world involves first twisting the cap and then lifting it off to open. The bottle is initially placed at a random position on the table.
For Opening Door, the initial position of the cabinet is randomly determined.
In Calligraphy, we use flat-bristled brushes and watercolor paper, which is randomly positioned on the table.
In Cloth Folding, the limited workspace of the Franka robot means it cannot reach every possible location. Therefore, the placement of clothes is not random but is instead based on locations accessible to the robot.

Qualitative Results. Results for our real robot tasks are given in Table 4.
Consistent with our simulation findings, in real world experiments, ET-SEED performs better than baselines in all the four tasks, given only 20 demonstrations. The evaluation shows the effectiveness and spatial generalization ability of our method.

## Appendix J Abalation Study

**Table 5: Ablation studies.**
| Design | Average |
| --- | --- |
| Ours w/o SE(3) | 24±4.48 |
| Ours w/o Eqv-Diff | 57±6.52 |
| Ours | 76±2.24 |

Ablation Studies. We conduct ablation studies on the New Pose (NP) scenario of the representative Opening Door task to evaluate the effectiveness of different components of our approach:

- •
Ours w/o SE(3): Our method without $SE(3)\$invariance and equivariance in the backbone architecture. In this variant, we use a standard PointNet++ to predict noise at each step.
- •
Ours w/o Eqv-Diff: Our method without the $SE(3)\$equivariant denoising process. Instead, we use a non-equivariant diffusion process (DDIM), following the approach of  Ze et al. [].

Table 5 shows quantitative comparisons with ablations. Clearly each component improves our method’s capability.

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 6
presentation: 4
questions: - What is exactly the format of observation? Is it RGB images? Or colored 3D point clouds? How did the authors guarantee the transformation of input observation T is a simple transformation so that they can verify the equivariance of the policy with T? 
- In Algorithm 1, I didn't see how the equivariance and invariance are included anywhere. Does it mean that equivariance are only guaranteed because of inference?
- How are $E_{equiv}$ and $E_{inv}$ defined?
soundness: 4
strengths: - The framework presented in this paper is interesting. Incorporating equivariance property in diffusion policy makes sense.
- The presentation of the paper is good.
- The experiments conducted in this paper is thorough and the results are convincing.
summary: This paper presents a framework for learning SE(3) equivariant policies with diffusion models. The training and inference of the diffusion model is based on invariance and the SE(3) equivariance properties. Both simulation and real-world experiments are conducted and show promising results on four robotic tasks.
weaknesses: - Not sure if including too much math background in the appendix is appropriate. Most content in Sections A and B, if not all of, is unrelated to this paper and should be deleted. Please don't expect that including seemingly sophisticated math in the paper can make the paper look awesome superficially and thus get better rating scores from reviewers. I actually lowered my score because of these unrelated math being included.
- Some math notations are not defined. For example, $E_{equiv}$ and $E_{inv}$.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: Please see my comments in the weakness section. Thanks.
soundness: 3
strengths: - Real-world experiments are valuable. I appreciate the authors' efforts to conduct real-world validations of their approach
- Baseline comparisons add value. I appreciate the authors' efforts to compare against 3D representation policy learning works and probabilistic modeling of multi-modal trajectories such as EquiBot.
summary: This paper combines score-based modeling and geometric deep learning for robot policy learning. The results are evaluated in both simulated and real-world robot environments. The authors claim that the proposed approach attains stronger performance than existing baselines.
weaknesses: **A. Presentation**.

The presentation of the paper needs a lot of care in both writing and figures.

There are currently too many display items in the teaser figure, some of which distract the readers from the fruit of the proposed technique. If the authors aim to convince the audience of probabilistic modeling of trajectories and geometric learning, then the figure should focus on those two things. Showing the performance graph directly in the first figure does not clarify that point. I would refer the authors to the teaser figures in diffusion policy and vector neurons, both of which are cited by the authors. Currently, the teaser figure does not strongly convey how trajectories are equivariant and how multi-modality is captured by score-based modeling. Perhaps part of the Figure 2 design can be used to improve this.

A few minor points: I am also unsure if you need almost ten lines of text to clarify the definitions of equivariant and invariant functions (L162-172). I'm not sure whether Algorithm and Figure 3 need to be taking that much space. Perhaps you can make more space there and add my suggested experiments.

**B. Missing critical experiments, novelty concerns**. 

I am currently unconvinced and cannot fully assess the performance of the score-based modeling and geometric modeling. If the goal of this work is to showcase the capabilities of combining score-based probabilistic modeling and geometric learning, then I think the two strongest values would be (1) handling multi-modality in the supervision signals for imitation learning, as shown in many works such as diffusion policy, and (2) equivariant representations. If these are the authors' aims, these two points must be substantiated more deeply with visualizations. 

I cannot find a single visualization of the generated trajectory in the main manuscript that showcases multi-modality. I cannot find a visualization that showcases equivariant representation either. These two points are not substantiated. 

This is deeply related to the novelty of the work. By just reporting on the criterion of success rate, it is truly difficult to understand what caused the performance gaps between the prior works and this one. The dynamical process is abstracted away into a single scalar evaluation criterion. I need more convincing visualizations and results to understand and assess this work. The leading performance reported by the authors could be due to a bunch of things, and whether it is related to the incorporation of probabilistic and geometric modeling is hard to assess.


**C. Implicit Assumptions, evaluation criteria**.

There seem to be several assumptions in this work that I encourage the authors to clarify, perhaps in a limitation section. For example, there seem to be assumptions about perception and policy representation. The perception part needs to be a point cloud and a specified coordinate system by experts. The policy part assumes 6 DOF end-effector representation and the availability of an inverse kinematics/dynamics controller. These assumptions are non-trivial when it comes to more dexterous tasks. Consider spinning a pen with a robot hand. Do we expect the commands of the robot hand to be specified in task space, which would be highly difficult for high-frequency control, and what is the notion of equivariance there? 

The presented tasks seem to be primarily trajectory generations in a single Euclidean frame. If this is true, it needs to be specified in the paper, perhaps a limitation section.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: see weakness section
soundness: 3
strengths: - This paper is overall very clearly written.
- The proposed method is straightforward in implementation yet very effective. 
- The experiments results show the strong performance of the proposed method.
summary: This paper proposes a new method for SE(3) equivariant diffusion policy. The key is a proved theorem that in the denoising process, only the last denoising step needs to be equivariant, and all previous denoising steps can be invariant. A new diffusion policy is designed based on this results, where the first K-1 denoising steps employ a invariant transformer, and the last one employs a equivariant transformer. Results on 6 simulation tasks show the proposed method, ET-SEED, performs better than existing baselines, especially on new poses not seen during training.
weaknesses: - Overall I think this is a good paper and do not have major weaknesses. One minor comment: for figure 3, the block for Inv. SE(3) transformer and Eqv. SE(3) transformer are identical, which seems like a typo?

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Suggestion: Why did you use the SE(3)-transformer? More advanced models, like Equiformer v2, with improved efficiency and scalability are available nowadays.
soundness: 2
strengths: ### **Strength 1. Novelty**
The paper introduces a novel approach to the SE(3)-equivariant pose/trajectory generation problem. Unlike previous works, ET-SEED achieves full equivariance by first generating the trajectory independently of the reference frame in which the context input (e.g., point cloud) is observed, then subsequently transforming it equivariantly with respective to the context frame. Results in Appendix E indicate potential advantages of this approach, though this conclusion should be interpreted with caution, as the experiment in Appendix E is limited to a single timestep and does not account for potential interactions across different timesteps.

Overall, the paper offers valuable new insights into defining equivariance condition for diffusion models, making it worth sharing with the community regardless of its limitations.

### **Strength 2. Experimental Results**
The authors present a benchmark comparison against two state-of-the-art methods (DP3 and EquiBot) across six simulated and four real-world tasks. ET-SEED outperforms these methods by a substantial margin in most settings, clearly demonstrating the benefits of this approach. It is particularly exciting to observe robots learning to perform complex, trajectory-level real-world tasks with only 20 demonstrations.
summary: The paper proposes a novel decomposition of an SE(3)-equivariant Markovian denoising process, classifying each denoising step into $p_1$, $p_2$, and $p_3$-type equivariance. In contrast to previous approaches that utilize $p_3$-type equivariance in which the denoising is equivariant to both the context input $c$ and the noised sample $x^{k}$, the authors demonstrate that equivariant denoising can be achieved through a combination of $p_1$ and $p_2$-type equivariance. Furthermore, the authors argue that training $p_1$ models is significantly easier than training $p_2$ and $p_3$ models, as $p_1$ denoising is invariant to $c$, whereas $p_2$ and $p_3$ denoising are equivariant to it. Consequently, the proposed ET-SEED model employs the invariant $p_1$ for all denoising steps except the final step, which uses $p_2$.
weaknesses: ### **Weakness 1. No interdependence between each keypose**
The primary limitation of the proposed ET-SEED is that each keypose is denoised independently, with no interaction term between them. Consequently, this is not a true trajectory-level generative model. Joint modeling of all keyposes within a trajectory is indispensable to recent successes in trajectory-level generative models such as diffusion policy and action chunking transformer. Assuming each keypose is independent from the others is unrealistic. 

For example, if the robot needs to open a bottle cap and a drawer, it could start by grasping either the bottle cap or the drawer handle. However, once it has selected one action, say, grasping the bottle cap, the subsequent step should be opening the bottle cap rather than the drawer. While this issue could be addressed in a closed-loop manner by putting the current state as input, this approach is less suitable from an open-loop planning perspective.

I am willing to raise my score if this is my misunderstanding. Otherwise, I cannot give a very high score to the soundness of the method.


### **Weakness 2. Experiment in Appendix E**
The experiments in Appendix E provide a compelling reason for preferring $p_1$-type over $p_3$-type denoising. However, the comparison is limited, as each model is evaluated only with a single-timestep kernel, ignoring potential positive interactions that might arise across timesteps with $p_3$-type denoising.


### **Weakness 3. Requirement for Segmentation**
Many recent equivariant robotic manipulation models exhibit strong robustness to unsegmented inputs. In contrast, ET-SEED relies on object segmentation. While segmenting target objects is not particularly difficult these days due to open-vocab segmentation models like SAM, this requirement restricts ET-SEED’s applicability to object-centric tasks. With segmentation, the model is unable to manipulate non-object entities (e.g., a pile of tiny objects as a whole) or understand global scene context.

Meta-review: The paper presents an approach for equivariant policy learning based on diffusion policy. The motivation to transfer ideas from equivariant deep networks to policy learning is solid. The reviewers agree that the core ideas in the paper are quite interesting and the experiments well done. Particularly, the experiments on both simulation and real robots are quite convincing. There are some concerns in the evaluations and the limitation of needing segmentations. The rebuttal did a good job in addressing many of these concerns and has resulted in an overall positive opinion of this paper.

Decision: Accept (Poster)