3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
Authors: Tsung-Wei Ke
Nikolaos Gkanatsios
Katerina Fragkiadaki
Venue: CoRL 2024 (2024)
Keywords: Diffusion models
3D representations
manipulation
imitation learning

Abstract: Diffusion policies are conditional diffusion models that learn robot action distributions conditioned on the robot and environment state. They have recently shown to outperform both deterministic and alternative action distribution learning formulations. 3D robot policies use 3D scene feature representations aggregated from a single or multiple camera views using sensed depth. They have shown to generalize better than their 2D counterparts across camera viewpoints. We unify these two lines of work and present 3D Diffuser Actor , a neural policy equipped with a novel 3D denoising transformer that fuses information from the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories. 3D Diffuser Actor sets a new state-of-the-art on RLBench with an absolute performance gain of 18.1% over the current SOTA on a multi-view setup and an absolute gain of 13.1% on a single-view setup. On the CALVIN benchmark, it improves over the current SOTA by a 9% relative increase. It also learns to control a robot manipulator in the real world from a handful of demonstrations. Through thorough comparisons with the current SOTA policies and ablations of our model, we show 3D Diffuser Actor ’s design choices dramatically outperform 2D representations, regression and classification objectives, absolute attentions, and holistic non-tokenized 3D scene embeddings.

3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
Tsung- Wei Ke
Nikolaos Gkanatsios
Katerina Fragkiadaki
Carnegie Mellon University
{ (tsungwek, ngkanats, katef (cs. emu. edu 3d-diffuser- actor. github.io
Diffusion policies are conditional diffusion models that learn robot action distributions conditioned on the robot and environment state. They have recently shown outperform both deterministic and alternative action distribution learning formulations. 3D robot policies use 3D scene feature representations aggregated from single or multiple camera views using sensed depth. They have shown to generalize better than their 2D counterparts across camera viewpoints. We unify these two lines of work and present 3D Diffuser Actor, a neural policy equipped with a novel 3D denoising transformer that fuses information from the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories. 3D Diffuser Actor sets on RLBench with absolute performance gain of 18.1 1% over the current SOTA on a multi-view setup and an absolute gain of 13.1% % on single-view setup. On the CALVIN benchmark, it improves over the current SOTA by 9% relative increase. It also learns to control robot manipulator in the real world from a handful of demonstrations. Through thorough comparisons with the current SOTA policies and ablations of our model, we show 3D Diffuser Actor's design choices dramatically outperform 2D representations, regression and classification objectives, absolute attentions, and holistic non-tokenized 3D scene embeddings.
Abstract:
state-of-the-art
1 Introduction
Many robot manipulation tasks are inherently multimodal: at any point during task execution, there may be multiple actions which yield task-optimal behavior. Indeed, human demonstrations often contain diverse ways that a task can be accomplished. A natural choice is then to treat policy learning as a distribution learning problem: instead of representing a policy as a deterministic map no (x). learn the entire distribution of actions conditioned on the current robot state p(y|>>) [1, 2, 4].
Recent works use diffusion objectives for learning such state-conditioned action distributions for robot manipulation policies from demonstrations [5, 6, 7J. They outperform deterministic or other alternatives, such as variational autoencoders [8], mixture of Gaussians [9], combination of clas- sification and regression objectives [4], or energy-based objectives [10]. They typically use either low-dimensional (oracle) states [5] 2D images [6] as their scene representation.
At the same time, 3D robot policies build scene representations by "lifting" features from perspective views to 3D robot workspace based on sensed depth and camera extrinsics [11, 12, 13, 14, 15, 16]. They have shown to generalize better than 2D robot policies across camera viewpoints and to handle novel camera viewpoints at test time [15, 16). conjecture this improved performance comes from the fact that the visual scene tokens and the robot's actions interact in a common 3D space, that robust to camera viewpoints, while in 2D policies the neural network need to learn the 2D-to-3D mapping implicitly.
In this work, we marry diffusion for handling multimodality in action prediction with 3D scene representations for effective spatial reasoning. We propose 3D Diffuser Actor, a novel 3D denoising
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

policy transformer that takes as input a tokenized 3D scene representation, a language instruction and a noised end-effector's s future translation and rotation trajectory, and predicts the error in translations and rotations for the robot's 's end-effector. The model represents both the scene tokens and the end- effector locations in the same 3D space and fuses them with relative-position 3D attentions [17, 18), which achieves translation equivariance and helps generalization.
We test 3D Diffuser Actor in learning robot manipulation policies from demonstrations on the simula- tion benchmarks of RLBench [19] and CALVIN [20], as well as in the real world. 3D Diffuser Actor sets a new state-of-the-ard on RIBench with a 18. % absolute gain on multi-view setups and 13. % on single-view setups, outperforming existing 3D policies and 2D diffusion policies. On CALVIN, outperforms the current SOTA in the setting of zero-shot unseen scene generalization by a 9% relative gain. We further show 3D Diffuser Actor learn multi-task manipulation in the real world across 12 tasks from handful of real-world demonstrations, We empirically show that 3D Diffuser Actor outperforms all existing policy formulations, which either do not use 3D scene representations, or do not use action diffusion. We further compare against ablative versions of our model and show the importance of the 3D relative attentions.
The main contribution of this work is to combine 3D scene representations and diffusion objectives for learning robot policies from demonstrations. 3D robot policies have not yet been combined with diffusion objectives. An exception is ChainedDiffuser [21]. that uses diffusion models as drop-in replacement for motion planners, rather than manipulation policies, since it relies on other learning-bused policies (Act3D [15]) to supply the target 3D keypose to reach. We compare against ChainedDiffuser in our experiments and show we greatly outperform it.
Our contributions:
Concurrent work:
Concurrent to our effort, 3D diffusion policy [22] shares similar goal of combining 3D representations with diffusion objectives for learning manipulation from demon- strations. Though the two works share the same goal, they have very different architectures. Un- like 3D Diffuser Actor, the model of [22] does not condition on a tokenized 3D scene representa- tion but rather on holistic ID embedding pooled from the 3D scene point cloud. We compare 3D Diffuser Actor against [22] in our experiments and show it greatly outperforms it. We believe this because tokenized scene representations, used in 3D Diffuser Actor, are robust to scene changes: a part of the scene changes, only the corresponding subset of 3D scene tokens is affected. In contrast, holistic scene emheddings pooled across the scene always affected by any scene change. Thanks to this spatial disentanglement of the 3D scene tokenization, 3D Diffuser Actor generalizes better.
Code and videos of our manipulation results are available at https: / fidddddfffuer--ac github.ies ic/.
2 Related Work
Learning manipulation policies from demonstrations Earlier works learning from demonstra- tions train deterministic policies with behavior cloning [23, 24]. To better handle action multimodality, approaches discretize action dimensions and use cross entropy losses [25, 14, 11]. Generative ad- versarial networks [1, 2, 26]. variational autoencoders [8], combined Categorical and Gaussian distributions [4. 27, 28] and Energy- Based Models (EBMs) [10, 29, 30] have been used to learn from multimodal demonstrations. Diffusion models [31, 32] are a powerful class of generative models related EBMs that they model the score of the distribution, else, the gradient of the energy, as opposed the energy itself [33, 34]. The key idea behind diffusion models is to iteratively transform a simple prior distribution into target distribution by applying sequential denoising process. They have been used for modeling state- conditiondi action distributions in imitation learning [35, 36, 5, 37, 7, 38] from low-dimensional input, as well as from visual sensory input, and show both better mode coverage and higher fidelity in action prediction than alternatives.
Diffusion models in robotics Beyond policy representations in imitation learning, diffusion models have been used to model cross-object and object-part arrangements [39, 40, 38, 41, 30], visual image subgoals [42, 43, 44, 45], and offline reinforcement lcarning [46, 47, 48]. ChainedDiffuser [21]

---

proposes to replace motion planners, commonly used for keypose to keypose linking, with a trajectory diffusion model that conditions on the 3D scene feature cloud and the predicted target 3D keypose to denoise a trajectory from the current to the target keypose. It uses a diffusion model that takes as input 3D end-effector keyposes predicted by Act3D [15] and a 3D representation of the scene to infer robot end-effector trajectories that link the current end-effector pose to the predicted one. 3D Diffuser Actor instead predicts the next 3D keypose for the robot's end-effector alongside the linking trajectory, which is much harder task than linking two given keyposes. 3D Diffusion Policy [22] also combines 3D scene representations with diffusion objectives but uses ID point cloud embeddings. We compare against both ChainedDiffuser and 3D Diffusion Policy in our experimental section and show we greatly outperform them.
2D and 3D scene representations for robot manipulation End-to-end image-to-action policy mod- els, such RT-1 [49], RT-2 [50], GATO [51], BC-Z [52], RT-X [53], Octo [54] and InstructRL. [55] leverage transformer architectures for the direct prediction of 6-DoF end-effector poses from 2D video input. However, this approach comes at the cost of requiring thousands of demonstrations to implicitly model 3D geometry and adapt to variations in the training domains. 3D scene-to-action policies, exemplified by C2F-ARM [13] and PerAct [14], involve voxelizing the robot's workspace and learning to identify the 3D voxel containing the next end-effector keypose. However, this becomes computationally expensive as resolution requirements increase. Consequently, related approaches resort to either coarse-to- to-ein voxelization, equivariant networks [12] or efficient attention opera- tions [56] to mitigate computational costs. Act3D [15] foregoes 3D scene voxelization altogether; it instead computes a 3D action map of variable spatial resolution by sampling 3D points in the empty workspace and featurizing them using cross-attentions to the 3D physical scene points. Robotic View Transformer (RVT) [16] re- re-projects the input RGB-D image to alternative image views, featurizes those and lifts the predictions to 3D to infer 3D locations for the robot's 's end-effector.
3D Diffuser Actor builds upon works of Act3D [15] from 3D policies and upon the works of [36, 6] from diffusion policies. It uses tokenized 3D scene representation, similar to [15], but it is probabilistic model instead of deterministic one. It does not sample 3D points and does not infer 3D action maps. uses diffusion objectives instead of classification or regression objectives used in [15]. In contrast to [40, 36), it uses 3D scene representations instead 2D images or low-dimensional states. We compare against both 2D diffusion policies and 3D policies in our experiments and show 3D Diffuser Actor greatly outperforms them. We highlight the differences between our model and related models in Figure 3 in the Appendix, and refer to Figures 4 and 5 for more architectural details of 3D Diffuser Actor and Act3D.
3 Method
3D Diffuser Actor is trained to imitate demonstration trajectories of the form of {(o1, a1), (02, a.., accompanied with a task language instruction I, similar to previous works [57, 14, 15, 58], where o, stands for the visual observation and a stands for robot action at timestep f. Each observation Or is one or more posed RGB-D images. Each action ag is an end-effector pose and is decomposed into 3D location, rotation and binary (open/close) state: a, (atoe E R$ attt R6 a, open E {0, 1}). We represent rotations using the 6D rotation representation of [59] for all environments in all our experiments, to avoid the discontinuities of the quaternion representation. We will use the notation 7. = (alce (at4 (.44) to denote the trajectory of 3D locations and rotations at timestep L, of temporal horizon T. Our model, at each timestep I open predicts a trajectory Tr and binary states a
The architecture of 3D Diffuser Actor is shown in Figure 1. It is a conditional diffusion probabilistic model [31, 60] of trajectories given the visual scene and language instruction; it predicts a whole trajectory 7 at once, gradually adds noise to sample T0. The diffusion process is associated with variance schedule {33² E (0, 1)) which defines how much noise is added at each diffusion step. The noisy version of sample 7 at diffusion step can then be written as T' V + vI- VI-@'e, where (0,
autoregressively. through iterative denoising, by inverting a process that
3

---

MLP
Proprocous Wain
Dancoiin sieo
MUP
tckend
Denelsng
imige Faosder
scane
Longuage
Irage
Erraader
20 to 3D unprojection
Mull-viow ragregation
"Open the drawer"
(a) model architecture
C-C
r" (0,
Trajectory prediction
(b) Denoising process
"Put the blocks in criangle on the plate"
"stack the cups
Real-world multi-task manipulation
Figure I: Overview of 3D Diffuser Actor. (a) Diffuser Actor is conditional diffusion proba- bilistic model of robot 3D trajectories. At diffusion step i, it converts the current noised estimate of the robot's future action trajectory T', the posed RGB-D views o, and proprioceptive information e to set of 3D tokens. It fuses information among these 3D tokens and language instruction tokens / using 3D relative denoising transformers to predict the noise of 3D robot locations foe (o, 4 c, T'. and the noise of 3D robot rotations tj (o, T',"), (b) During inference, 3D Diffuser Actor iteratively denoises the noised estimate of the robot's future trajectory. (c) 3D Diffuser Actor works in the real world and captures the multiple behavioural modes in the training demonstrations.
sample from a Gaussian distribution (with the same dimensionality as T a² 10==1- and a -I1i=10 a".
3D Relative Denoising Transformer 3D Diffuser Actor models a learned gradient of the denoising process with a 3D relative transformer é ea (T/i i, o,, cz.) that takes as input the noisy trajectory T/ at timestep t, diffusion step i, and conditioning information from the language instruction 1, the visual observation o, and proprioception et of timestep 4, to predict the noise component At each timestep and diffusion step i, we convert visual observations o,, proprioception or and noised trajectory estimate T to set of 3D tokens. Each 3D token represented by latent embedding and 3D position. Our model fuses all 3D tokens using relative 3D attentions, and additionally fuses information from the language instruction using normal attentions, since it is meaningless to define 3D coordinates for language tokens. Next, we describe how we featurize each piece of input (when not ambiguous, we omit the subscript t for clarity).
3D tokenization At each diffusion step i, we represent the noisy estimate 7 of the clean trajectory TO as sequence of 3D trajectory tokens, by mapping each noisy pose a' of m', latent embedding vector with MLP, and 3D position which comes from the noised 3D translation component a loc We featurize each image view using a 2D feature extractor and obtain a corresponding 2D feature map E I xc xc. where e the number of feature channels and H, W the spatial dimensions, using a pre-trained CLIP ResNet50 2D image encoder [61]. Given the corresponding depth map from that view, we compute the 3D location (X, Y, Z) for each of the H x W feature patches by averaging the depth value inside the patch extent. We map the pixel coordinate and depth value of patch to corresponding 3D coordinate using camera intrinsics and extrinsics and the pinhole camera model. This results in 3D scene token set of cardinality H x W. Each scene token is

---

represented by the corresponding patch feature vector Fz.g. the feature vector at the corresponding patch coordinate (z, y). and a 3D position. If more than one views are available, we aggregate 31) seene tokens from each, to obtain the final 3D scene token set. The proprioception e is also a 3D scene token, with learnable latent representation and the 3D positional embedding corresponding to the end-effector's current 3D location. Lastly, we map the language task instruction language tokens using pre-trained CLIP language encoder, following previous works [15].
Our 3D Relative Denoising Transformer applies relative self attentinss among all 3D tokens, and cross-attentions to the language tokens. For the 3D self-attentions, we use the rotary positional embeddings [18] to encode relative positional information in attention layers. The attention weight between query q and key l written as: Eq.k x x MP, P&)XK, where Xy / X2 and Py Pk denote the features and 3D positions of the query / key, and M is matrix function which depends only the relative positions of the query and key, inspired by recent work in visual correspondence [62, 63] and 3D manipulation [15, 21J. We feed the final trajectory tokens to MLPs to predict: (1) the noise (o, c, T', .i) and (o, rotations, respectively, and (2) the end-effector opening fo (o, l,., G, T i) E 1)..
L,
added to s sequence of 3D translations and 3D
Training and inference During training, we randomly sample a time step and diffusion step and add noise (eloe, = ground-truth trajectory 7f. We use the L1 loss for reconstructing the sequence of 3D locations and 3D rotations. We use binary cross-entropy (BCE) loss to supervise the end-effector opening fopen, we use the prediction from i=l at inference time. Our objective reads:
Co wulll(ee (o, (o, - - |0 wall(r (o, -
(o,l,,, (i), ope"),
where W1, W2 are hyperparameters estimated using cross-validation. To draw a sample from the learned distribution pe pr c), start by drawing sample TN (0, Then, we progressively denoise the sample by iterated application of eg N times according to a spec- ified sampling schedule [32, 64], which terminates with To sampled from po(T): -'
8 Va ('' vT-a, Vi colo,,,,,,
T-a- B'z, where z ~ (0, 1) is a random variable of ap- 1 1-a"t1 f'z,
propriate dimension. Empirically, found that using separate noise schedulers for loe and rot specifically, scaled-lineal and square cosine scheduler, respectively, achieves better performance.
Implementation details At training, we segment demonstration trajectories at detected end-effector keyposes. such as a change in the open/close end-efffctorr state or local extrema of velocity/accel- eration, following previous works [57. 14. 15, 55]. We then resample each trajectory segment to have the same length T. During inference, 3D Diffuser Actor can either predict and execute the full trajectory of actions up to the next keypose (including the keypose), or just predict the next keypose and use a sampling-bass
motion planner to reach it, similar to previous works [14, 27, 15].
Due to space limits, please check Section B.3 for detailed model diagram, Section B.5 for our choice of hyper-parameters, Section B.6 for detailed formulation of denoising diffusion probabilistic models and Section B.7 for discussion of noise schedulers in the Appendix.
4 Experiments
We test 3D Diffuser Actor in multi-task manipulation on RLBench [19] and CALVIN [20], two established learning from demonstrations benchmarks, and in the real world.
4.1
Evaluation on RLBench
RLBench is built atop the CoppelaSim [65] simulator, where a Franka Panda Robot is used to manipulate the scene. On RLBench, our model and all baselines are trained to predict the next end-effector keypose as opposed to keypose trajectory; all methods employ a low-level motion planner BiRRT [66], native to RLBench, to reach the predicted robot keypose. We train and evaluate 3D Diffuser Actor on two setups based on the number of available cameras: 1. Multi-view setup, introduced in [14], that uses a suite of 18 manipulation tasks, each with 2-60 variations, which concern scene variability across object poses, appearance and semantics. There are four RGB-D cameras available, front, wrist, left shoulder and right shoulder. The wrist camera moves during manipulation. 2. Single-view setup, introduced in [67], that uses a suite of 10 manipulation tasks.
5

---

Avg. Success
open drawer
slide block
sweep to dustpan
meat off grill
turn tap
put drawer
close jar
drag stick
stack
blocks
C2E.ARM-BC BC [13]
20.1
20
16
0
20
68
4
24
24
0
PerAct [14)
49.4
88.0
21.0
52.0
70.4...
88.0 g.
51.2
55.2
26.4 89.6
HiveFormer (27)
52
64
28
100
80
68
52
76
PolarNet [68]
46
84
56
52
100
80
32
36
92
4
RVT [16]
62.9
71.2
81.625.4
42.0
88.0+2.
23.6
88. 88.0
52.0
99.2+
23.9
Act3D [15]
63.2
78.4 18.2
96. O.2.6
86.4
25.2+1.0
94.4220
91.2+..9
96.8_a..
80.8
4.0
3D Diffuser Actor (ours)
81.3
89.6
97.6
g4.0
96.8
29.2.1
96.0
96.0
100.0.0.
68.3
screw
put
place wine
put
sort
push buttons
insert
stack
place
bulb
safe
cuphoard
shape
peg
cups
cups
CZEAARMBB [13]
12
8
8
72
0
PerAct [14]
17.6g2.u
86.0ga.
44.8
28.0 28.0
26.8
92.8
5.6 0
2,4
2.4
HiveFormer (27)
76
80
32
8
84
0
PolarNet [68]
84
40
12
12
96
0
RVT [16]
48.0
91.2+a..
91.0.8.2
49.6
36.0
100.0
11.2
26.4
4.0.2..
Act3D [15]
32.8
95.24440
59.249.3
59,
67.2+3.0
29,6
93.622.
24.0
9.640.0
3.2.3..
3D Diffuser Actor (ours)
82.4
97.6:2.0
93.6g-15
85.6
4..0
98.4
65.6
47.2.8.5
24.0+7.0
Table 1: Evaluation on RLBench on the multi-view setup. We show the mean and standard
deviation of success rates average across all random seeds. 3D Diffuser Actor outperforms all prior arts on most tasks by a large margin. Variances are included when available.
Avg. Success
close jar
open drawer
sweep dustpan
off grill
turn tup
slide block
put drawer
drag stick
push buttons
stack blocks
GNFactor (67)
31.7
25.3
76.0
28.0
$7.3
50.7
20.0
0.0
37.3
18.7
4.0
Act3D [15]
65.3
52.0ga.7
84.0
80.0g9.
66.7 21.9
64.026.7
100.0.o.o
54.7
86.7.1.
64.0z1.
0.0g0.
2D Diffuser Actor
8.4
0.e
0.o
4.0
56.0u3.3
16.0
0.o
0_0
8.0
0go
Difluser Actor
78.4
82.7.1.
89.3
94.7.1.
88.0
80.0
92.0g0.
77.3
98.7
69.3
12.0
Table 2: Evaluation on RLBench on the single-view setup. 3D Diffuser Actor outperforms prior state-of-the-an baselines, GNFactor and Act3D, on most tasks by a large margin.
Only the front RGB-D camera view is available. We evaluate policies by task completion success rate, which is the proportion of execution trajectories that achieve the goal conditions specified in the language instructions [15, 14].
Baselines All compared methods on RLBench are 3D policies that use depth and camera parameters during featurization of the RGB-D input. We compare against the following: C2F-ARM-BC [13] and PerAct [14] that voxelize the 3D workspace, Hiveformer [27] that featurizes XYZ coordinates aligned with the 2D RGB frames, PolarNet [68] that featurizes a scene 3D point cloud, GNFactor [67] that uses a single RGB-D view and is trained to complete the 3D feature volume, RVT [16] and Act3D [15], that the previous SOTA methods on RLBench. We report results for RVT, PolarNet and GNFactor based on their respective papers. Results for CF2-ARM-BC and PerAct are presented as reported in [16]. Results for Hiveformer are copied from [68]. We retrained Act3D on the multi-view setup using the publicly available code. because we found some differences on the train and test splits used in the original paper. We also trained Act3D on the single-view setup to use as additional baseline for the single-view setup, alongside GNFactor. 3D Diffuser Actor and all baselines are trained on the same set of keyposes extracted from expert demonstrations [57].
We show quantitative results for the multi-view setup in Table 1 and for single-view setup in Table 2. On multi-view, 3D Diffuser Actor achieves average 81.3% success rate among all 18 tasks, an absolute improvement of +18.1% 1% over Act3D, the previous state-of-the-are In particular, 3D Diffuser Actor achieves big leaps on long-horizon high-precision tasks with multiple modes, such as stack blocks, stack cups and place cups, which most baselines fail to complete. All baselines use classification or regression losses, our model is the only one to use diffusion for action prediction.
On single-view, view, 3D Diffuser Actor outperforms GNFactor by +46.7% and Act3D by + +13.1%. Sur- prisingly, Act3D also outperforms GNFactor by big margin. This suggests that the choice of 3D scene representation is more crucial than 3D feature completion. Since Act3D has a similar 3D scene tokenization as 3D Diffuser Actor, this shows the importance of diffusion over alternatives to handle multimodality in prediction, specifically over sampling based 3D action maps and rotation regression.
Ablations We consider the following ablative versions of our model: 1. 2D Diffuser Actor, our implementation of 2D Diffusion Policy of [6]. We remove the 3D scene encoding from 3D Diffuser Actor and instead use per-image 2D representations by average-pooling features

---

within each view. We add learnable embeddings to distinguish between different views and fuse them with action estimates, as done in [6]. See Fig. 6 in the Appendix for more details. 2. 3D Diffuser Actor w/o RelAttn., an ablative version that uses absolute (non-relative) attentions.
We show ablative results in Table 3. 3D Diffuser Actor largely outperforms its 2D counterpart, 2D Diffuser Actor. This shows the importance of 3D scene representations in performance. 3D Diffuser Actor with absolute 3D at- tentions (3D Diffuser Actor w/o RelAttn.) is worse than 3D Diffuser Actor with relative 3D attentions. This shows that translation equivariance through relative attentions is important for generalization. Despite that, this ablative version already outperforms all prior arts in Table 1, prov- ing the effectiveness of marrying 3D representations and diffusion policies.
Avg. Success.
2D Diffuser Actor
47.0
3D Diffuser Actor w/o RelAttn.
71.3
3D Diffuser Actor (ours)
81.3
Table 3: Ablation study. Our model sig- nificantly outperforms its counterparts that do not use 3D scene representations or Fanslation-equivariann 3D relative at- tention.
4.2 Evaluation on CALVIN
The CALVIN benchmark is built on top of the PyBullet [69] simulator and involves Franka Panda Robot arm that manipulates the scene. CALVIN consists of 34 tasks and 4 different environments (A, B, C and D). All environments are equipped with a desk, sliding door, drawer, button that turns on/off an LED, a switch that controls a lightbulb and three different colored blocks (red, blue and pink). These environments differ from each other in the texture of the desk and positions of the objects. CALVIN provides 24 hours of tele-operated unstructured play data, 35% of which are annotated with language descriptions. Each instruction chain includes five language instructions that need to be executed sequentially. We evaluate on the so called zero-shot generalization setup, where models are trained in environments A, B and C and tested in D. We report the success rate and the average number of completed sequential tasks, following previous works [70, 58]. No motion planner available in CALVIN so all models need to predict robot pose trajectories. We found beneficial to employ more layers of language attention, as language understanding on CALVIN is more challenging. See Fig. 4 in the Appendix for more details.
Baselines All methods tested so far in CALVIN are 2D policies, that do not use depth or camera extrinsics. We compare against the hierarchical 2D policies of MCII[[1]] HULC [70] and SuSIE [45] which predict latent features or images of subgoals given a language instruction, which they feed into lower-level coogoall-ondiiionns
policies. They train the low-level policy on all data available in CALVIN as opposed to the language annotated subset only. We compare against large scale 2D transformer policies of RT-1 [49], RoboFlamingo [72] and GR-1 [58] which pretrain on large amounts of interaction or observational (video alone) data. We report results for HULC, RoboFlamingo, SuSIE and GR-1 I from the respective papers. Results from MCIL are borrowed from [70]. Results from RT-1
are copied from [58].
We also compare against 3D Diffusion Policy [22] and ChainedDiffuser [21], which we train both on the language annotated training set. We evaluate 3D Diffusion Policy, ChainedDiffuser and 3D Diffuser Actor with the final checkpoints across 3 seeds. We report both the mean and standard deviation of evaluation results. We devise an algorithm to extract keyposes on CALVIN, since prior works do not use keyposes. We define keyposes as those at frames with significant motion change. Both ChainedDiffuser and 3D Diffuser Actor segment the demonstations based on the keyposes. Notably, though keyposes extracted in RLBench have clear structure since they correspond to programmatically labeled 3D waypoints [19, 57], keyposes extracted in CALVIN are instead noisy and random, as the benchmark consists of human play trajectories.
For evaluation, for each task, our model predicts at most 60 trajectories in sequence, one after the other (it conducts 60 inferences per task). Each trajectory is average 6 action long. We did this to ensure a fair comparison to prior works that predict a maximum of 360 actions in total per task [70, 45, 72].
We show quantitative results Table 4. 3D Diffuser Actor outperforms the state-of-the-art. Chained- Diffuser does not work well on this benchmark, as its deterministic keypose prediction module fails to predict end-effector keyposes accurately due to multimodality present in human demonstrations of CALVIN, in comparison to programmatically collected demonstrations of RLBench.
7

---

Train episodes
Task completed in a row
2
3
4
5
Avg.
Len
3D Ditfusion Policy (22)
Lang
28.7
2.7.1..
0.0
0.010.0
0.0 0.0_...
0.31 10.04
MCTL [711
All
30.4
1.3
0.2
0.0
0.0
0.31
IIULC [70]
All
5.7
1,9
1.1
0.67
RT-1 [49]
Lang
53.3
22.2
9.4
3.8
1.3
0.90
ChainedDiffuser [21] (60 keyposes)
Lang
49.9
21.110.01
8.0
3.5
1,5_0.0
0.84
RoboFlamingo [72]
Lang 82.4
61.9
46.6
33.1
23.5
2.48
SuSIE [45]
All
87.0
69.0
49.0
38.0
26.0
2.69
GR-1 [58]
Lang 85.4
71.2
59.6
49.7
40.1
3.06
3D Diffuser (ours)
Lang
93.8
80.3
66.2.2001
53.3
41.2
3.35
Table 4: Zero-shot long-horizon evaluation on CALVIN on 3 random seeds.
4.3 Evaluation in the real world
We validate 3D Diffuser Actor in learning manipu- lation tasks from real-world demonstrations across 12 tasks. We use Franka Emika robot equipped with Azure Kinect RGB-D sensor front view. Images are originally captured at 1280 x 720 resolu- tion and downsampled to resolution of 256 256. During inference. we utilize the BiRRT [66] planner provided by the Movelt! ROS package [73] to reach the predicted poses. We collect 15 demonstrations per task, most of which naturally contain noise and multiple modes of human behavior. For example,
close
put duck
insert peg into hole
insert peg into torus
put
open
bux
mouse
pen
100
100
50
30
80
100
press
put grapes
sort rectangle
stack blocks
stack cups
put block
stapler
in triangle
90 90
50
20
40
90
Table 5: Multi-Tane Task performance real- world tasks.
we pick one of two ducks to put in the bowl, we insert the peg into one of two holes and we put one of three grapes in the bowl. We evaluate 10 episodes for each task and report the success rate. We show quantitative results Table 5 and video results on project webpage. 3D Diffuser Actor effectively learns real-world manipulation from handful demonstrations.
Runtime We compare the inference time of our model against ChainedDiffussr [21] and 3D Dif- fusion Policy [22] on CALVIN, using an NVIDIA 2080 Ti graphic card. The inference speed of 3D Diffuser Actor, ChainedDiffuser and 3D Diffusion Policy is 600ms, 1 1170ms (50 for keypose detection and 1120 for trajectory optimization) and 581ms.
Limitations Despite its SOTA performance with large margins over existing methods, our frame- work currently has the following limitations: requires camera calibration and depth information, same as all 3D policies. 2. All tasks in RLBench and CALVIN are quasi-static. Extending our method to dynamic tasks and velocity control direct avenue of future work. It on average slower than non- dindffffuionn policies. This can be improved by recent techniques reducing the inference steps of diffusion models.
Please, refer to our Appendix for more experiments and details: Section A.1 for robustness of 3D Diffuser Actor to depth noise, Section A.2 for discussion of failure cases, Section A.3 and A.4 for descriptions of tasks in RLBench and the real world, Section A.5 for descriptions of baselines, Section 1 for implementation details of re-training 3D Diffusion Policy on CALVIN, Section A.6 for keypose discovery on CALVIN. Video results can be found in our supplementary file.
Conclusion
We present 3D Diffuser Actor, manipulation policy that combines 3D scene representations and action diffusion. Our method sets new state-of-the-an on RLBench and CALVIN by a large margin over existing 2D and 3D policies, and learns robot control in the real world from handful of demonstrations, 3D Diffuser Actor builds upon recent progress on 3D tokenized scene representations for robotics and generative models and shows how their combination is a powerful learning from demonstration method. Our future work includes learning 3D Diffuser Actor policies from suboptimal demonstrations and scaling up training data in simulation and in the real world.

---

Acknowledgements
This work supported by Sony AI, NSF award No 1849287, DARPA Machine Common Sense, an Amazon faculty award, and an NSF CAREER award. The authors would like to thank Moritz Reuss for useful training tips on CALVIN; Zhou Xian for help with the real-robot experiments; Brian Yang for discussions, comments and efforts in the early development of this paper.
References
[1] J. Ho and S. Ermon. Generative adversarial imitation learning. CoRR, abs/ 160..03466. 2016. URL http: arxiv. wwtoo////aaao. absa 034776666600666..
[2] Y. Tsurumine and T. Matsubara. Goal-aware generative adversarial imitation learning from imperfect demonstration for robotic cloth manipulation, 2022.
[3] K. Hausman, Y. Chebotar, S. Schaal, G. S. Sukhatme, and J. Lim. Multi-modal imitation learn- ing from unstructured demonstrations using generative adversarial nets. CoRR, abs/1 2017. URL wwtpt:tt/// http///axiivv/////.//... aaas//////// /abs/1/0 1100/170..0 10479.
[4] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone, 2022.
[5] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, and S. Devlin. Imitating human behaviour with diffusion models, 2023.
[6] C. Chi, S. Feng. Y. Du. Z. Xu, E. Cousineau, B. Burchfiel, and Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.0413: 2023.
[7] M. Reuss, M. Li, X. Jia, and R. Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint ar XxXv:2334.00233.. 2023.
[8] A. Mandlekar, F. Ramos, B. Boots, L. Fei-Fei, A. Garg, and D. Fox. IRIS: implicit reinforcement without interaction at scale for learning control from offline robot manipulation data. CoRR, abs/1911.0533) 2019. URL http::/arxiv.cr
abs /1911. 05321.
[9] S. Chernova and M. Veloso. Confidence- policy learning from demonstration using gaussian mixture models. In Proceedings of the 6th International Joint Conference on Au- tonomous Agents and Multiagent Systems, AAMAS '07, New York, NY, USA, 2007. Associa- tion for Computing Machinery. ISBN 9788190426255. 101:100.1445/1329155 1322994477 URL
https: /tw.///////////////// /10/ 1eee/111111111 211.0/....11111111211155444444
[10] Florence, C. Lynch, A. Zeng, O. Ramirez, A. Wahid, L.. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. CoRR, abs/2109.00137,, 2021. URL https: //arxiv. org/abs s 22199. 00137.
[11] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning, pages 726-747. PMLR, 2021.
[12] H. Huang, O. Howell, Zhu, D. Wang, R. Walters, and R. Platt. Fourier transporter: Bi- equivariant robotic manipulation in 3d. In ICLR, 2024.
[13] James, Wada, T. Laidlow, and J. Davison. Coarse-to-fie fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13739-13748, 2022.
[14] M. Shridhar, I.. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023.
[15] Gervet, Z. Xian, Gkanatsios, and K. Fragkiadaki. Act3d: feature field transformers for multi-task robotic manipulation. CoRL, 2023.

---

[16] A. Goyal, J. Xu, Y. Guo, V. Blukis, Y.-W. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. arXiv preprint arXiv:3306.44896 2023.
[17] P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations, 2018.
[18] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:210..09864, 2021.
[19] James, Z. Ma, Arrojo, and A. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2): -006,, 2020.
[20] O. Mees, Hermann, E. Rosete-Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3):7327-7334, 2022.
[21] Z. Xian, N. Gkanatsios, T. Gervet, T.-W. Ke, and K. Fragkiadaki, Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In Conference Robot Learning, pages
PMLR, 2023.
[22] Y. Ze. G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. Proceedings of Robotics: Science and Systems (RSS), 2024.
[23] D. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In D. Touretzky, editor, Proceedings of (NeurIPS) Neural Information Processing Systems, pages 305 313. Morgan Kaufmann, December 1989.
[24] M. Bojarski, D. D. Testa, D. Dworakowski. B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Mon- fort, U. Muller, Zhang, X. Zhang, J. Zhao, and K. Zieba. End end learning for self-driving cars. CoRR, obs/1600.00336, 2016. URL http: //axxvvv...////////////////awav/ www.wiwww//aaaa.aeee.e....ooeeee orggorgg//a......oooo... hegg/bb//ai /104 07316.
[25] C. Lynch and P. Sermanet. Grounding language in play. CoRR, abs/2005.07644 2020. URL https: / arxiv. org/ abs 22055. 07648.
[26] Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp. Goal-conditioned imitation learning. In
H. Wallach, Larochelle, A. Beygelzimer, F. F'AlcééBBue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httph:ttss/ccmcc http:::://ppccceednngneeeeeeeeee waw.:eepabb311-PpPeeeeee
c8d3a760 3Bccaf70bbab01116555998833333333
[27] P.-L. Guhur, Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven history-aware policies for robotic manipulations. In Conference on Robot Learning, pages 175-187. PMLR, 2023.
[28] Z. J. Cui, Y. Wang, N. M. M. Shafiullah, and L. Pinto. From play to policy: Conditional behavior generation from uncurated robot data. ArXiv, abs/2210. 10047, 2022.
[29] D.-N. Ta, E. Cousineau, H. Zhao, and Feng. Conditional energy-based models for implicit policies: The gap between theory and practice, 2022.
[30] N. Gkanatsios, A. Jain, Z. Xian, Y. Zhang, C. Atkeson, and Fragkiadaki. Energy- models as zero-shot planners for compositional scene rearrangement. arXiv preprint arXiv:2304. 14391, 2023.
[31] J. Sohl-Dickstein. E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
[32] J. Ho, A. Jain, and P. Abbeel. Denoising diffuuson probabilistic models. CoRR, abs/2006.1 1239, 2020. URL https: / /aaariivoorrr abs 20000 11239.
[33] Singh, S. Tu, and V. Sindhwani. Revisiting energy based models as policies: Ranking noise contrastive estimation and interpolating energy models, 2023.
10

---

[34] T. Salimans and J. Ho. Should EBMs model the energy or the score? In Energy Based Models Workshop ICLR 2021, 2021. URL https: https://openreview net forum?id=
9AS 9RR.TF2JRND.
[35] H. Ryu, J. Kim, J. Chang, H. S. Ahn, Seo, T. Kim, Choi, and R. Horowitz. Diffusion-edds: Bi-equivariant denoising generative modeling (3) for visual robotic manipulation. arXiv preprint arXiv:2309.02666 2023.
[36] Urain, N. Funk, Peters, and G. Chalvatzaki. Se (3)-diffusionffeldd:: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pages 5923-5930. IEEE, 2023.
[37] Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.061 06193, 2022.
[38] U. A. Mishra and Y. Chen. Reorientdiff: Diffusion model based reorientation for object manipulation. arXiv preprint arXiv:333.117000 2023.
[39] W. Liu, T. Hermans, S. Chernova, and C. Paxton. Structdiffusion: Object-centric diffusion for semantic rearrangement of novel objects. arXiv preprint arXiv:221 1.02111046044 2022.
[40] A. Simeonov, A. Goyal, Manuelli, L. Yen-Chen. Sarmiento, A. Rodriguez, P. Agrawal, and D. Fox. Shelving, stacking, hanging: Relational pose diffusion for multi-modia rearrangement. arXiv preprint
St0i::2307.0475555 2023.
[41] X. Fang, C. R. Garrett, C. Eppner, T. Lozano-Pérez L. P. Kaelbling, and D. Fox. Dimsam: Diffusion models as samplers for task and motion planning under partial observability, arXiv preprint arXiv:2306. 13196, 2023.
[42] I. Kapelyukh, V. Vosylius, and E. Johns. Dall-e-bot: Introducing web-scale diffusion models to robotics. IEEE Robotics and Automation Letters, 2023.
[43] Y. Dai, M. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.00111 2023.
[44] A. Ajay, S. Han, Du, Li, Abhi, T. Jaakkola, J. Tenenbaum, L., Kaelbling, Srivastava, and P. Agrawal. Compositional foundation models for hierarchical planning. arXiv preprint arXiv:2330..m 088877 2023.
[45] K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero- shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint a1X 10639, 2023.
[46] H. Chen, C. Lu, C. Ying, H. Su, and J. Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling, 2023,
[47] B. Yang, H. Su, N. Gkanatsios, T.-W. Ke, A. Jain, J. Schneider, and K. Fragkiadaki. Diffusion- es: Gradient-free plamning with diffusion for autonomous driving and zero-shot instruction following. ArXiv, abs/2402.006559, 2024.
[48] Hansen-Estruchh, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. ldql: Implicit g-learning as actor-critic method with diffusion policies. arXiv preprint wwXiv:2304.10533. 2023.
[49] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22/2.006817 2022.
[50] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee. S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, Sanketi, Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-2: Vision-language-action models transfer web knowledge robotic control, 2023.

---

[51] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G.
M. Gimenez, Y. Sulsky, Kay, J. T. Springenberg, al. A generalist agent. arXiv preprint arXiv:2205.06177 067775 2022.
[52] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[53] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh,
A. Brohan, A. Raffin, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Ichter, C. Lu, C. C. Finn, C. Xu, C. Chi, C. Huang, C. Chan, C. Pan. C. Fu, C. Devin, D. Driess, D. Pathak, Shah, D. Büchler. D. Kalashnikov, D. Sadigh, E. Johns, F. Ceola, F. Xia, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Schiavi, H. Su, H. Fang, H. Shi, H. B. Amor, H. I. Christensen, H. Furuta, H. Walke, H. Fang. I. Mordatch,
Radosavovic, I. Leal, J. Liang, J. Kim, J. Schneider, J. Hsu, J. Bohg, J. Bingham, J. Wu, J. Wu, J. Luo, J. Gu, J. Tan, J. Oh, J. Malik, J. Tompson, J. Yang, J. Lim, J. Silvério, J. Han, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Bymne, K. Oslund, K. Kawaharazuka, K. Zhang, K. Majd, K. Rana, K. P. Srinivasan, L. Y. Chen, L. Pinto, L. Tan, L. Out, L. Lee, M. Tomizuka, M. Du, M. Ahn, M. Zhang, M. Ding, M. K. Srirama, M. Sharma, M. J. Kim, N. Kanazawa, N. Hansen, N. M. O. Heess, N. J. Joshi, N. Suenderhauf, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, P. R. Sanketi, P. Wohlhart, P. Xu, P. Sermanet, P. Sundaresan, Q. H. Vuong, R. Rafailov. R. Tian, R. Doshi, R. Mendonca, R. Shah. R. Hoque. R. C. Julian, S. Bustamante,
Kirmani, S. Levine. S. Moore, S. Bahl, S. Dass, S. Song. Xu. S. Haldar, S. O. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Dasari, S. Belkhale, T. Osa, T. Harada, Matsushima, T. Xiao, T. Yu, T. Ding, T. Davchev, T. Zhao, T. Armstrong, Darrell, V. Jain, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Wang, X. Zhu, X. Li, Y. Lu, Chebotar, Y. Zhou, Y. Zhu, Y. Xu, Y. Wang, Y. Bisk, Y. Cho, Y. Lee, Y. Cui, Y. hua Wu,
Y. Tang. Y. Zhu, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Xu, and Z. J. Cui. Open x-embodiment: Robotic learning datasets and rt-x models. ArXiv, abs/23 10.08844 2023.
[54] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. https: / /octo-too//// /oo/toottooeeeee models .iggitttuhi io, 2023.
[55] H. Liu, L. Lec, K. Lee, and P. Abbecl. cntructiio-followwii agents with jointly pre-trained vision-language models. arXiv preprint arXiv:2210.134311 2022.
[56] A. Jaegle, F. Gimeno, A. Brock, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver: General perception with iterative attention, 2021.
[57] S. James and J. J. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Roboties and Automation Letters, 7(2):1612 1619, 2022.
(58) H. Wu, Y. Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleash- ing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023.
[59] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks, 2020.
[60] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688 2020.
[61] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[62] Y. Li and T. Harada. Lepard: Learning partial point cloud matching in rigid and deformable scenes. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[63] N. Gkanatsios, M. K. Singh, Z. Fang, S. Tulsiani, and K. Fragkiadaki. Analogy-forming transformers for shot 3d parsing. ArXiv, abs/ 143822 2023.
12

---

[64] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[65] E. Rohmer, S. P. Singh, and M. Freese. V-rep: A versatile and scalable robot simulation framework. In 2013 TEEE/RSJ international conference on intelligent robols and systems, pages 1321-1326. IEEE, 2013.
[66] J. Kuffner and M. LaValle. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00C117065), volume 2, pages 995-1001. IEEE, 2000.
[67] Y. Ze. G. Yan, Y.H. Wu, A. Macaluso, Y. Ge, J. Ye, N. Hansen, L. Li, and X. Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. arXiv preprint arXivv:308. 16891, 2023.
[68] S. Chen, Pinel, Schmid, and Laptev. Polarnet: 3d point clouds for language-guidee guided robotic manipulation. ArXiv, 1ttp//239915566.. 2023.
[69] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[70] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letlers (RA-L), 7(4): 1205- 1 11212, 2022.
[71] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.072 07648, 2020.
[72] X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y. Jing, W. Zhang, H. Liu, et al. Vision- language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01376 2023.
[73] D. Coleman, I. Sucan, S. Chitta, and N. Correll. Reducing the barrier to entry of complex robotic software: a moveit! case study. arXiv preprint arXiv:1404.317 2014.
[74] G. Qian, L.,, H. Peng. J. Mai, H. A. A. K. Hammoud, M. Elhoseiny, and B. Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In NeurIPS, 2022.
[75] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402 2023.
[76] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with general conditioning layer. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
13

---

Appendix
Additional Experimental Results and Details
15
A.1 Robustness to noisy depth information on RLBench
15
A.2 Failure cases on RLBench
15
A.3 RLBench tasks under multi-view setup
16
A.4 Real-world tasks
17
A.5 Additional details on baselines
18
A.6 Keypose discovery
19
A.7 Comparison between 3D Diffuser Actor and 3D Diffusion Policy under single-task setup
21
Additional Method Details
B.1
Architectural differences between our model and baselines
21
B.2
The formulation of relative attention
21
B.3
Detailed Model Diagram of 3D Diffuser Actor
22
B.4
Detailed Model Diagram of Act3D
22
B.5
Hyper-parameters for experiments
24
B.6
Denoising Diffusion Probabilistic Models
24
B.7
The importance of noise scheduler
25
14

---

A Additional Experimental Results and Details
A.1 Robustness to noisy depth information on RLBench
We evaluate the robustness of 3D Diffuser Actor to noisy depth information. Our results are presented in Table 6. We adopt the single-view setup on RLBench, reporting success rates on 10 tasks using the front camera view. We add Gaussian noise with a variance of 0.01 and 0.03 to the 3D location of each pixel. We evaluate the final checkpoints across 3 seeds with 25 episodes for each task. With mild perturbation, 3D Diffuser Actor can still predict actions precisely. achieving an average success rate of 72.4%. We only observed 6% absolute performance drop. With strong perturbation, 3D Diffuser Actor obtains an average success rate of 50.1%. Even though 3D Diffuser Actor was trained with clean depth maps, performs reasonably well with noisy depth information.
Avg. Success.
close jar
open drawer
sweep dustpan
meat off grill
turn tap
slide block
put in drawer
drag stick
push buttons
stack blocks
0 (clean)
78.4
82.7
89.3
88.0 91.7
80.0=s.6
92.0+0.0
77.3
98.7
69.3_55. 70.7
12.0=3.7
0.01
72.4
80.0
81.3
68.04a.3
85.3
72.7
92.0.a.0
65.3
93.3
5.3
0.03
50.1
41.3
10.3
45.3
74.713.
89.3
89.3
18.711.
48.0
68.0
1.3119
(clean)
= 0.01
0.03
Table 6: Multi- Masti performance on noisy depth information. Top:
We evaluate the robustness of 3D Diffuser Actor to noisy depth information. We adopt the single-view setup in RLBench, reporting success rates on 10 tasks using the front camera view. We add Gaussian noise with a variance of 0.01 and 0.03 to the 3D location of each pixel. We evaluate the final checkpoints across 3 seeds with 25 episodes for each task. 3D Diffuser Actor performs reasonably well with noisy depth information. Visualization of the 3D location of each pixel with different levels of noise.
Bottom:
A.2 Failure cases on RLBench
We analyze the failure modes of 3D Diffuser Actor RLBench. We categorize the failure cases into 4 types: 1) pose precision, where predicted end-effector poses are too imprecise to satisfy the success condition, 2) instruction understanding, where the policy fails to understand the language instruction to grasp the target object, 3) long-horizon task completion, where the policy fails to complete an intermediate task of long-horizon task, and 4) path planning, where the motion planner fails find an optimal path to reach the target end-effector pose.
We conduct the experiment with single-view setup on RLBench, aggregating the failure cases among 10 manipulation tasks. As shown in Fig. 2, the major failure mode is imprecise prediction of end- effector poses. This failure mode often occurs in manipulation tasks that require high precision of predicted poses, such as stack blocks, open drawer and turn tap. Confusion of language instruction is another major failure mode. This mode often occurs in the scene with multiple objects, where the policy fails to grasp the target object specified in the instruction, such as stack blocks, close jar and push buttons. On RLBench, keypose. The motion planner could sometimes fail to find the path. Lastly, 3D Diffuser Actor might fail to complete an intermediate task, such as open drawer of pul item in drawer task.
deploy sample-basee based motion planner to find the path reach the
In the following, we discuss ideas on how to mitigate each failure case:
Imprecise pose prediction: e.g., the model reaches the right object but fails to grasp it due to last centimeter errors (misses while executing fine motor control). We expect that aggregating datasets and training the policy on wider and more diverse set of manipulation tasks would benefit learning finer control.
Language understanding: e.g., the model misunderstands the language instruction and grasps the wrong object. Equiping our model with perception and language understanding
15

---

modules from modern VLMs, such as explicitly detecting the target objects through a VLM pre-trained on internet-scale data, would drastically improve the cognitive capabilities of our model.
Intermediate tasks: e.g.. failing to open the drawer, when the task is to put something inside the drawer. We believe that decomposing task into simpler ones through high-level planning (c.g.. using LLMs) and training reward models for independent subtasks will boost our model's generalization longer-horizon tasks.
Planner failures: we predict the full trajectory up the next keypose, instead of relying to a planner, as we do on CALVIN. We didn't 't do it on RLBench since the amount of planning errors is small.
Failure cases
Figure 2: Failure cases RLBench the setup of GNFactor. We categorize the failure cases into 4 types: 1) precise pose prediction, where predicted end-effector poses satisfy the success condition, 2) instruction understanding, where the policy fails understand the language instruction to grasp the target object, 3) long-horizon task completion, where the policy fails complete an intermediate task of long-horizon task, and 4) path planning, where the motion planner fails to find an optimal path reach the target end-effector pose. Imprecise action prediction and confusion of language instruction are two major failure modes on RLBench.
%
8%
imprecise
50%
38%
Pose precision
Instrudion underslanding Long-harizon task completion Path planning
A.3 RLBench tasks under multi-view setup
We provide an explanation of the RLBench tasks and their success conditions under the multi-view setup for self-completeness. All tasks vary the object pose, appearance and semantics, which are not described in the descriptions below. For more details, please refer to the PerAct paper [14].
1. Open a drawer: The cabinet has three drawers (top, middle and bottom). The agent is successful if the target drawer is opened. The task on average involves three keyposes.
2. Slide a block blue, pink, and yellow). The end-effector must push the block to the zone with the specified color. On average, the task involves approximately 4.7 keyposes
colored zone: There is one block and four zones with different colors (red,
3. Sweep the dust into a dustpan: There are two dustpans of different sizes (short and tall). The agent needs to sweep the dirt into the specified dustpan. The task on average involves 4.6 keyposes.
4. Take the meat off the grill frame: There is chicken leg or steak. The agent needs to take the meat off the grill frame and put it on the side. The task involves keyposes.
5. Turn on the water tap: The water tap has two sides of handle. The agent needs to rotate the specified handle 90" The task involves 2 keyposes.
6. Put a block in the drawer: The cabinet has three drawers (top, middle and bottom). There is block on the cabinet. The agent needs to open and put the block in the target drawer. The task on average involves 12 keyposes.
7. Close a jar: There are two colored jars. The jar colors are sampled from a set of 20 colors. The agent needs to pick up the lid and screw in the jar with the specified color. The task involves six keyposes.
8. Drag a block with the stick: There is a block, stick and four colored zones. The zone colors sampled from a set of 20 colors. The agent is successful if the block is dragged to the specified colored zone with the stick. The task involves six keyposes.
9. Stack blocks: There are colored blocks and green platform. Each four of the 8 blocks share the same color, while differ from the other. The block colors are sampled from a set of 20 colors. The agent needs to stack N blocks of the specified color on the platform. The task involves 14.6 keyposes.
16

---

10. Screw light bulb: There are 2 light bulbs, 2 holders, and 1 lamp stand. The holder colors are sampled from a set of 20 colors. The agent needs to pick up and screw the light bulb in the specified holder. The task involves 7 keyposes.
11. Put the cash in a safe: There a stack of cash and a safe. The safe has three layers (top, middle and bottom). The agent needs to pick up the cast and put in the specified layer of the safe. The task involves 5 keyposes.
12. Place wine bottle on the rack: There is bottle of wine and a wooden rack. The rack has three slots (left, middle and right). The agent needs to pick up and place the wine at the specified location of the wooden rack. The task involves 5 keyposes.
13. Put groceries in the cardboard: There are 9 YCB objects and a cupboard. The agent needs to grab the specified object and place in the cupboard. The task involves 5 keyposes.
14. Put a block the shape sorter: There are blocks of different shapes and sorter with the corresponding slots. The agent needs to pick up the block with the specified shape and insert it into the slot with the same shape. The task involves keyposes.
15. Push button: There are buttons, whose colors are sampled from set of 20 colors. The agent needs to push the colored buttons in the specified sequence. The task involves 3.8 keyposes.
16. Insert a peg: There is I square, and spoke platform with three colored spoke. The spoke colors are sampled from set of 20 colors. The agent needs to pick up the square and put onto the spoke with the specified color. The task involves 5 keyposes.
17. Stack cups: There are 3 cups. The cup colors are sampled from a set of 20 colors. The agent needs to stack all the other cups the specified one. The task involves 10 keyposes.
18. Hang cups on the rack: There are 3 mugs and a mug rack. The agent needs to pick up N mugs and place them onto the rack. The task involves 11.5 keyposes.
A.4 Real-world tasks
We explain the the real- world tasks and their success conditions in more detail. All tasks take place in cluttered scene with distractors (random objects that do not participate in the task) which are not mentioned in the descriptions below.
1. Close a box: The end-effector needs to move and hit the lid of an open box so that it closes. The agent successful the box closes. The task involves two keyposes.
2. Put duck in bowl: There two toy ducks and two bowls. One of the ducks have be placed one of the bowls. The task involves four keyposes.
3. Insert a peg vertically into the hole: The agent needs to detect and grasp a peg, then insert into hole that is placed on the ground. The task involves four keyposes.
4. Insert a peg horizontally into the torus: The agent needs to detect and grasp a peg, then insert it into torus that is placed vertically the ground. The task involves four keyposes.
5. Put a computer mouse on the pad: There two computer mice and one mousepad. The agent needs pick one mouse and place on the pad. The task involves four keyposes.
6. Open the pen: The agent needs to detect pen that is attached vertically to the table, grasp its lid and pull it to open the pen. The task involves three keyposes.
7. Press the stapler: The agent needs to reach and press stapler. The task involves two
keyposes.
8. Put grapes the bowl: The scene contains three vines of grapes of different color and one bowl. The agent needs to pick one vine and place it in the bowl. The task involves four keyposes.
9. Sort the rectangle: Between two rectangle cubes there is space for one more. The task comprises detecting the rectangle to be moved and placing it between the others. It involves four keyposes.
10. Stack blocks with the same shape: The scene contains of several blacks, some of which have rectangular and some cylindrical shape. The task is to pick the same-shape blocks and stack them on top of the first one. It involves eight keyposes.
17

---

11. Stack cups: The scene contains three cups of different colors. The agent needs to successfully stack them in any order. The task involves eight keyposes.
12. Put block in triangle the plate: The agent needs to detect three blocks of the same color and place them inside a plate to form an equilateral triangle. The task involves 12 keyposes.
The above tasks examine different generalization capabilities of 3D Diffuser Actor, for example multimodality in the solution space (5, 8), order of execution (10, 11, 12), precision (3, 4, 6) and high noise/variance in keyposes (1).
A.5 Additional details on baselines
We provide more details of the baselines used this paper. For RLBench, we consider the following baselines:
C2F-ARM-BO [13], a 3D policy that iteratively voxelizes RGB-D images and predicts actions in a coarse-to-fine manner. Q-values are estimated within each voxel and the translation action is determined by the centroid of the voxel with the maximal Q-values.
2. PerAct [14], a 3D policy that voxelizes the workspace and detects the next voxel action through global self-attention.
3. Hiveformer [27], a 3D policy that enables attention between features of different history time steps.
4. PolarNet [68], a 3D policy that computes dense point representations for the robot workspace using PointNext backbone [74].
5. RVT [16], a 3D policy that deploys multi-view transformer to predict actions and fuses those across views by back-projecting to 3D.
6. Act3D [15], 3D policy that featurizes the robot's 3D workspace using carse-to-ffine sampling and featurization. We observed that Act3D does not follow the same setup as PerAct on RLBench. Specifically, Act3D uses different 1) 3D object models, 2) success conditions, 3) training/test episodes and 4) maximum numbers of keyposes during evaluation. For fair comparison, we retrain and test Act3D on the same setup.
7. GNFactor [67], a 3D policy that co-optimizes a neural field for reconstructing the 3D voxels of the input scene and a PerAct module for predicting actions based on voxel representations.
We consider the following baselines for CALVIN:
1. MCIL [71], multi-mode modal goad-condi 2D policy that maps three types of goals-goat images, language instructions and task labels-to shared latent feature space, and conditions on such latent goals to predict actions.
2. HULC [70], a 2D policy that uses a variational autoencoder sample latent plan based on the current observation and task description, then conditions on this latent to predict actions.
3. RT-1 [49], a 2D transformer-basee policy that encodes the image and language into a sequence of tokens and employs a Transformer-based architecture that contextualizes these tokens and predicts the arm movement or terminates the episode.
4. RoboFlamingo [72], 2D policy that adapts existing vision-language models, which are pre-trained for solving vision and language tasks, to robot control. It uses frozen vision and language foundational models and learns a cross-attention between language and visual features, as well as a recurrent policy that predicts the low-level actions conditioned on the language latents.
5. SuSIE [45], 2D policy that deploys large-scale pre-trained image generative model [75] to synthesize visual subgoals based on the current observation and language instruction. Actions are then predicted by low-level coal-conditioned 2D diffusion policy that models inverse dynamics between the current observation and the predicted subgoal image.
6. GR-1 [58], a 2D policy that first pre-trains an autoregressive Transformer on next frame prediction, using a large-scale video corpus without action annotations. Each video frame is encoded into an 1d vector by average-pooling visual features. Then, the same architecture is fine-tuned in-domain to predict both actions and future observations.
18

---

7. ChainedDiffuser [21], a combination of Act3D [15] that predicts end-effector poses at key frames and 3D trajectory predictor that generates intermediate trajectories reach target end-effector poses.
8. 3D Diffusion Policy [22], that encodes sparsely sampled point cloud into 3D representations using PointNeXt [74] encoder. The 3D representations average pooled into ID feature vector. Then, a UNet conditions on the holistic 3D scene representations and predicts the robot end-effectr poses. Notably, in its original implementation, 3D Diffusion Policy is only applied to single-task experimental setup and does not condition on language instructions. For fair comparison, we update the architecture enable language conditioning. See Section A.5.1 for more details.
A.5.1 Re-training of 3D Diffusion Policy on CALVIN
We compare 3D Diffuser Actor against 3D Diffusion Policy on CALVIN. However, in its original implementation, the 3D Diffusion Policy does not condition on language instructions and is applied only in single-task setup. For fair comparison, we enabled language conditioning in the 3D Diffusion Policy by updating its architecture with additional point- point-cloud-te cloud-to-languaatt cross-attention layers. We used language encoder similar to 3D Diffuser Actor to encode language instructions into latent embeddings and applied three cross cross-attention layers among average pooled point-cloud features and language tokens. ensure the maximal performance, we do not use the simplified backbone of DP3 (Simple DP3), but the original DP3 architecture.
We adopt the common setup in CALVIN, using both the front and wrist camera view. Based the suggestion of the paper [22], we crop 160 x 160 and 68 x 68 bounding box from the depth map of the front and wrist camera, sampling 1024 points within each bounding box. We train 3D Diffusion Policy with a batch size of 5400 and a total epoch of 3000. Otherwise, we use the default apperpprraettrs of 3D Diffusion Policy. We set the horizon of action prediction executed actions to 3, and the number observed timesteps to 2. During inference, we allow the model predict 360 times, resulting in a maximum temporal horizon of 1080 actions.
the number of
A.6 Keypose discovery
For RLBench we use the heuristics from [57, 13]: a pose is a keypose if (1) the end-effector state changes (grasp release) or (2) the velocity's magnitude approaches zero (often pre- pre-grasp poses or a new phase of a task). For our real-world experiments we maintain the above heuristics and record pre-grasp poses well the poses the beginning of each phase of task, e-g., when the end-effector is right above an object of interest. We report the number of keyposes per real-world task in this Appendix (Section A.4). Lastly, for CALVIN we adapt the above heuristics devise more robust algorithm to discover keyposes. Specifically, we track end-effector state changes and significant changes of motion, i.c. both velocity and acceleration. For reference, we include our Python code for discovering keyposes in CALVIN here:
"Utilltty functions for computing keyposes. import numpy np from scipy.signn signal import argrelextren
def motion. motion sreiiinnchaneddddee (trajectories, buffer_size).
"**Select keyposes where motion changes signi ficantl... The chosen poses shall be sparse.
Args:
trajectories: 1ist of 1D array buffer_size: integer indicates the minimum distance of waypcints
11
12
13
Returns:
key_poses: list of integers indicates the time steps when mation of the end effector changes significantly.
15
16
17
18
compute velocity
trajectories (
19

---

[trajectories [0]] trajectories, axis=0
22
velocities
23
trajectories [1:] - trajectories[;-1
24
compute accoleration velocities np. concateemate
27
(velocities, [velocities(-1]]} axis=0
28
accelerations velocities [1:1 velocities [=-1]
31
compute the magnitude of acceleration
32
A np.linalg.norm,
33
accelerations(: :3], axis--1
local maximas of acceleration indicates significant motion change of the end effector @ocal_max_A argrelext (A, np.greater) [0]
36
37
38
consider the top 20% of local maximas
40
int (A. shape [0] 0.2)
41
topk no. (A) [::-1]
42
Aarge_A A @1local_max__ >- tepk
local_ _ 1oaa__ax_A(largeAA.ccom tolistt
45
select waypoints sparsely key poses = [local_man max (0)) for i in local_max_A:
46
47
if i key_poses [-1] buffer_size:
49
key_poses. append
append (1)
50
51
return key_poses
def gripper_state_change (trafectories): "**select keyposes where end-effector opens/closes.
54
55
56
Args:
trajectories: 1ist of ID array
59
60
Returns:
key_poses: list of integers indicates the time steps when the end-effector opens/closes.
64
65
trajecteries = ( (trajectories trajectories, axis=0
68
openess crajectoriess -1]
69
changed openessl:-1] 1= openess [1:]
70
key_poses whewhere (changed) tolist ()
72
return key_poses
73
74
def keypoint_discoven (trajectories, buffer_size-5):
"""Select keyposes where motion changes significantly. The chosen
poses shall be sparse.
76
77
Args:
78
trajectories: list of 1D array buffer_size: integer indicates the minimum distance of waypeints
82
Returns:
key_poses: an Integer array indicates the
20

---

indices of keyposes
86
motion moticn_changed motion changedd trajectories, buffer_size
87
88
prripper_changer
91
gripper_state_hhte (tra (trajectooies)
92
93
one frame weworeeeeeeeeeeeee wtw.reeerropeeeeeeeeeeeeeeeeeeee
for in siipper_changes if
95
96
97
last_frame frame
[len (trajeectories) - 1]
key_ sey_ inds
100
seition_changel +
101
hwg___aan_.etooregggpppeooooooop last_frane
()
105
poy_pose_ndds np. np.unique (key_pose_inds)
106
return pey_pose_inds
A.7
Comparison between 3D Difffuser Actor and 3D Diffusion Policy under single-task setup
We test 3D Diffusion Policy [22] to our 3D Diffuser Ac- tor the stack-cup task of RLBench. Under single-task setup, we do not need to condition on language instruction, which matches the original setup of DP3. We trained both methods on 100 episodes and tested on 25 unseen episodes. As shown in Table 7, 3D Diffuser Actor achieves success rate of 80%, and 3D Diffusion Policy achieves a success rate of 8%. We believe this experiment clearly highlights the importance of our tokenized 3D scene and action trans- former, missing from 3D Diffusion Policy, which encodes a whole 3D point cloud into a single embedding vector.
stack cups
3D Diffuser Actor
80
3D Diffusson Policy [22]
Table 7: Single-Tass performance on stack-cup task of RLBench.
B Additional Method Details
B.1
Architectural differences between our model and baselines
We describe the architectural differences between 3D Diffuser Actor and other 3D policies in Figure We compare 3D Diffuser Actor with Act3D [15], PerAct [14], and 3D Diffusion Policy [22]. PerAct encodes RGB-D images into voxel representations, while 3D Diffusion Policy average pools point cloud representations into holistic 1D feature vector. Act3D encodes input images with 2D feature extractor and lifts 2D feature maps to 3D. Act3D determines end-effector poses by iterating a coarse-to-fine fine classification procedure to identify the XYZ location of the ghost point with the highest classification score. Although our model adopts a similar 3D scene encoder as Act3D, 3D Diffuser Actor uses relative 3D transformer denoise end-effector poses.
B.2 The formulation of relative attention
To ensure translation invariance in the denoising transformer, we use the rotary positional embed- dings [18] encode relative positional information in attention layers. The attention between query qs key k, and value v, is written as:
expe
Attention(q. k,, v2) - Expeet
(I)
21

---

Modham
Qrat
Queens
(a) Ac13D
Denclle Spese
Thae tD The aeferre
Qrot
Qopen
pore
(b) PerAct
(c) 3D Diffusion Policy
(d) 3D Diffuser Actor
Comparison of 3D Diffuser Actor and other methods. (a) Act3D [15] encodes input images into with 2D feature extractor, lifts feature maps to 3D, samples ghost points within the scene, featurizes ghost points with relative cross attention to the 3D feature cloud, and classifies each ghost point. The position is determined by the XYZ location of the ghost point with the highest score. Act3D then crops the scene with predicted end-effector position, and iterates the same classification procedure. The rotation and openess are predicted with learnable query embeddings. voxelizes input images and uses Perceiver Transformer [56] to predict the end-effector pose. The position is determined by the XYZ location of the voxel with the highest Q-value. The rotation and openess are predicted from the max-pooled features with MLPs. (c) 3D Diffusion Policy [22] encodes 3D point cloud into ID feature vectors, followed by a A UNet that conditions on point-cloud features and denoises end-effector poses. (d) 3D Diffuser Actor uses similar 3D) scene encoder as Act3D, A relative 3D transformer conditions on 3D feature cloud to denoise end-effector poses.
Figure 3:
(b) PerAct [14]
where ei = q MD, - p.)kj. P. / P, denote the positions of the query key, and M is matrix function which depends only on the relative positions of points p; and Pj-
B.3 Detailed Model Diagram of 3D Diffuser Actor
We present a more detailed architecture diagram of our 3D Diffuser Actor in Figure 4a. We also show variant of 3D Diffuser Actor with enhanced language conditioning in Figure 4b, which achieves SOTA results on CALVIN.
The inputs to our network are i) a stream of RGB-D views; ii) language instruction; iii) proprio- ception in the form of current end-effector's poses; iv) the current noisy estimates of position and rotation; v) the diffusion step i. The images are encoded into visual tokens using pretrained 2D backbone. The depth values are used "lift" the multi-view tokens into 3D feature cloud. The language is encoded into feature tokens using a language backbone. The proprioception is represented as learnable tokens with known 3D locations in the scene. The noisy estimates are fed to linear layers that map them to high-dimensional vectors. The denoising step is fed an MLP.
The visual tokens cross-altend to the language tokens and get residually updated. The proprioception tokens attend to the visual tokens to contextualize with the scene information. We subsample number of visual tokens using Farthest Point Sampling (FPS) in order decrease the computational requirements. The sampled visual tokens, proprioception tokens and noisy position/rotation tokens attend each other. We modulate the attention using adaptive layer normalization and FiLM [76]. Lastly, the contextualized noisy estimates are fed to MI.P to predict the error terms as well as the end-effector's state (open/close).
B.4 Detailed Model Diagram Act3D
To make the paper self-contained, we present detailed architecture diagram of Act3D in Figure 5. Act3D first encodes posed RGB-D images into 3D scene tokens and samples ghost points uniformly from a fixed-radiu sphere, centered at the current end-effector position. Act3D initializes learnable
22

---

Multi-iew RGB-D
Visual Encoder
the middle the rack"
Noley position
Naley rotation 0
Language Encoder
3D Lifting
0000000000
000000
Linear
Lineer
Cress Attention: instructions
Concatenate
00000
FILM
Proprioception
Cross Attentio: pose estimate scene
Self Attention: estimate
MLP
0
Attention: pose scerte
Dencising step
MLP
MLP
Position error open/close
Rotation error
Multi-vlew RGB-D
"Stack the bottle midale
Noisy posittin
Noisy rotation
Language Encoder
3D Lifting
0000000000
000000
Linear
Lineer
Cress Attention: scene instructions
Concatenate
00000
Praprisception
FILM
Cross Attentio: pose estimate scene
Cross Attention: pose estimate, scene - instructions
MLP
Self Attention: pose estimate scene
Denoising step
Cross Attention: estimate, instructions
pose
MLP
MLP
Position error open/close
Rotation
error
Figure 4: 3D Diffuser Actor architecture in more detail. Top: Standard version: the encoded inputs are fed to attention layers that predict the position and rotation error for each trajectory timestep. The language information is fused to the visual stream by allowing the encoded visual feature tokens to attend to language feature tokens. There are two different attention and output heads for position and rotation respectively. Bottom: version with enhanced language conditioning: cross-attention layers from visual and pose estimate tokens to language tokens are interleaved between pose estimate- visual token atlention layers.
query and the ghost points with learnable latent embeddings, which are contexturalized by scene tokens using 3D relative attention (Eqn. 1). Act3D measures the feature similarity of each ghost point and the learnable query to classify if ghost point is close position of target end-effector pose, which is determined by the XYZ location of the ghost point with the highest score. Act3D then crops
23

---

Mutt RGB
"Stack che botcle to the niddle of the rack
Language
3D Lifting
Proprioception
Scenie festure scale sampling
Cross Attention:
instructions
Ghost point sampling
Cross Attention points scenee propriacepiion instructions
Learnable query
query - scene, proprioception.
Score ghost points
Position
Rotation
x3
Figure 5: Act3D architecture in more detail. Ghost points and encoded scene feature inputs are sampled based on current end-effector position. The 3D location ghost points are even sampled from fixed-radius sphere, which is centered at the current end-effector position.
Multi-iiew RGB
Noley position
Noisy rotation 0
2D Visual Encoder Avg. Pooling per embedcing
the
Language Encoder
0000
Concatenate
Cross Attention: scene
Instructions
0000
FILM
Propriacaption
Cross Attention: pose estimate scene
Self Attention: pose estimate scene
MLP
Sell Attention: pose estimate
Denoising step
MLP
MLP
Position error
Rotation error
open/ciose
Figure 6: Architecture of 2D Diffuser Actor, our re-implementation of 2D Diff'usion Policy [6].
the scene with estimated end-effector position, and iterates the same classification procedure. The rotation and openess are predicted with learnable query embeddings.
B.5 Hyper-parameters for experiments
Table 8 summarizes the syper-parameters used for training/evaluating model. On CALVIN observed that our model overfits the training data, resulting in lower test performance. We use higher beight_decapp and shorter total_epoch CALVIN compared RLBench.
B.6 Denoising Diffusion Probabilistic Models
For seff-completenesss, we provide a formulation denoising diffusion process. Please check [60] for more details. A diffusion model learns to model probability distribution p(z) by inverting process

---

Multi-view RLBench
Single-view RLBench
CALVIN
Model
image size
256
256
200
embedding dim
120
120
192
camera_vient
4
2
FPS: % of sampled tokens
20%
20%
33%
diffusion_timmette
100
100
25
noise_scherr sess__cheedllrr: position
lineerr squaredoos absolute pose 3.6M
scaled_ _eneed_lneer squaredcos absolute pose 3.6M
sca scaled_linear squaredcos
noise scheduler rotation action_space
relative displacement 9.6M
# of parameters
Training
batch_size
240
240
5400
learning_rate
le
le
3e 4
wright_decay
5e
5e
5e
total_epochl
1.6c*
Se
90
optimizer
Adam
Adam
Adam
loss weight Wi
30
30
30
loss weight:: 102
10
10
10
Evaluation
maximal # of keyposes
25
25
60
Table
our experiments. We list the hapereparameters used for training/e- valuating our model on RI RIBench and CALVIN simulated benchmarks. On RLBench we conduct experiments under multi-view and single-view setup.
that gradually adds noise to a sample x. For us, x represents sequence of 3D translations and 3D rotations for the robot's end-effector. The diffusion process is associated with variance schedule { E (0, 1) 1 which defines how much noise added each time step. The noisy version of sample æ at time can then be written as x' V + vI - a'e where ~ NO 1), is sample from a Gaussian distribution (with the same dimensionality as z), a² = 1 - 3 and a' IT a². The denoising process is modeled by neural network ca (x*: i) that takes input the noisy sample æ' and the noise level and tries to predict the noise component e.
Diffusion models can be easily extended to draw samples from distribution p(z|e) conditioned on input c, which is added as input to the network Eg. For us, e is the visual scene captured by one or more calibrated RGB-D images, language instruction, well as the proprioceptive information. Given a collection of D = {(r", ct) } of end-effector trajectories x' paired with observation and robot proprioceptive context e', the denoising objective becomes:
(an D)
Σ xieeED
IVeo(Va'a' V1 -(1
(2)
PDd
This loss corresponds to reweighted form of the variational lower bound for (og(()))) [32].
In order to draw sample from the learned distribution ps (=(e),, start by drawing sample N ~ (O, 1). Then, we progressively denoise the sample by iterated application Eø N times according to a specified sampling schedule [32, 64], which terminates with x sampled from (o
g
a cola co(xx,,,)) 2,c)) 1 1 -
x
- σ² 18z,
Va
V1
where z ~ (0)
B.7
The importance noise scheduler
Existing diffusion policies [21, 6] often deploy the same noise scheduler for estimate of position and rotation. However, we empirically found that selecting good noise scheduler is critical. We visualize the clean/noised 6D rotation representations as two
unit-length vectors
25

---

141
neer
00
Traning Step
(a) Clean
(b) Scaled Linear
(c) Square Cosine
(d) Rotation Accuracy
Figure 7: Visualization of noised rotation based on different schedulers. We split the 6 DoF rotation representations into three-dimension unit-length vectors, and plot the first/second vector as point in 3D. The noised counterparts are colorized in magenta/cyan. We visualize the rotation of all keyposes in RLBench insert_peg task. From left to right, (b) noisy rotation with a scaled-linear scheduler, and (c) that with a square cosine scheduler. Lastly, compare (d) the denoising performance curve of two noise schedulers. Here, accuracy is defined as the percentage of times the absolute rotation error is lower than a threshold of 0.025, Using the square cosine scheduler helps model denoise from the pure noise better.
visualize the (a) clean rotation,
in Figure 7. We plot each vector point in the 3D space. We show that noised rotation vectors generated by the squared linear scheduler cover the space more completely than those by the scaled linear scheduler, resulting in better performance than a square cosine scheduler
26


Reviews:
[{"review_id": "Zcwrj6cpri", "reviewer": "Reviewer_cssK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel architecture for diffusion policies that aims to better leverage 3D information often available in robotic applications while also allowing for language conditioning. Consequently, the authors opt for a transformer-based architecture that leverages spatial attention mechanisms. The method shows strong performance in experiments in the CALVIN and RLBench benchmarks.", "summary_of_recommendation": "As detailed, I like the paper except for some improperly presented findings and lacking ablations. If the authors can provide the missing details and fix the presentation accordingly, I am happy to increase my score to a strong accept.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Improvements:\n* A visualization of the 2D diffuser ablation similar to the visualizations in Figure 4 should be added.\n* Please correct the statement about control frequency\n* I would like to see ablations of the method in the single-view RLBench benchmark. After all, there may be many robotic tasks with a single view; hence, it is important to consider the architectural choices in this domain.\n* I would like to see the performance of the \"default\" 3D diffuser policy presented in the main paper (in addition to the improved version).\n* I would be happy if the authors can address my questions regarding the longer 3d diffuser horizon in the CALVIN benchmark.\n* I would like to see an ablation against 3D Diffusion Policy in the stack cups task of the real-world evaluation as it seems to be one of the tasks with which the proposed methods struggles but which does not rely on language (please correct me if I am wrong here).\n\nPOST REBUTALL UPDATE:\n\nThe authors addressed all my points except for the last one. Hence I increased my score to a strong accept.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ItKndsW1w", "reviewer": "Reviewer_45HC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel approach to robot manipulation policies by combining 3D scene representations with diffusion objectives, named 3D Diffuser Actor. This approach has demonstrated significant performance gains over existing methods on benchmarks like RLBench and CALVIN, as well as in real-world applications.", "summary_of_recommendation": "I would possibly recommend accept since this paper presents strong empirical evaluations on 3D Diffuser Actor.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please see weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "swc2mk23Vg", "reviewer": "Reviewer_BGu2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel architecture for diffusion policies operating on point clouds. They compare against a large selection of baselines, and show substantial improvements in success rates.", "summary_of_recommendation": "Accept", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. I find the choice of action representation a bit odd: singling out the discrete open/close action, and having separate noise schedules for the target orientation and position seems like a kludge. Have you considered representing actions as a bunch of 3d keypoints on the end-effector? That is a much more natural action representation for point clouds.\n\n2. Why do you use horizon of length 1 for RLBench experiments? The ability to model trajectories is one of the main benefits of using diffusion, at the very least it would be good to show that your results hold when you predict longer trajectories.\n\n3. Regarding comparison to DP3. Why did you choose to evaluate DP3 on CALVIN instead of RLBench? The DP3 paper doesn't consider language conditioning so it's unclear if this evaluation is meaningful. RLBench seems closer to the evaluations considered in DP3. (Note that this is not important for score given that DP3 was a concurrent work but it's good to clarify this.)\n\n4. I couldn't find information about model sizes in the paper, please add those.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summarizing the weaknesses and strengths pointed out by the reviewers:

Strengths:
- The integration of 3D scene representations with diffusion models is novel
- Strong results on CALVIN and RLBenchmark
- Good ablation analysis
- Well-written
- Experiments are well-executed including extensive comparisons to baselines

Weaknesses:
- While well-written, some minor clarity issues (see reviews for specifics)
- Some architecture choices are not well-justified (e.g., why is 3D diffusion policies only used for CALVIN and not RLBenchmark?)
- Experiments are limited to a parallel gripper
- The discussion of failure cases is rather limited
- The methodology is an incremental integration of existing methods

Overall, all 3 reviewers recommend acceptance (though weakly) and have fairly positive things to say. This appears to be a fairly strong, well-executed paper. However, the authors should take care to address the reviewers's concerns as they point out many.

**Update after rebuttal**: The authors' rebuttal responded directly to the reviewers' concerns, including updating the paper. All 3 reviewers responded to the rebuttal, 2 choosing to maintain their recommendation of weak accept and 1 moving up to strong accept. Considering this, it seems prudent to recommend this paper be accepted.
Decision: Accept