3D Diffuser Actor: Policy Diffusion with 3D Scene Representations

Authors: tsung-wei  ke
nikolaos  gkanatsios
katerina  fragkiadaki
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 05:59:22

Abstract: Diffusion policies are conditional diffusion models that learn robot action distributions conditioned on the robot and environment state. They have recently shown to outperform both deterministic and alternative action distribution learning formulations. 3D robot policies use 3D scene feature representations aggregated from a single or multiple camera views using sensed depth. They have shown to generalize better than their 2D counterparts across camera viewpoints. We unify these two lines of work and present 3D Diffuser Actor , a neural policy equipped with a novel 3D denoising transformer that fuses information from the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories. 3D Diffuser Actor sets a new state-of-the-art on RLBench with an absolute performance gain of 18.1% over the current SOTA on a multi-view setup and an absolute gain of 13.1% on a single-view setup. On the CALVIN benchmark, it improves over the current SOTA by a 9% relative increase. It also learns to control a robot manipulator in the real world from a handful of demonstrations. Through thorough comparisons with the current SOTA policies and ablations of our model, we show 3D Diffuser Actor ’s design choices dramatically outperform 2D representations, regression and classification objectives, absolute attentions, and holistic non-tokenized 3D scene embeddings.

Main paper: 
 introduction: 
 Many robot manipulation tasks are inherently multimodal: at any point during task execution, there may be multiple actions which yield task-optimal behavior. Indeed, human demonstrations often contain diverse ways that a task can be accomplished. A natural choice is then to treat policy learning as a distribution learning problem: instead of representing a policy as a deterministic map π θ (x), learn the entire distribution of actions conditioned on the current robot state p(y|x) [1, 2, 3, 4] .Recent works use diffusion objectives for learning such state-conditioned action distributions for robot manipulation policies from demonstrations [5, 6, 7] . They outperform deterministic or other alternatives, such as variational autoencoders [8] , mixture of Gaussians [9] , combination of classification and regression objectives [4] , or energy-based objectives [10] . They typically use either low-dimensional (oracle) states [5] or 2D images [6] as their scene representation.At the same time, 3D robot policies build scene representations by "lifting" features from perspective views to a 3D robot workspace based on sensed depth and camera extrinsics [11, 12, 13, 14, 15, 16] . They have shown to generalize better than 2D robot policies across camera viewpoints and to handle novel camera viewpoints at test time [15, 16] . We conjecture this improved performance comes from the fact that the visual scene tokens and the robot's actions interact in a common 3D space, that is robust to camera viewpoints, while in 2D policies the neural network need to learn the 2D-to-3D mapping implicitly.In this work, we marry diffusion for handling multimodality in action prediction with 3D scene representations for effective spatial reasoning. We propose 3D Diffuser Actor, a novel 3D denoising 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. policy transformer that takes as input a tokenized 3D scene representation, a language instruction and a noised end-effector's future translation and rotation trajectory, and predicts the error in translations and rotations for the robot's end-effector. The model represents both the scene tokens and the endeffector locations in the same 3D space and fuses them with relative-position 3D attentions [17, 18] , which achieves translation equivariance and helps generalization.We test 3D Diffuser Actor in learning robot manipulation policies from demonstrations on the simulation benchmarks of RLBench [19] and CALVIN [20] , as well as in the real world. 3D Diffuser Actor sets a new state-of-the-art on RLBench with a 18.1% absolute gain on multi-view setups and 13.1% on single-view setups, outperforming existing 3D policies and 2D diffusion policies. On CALVIN, it outperforms the current SOTA in the setting of zero-shot unseen scene generalization by a 9% relative gain. We further show 3D Diffuser Actor can learn multi-task manipulation in the real world across 12 tasks from a handful of real-world demonstrations. We empirically show that 3D Diffuser Actor outperforms all existing policy formulations, which either do not use 3D scene representations, or do not use action diffusion. We further compare against ablative versions of our model and show the importance of the 3D relative attentions.Our contributions: The main contribution of this work is to combine 3D scene representations and diffusion objectives for learning robot policies from demonstrations. 3D robot policies have not yet been combined with diffusion objectives. An exception is ChainedDiffuser [21] , that uses diffusion models as a drop-in replacement for motion planners, rather than manipulation policies, since it relies on other learning-based policies (Act3D [15] ) to supply the target 3D keypose to reach. We compare against ChainedDiffuser in our experiments and show we greatly outperform it.Concurrent work: Concurrent to our effort, 3D diffusion policy [22] shares a similar goal of combining 3D representations with diffusion objectives for learning manipulation from demonstrations. Though the two works share the same goal, they have very different architectures. Unlike 3D Diffuser Actor, the model of [22] does not condition on a tokenized 3D scene representation but rather on a holistic 1D embedding pooled from the 3D scene point cloud. We compare 3D Diffuser Actor against [22] in our experiments and show it greatly outperforms it. We believe this is because tokenized scene representations, used in 3D Diffuser Actor, are robust to scene changes: if a part of the scene changes, only the corresponding subset of 3D scene tokens is affected. In contrast, holistic scene embeddings pooled across the scene are always affected by any scene change. Thanks to this spatial disentanglement of the 3D scene tokenization, 3D Diffuser Actor generalizes better.Code and videos of our manipulation results are available at https://3d-diffuser-actor. github.io/. 
related work: 
 Learning manipulation policies from demonstrations Earlier works on learning from demonstrations train deterministic policies with behavior cloning [23, 24] . To better handle action multimodality, approaches discretize action dimensions and use cross entropy losses [25, 14, 11] . Generative adversarial networks [1, 2, 26] , variational autoencoders [8] , combined Categorical and Gaussian distributions [4, 27, 28] and Energy-Based Models (EBMs) [10, 29, 30] have been used to learn from multimodal demonstrations. Diffusion models [31, 32] are a powerful class of generative models related to EBMs in that they model the score of the distribution, else, the gradient of the energy, as opposed to the energy itself [33, 34] . The key idea behind diffusion models is to iteratively transform a simple prior distribution into a target distribution by applying a sequential denoising process. They have been used for modeling state-conditioned action distributions in imitation learning [35, 36, 5, 37, 7, 38] from low-dimensional input, as well as from visual sensory input, and show both better mode coverage and higher fidelity in action prediction than alternatives.Diffusion models in robotics Beyond policy representations in imitation learning, diffusion models have been used to model cross-object and object-part arrangements [39, 40, 38, 41, 30] , visual image subgoals [42, 43, 44, 45] , and offline reinforcement learning [46, 47, 48] . ChainedDiffuser [21] proposes to replace motion planners, commonly used for keypose to keypose linking, with a trajectory diffusion model that conditions on the 3D scene feature cloud and the predicted target 3D keypose to denoise a trajectory from the current to the target keypose. It uses a diffusion model that takes as input 3D end-effector keyposes predicted by Act3D [15] and a 3D representation of the scene to infer robot end-effector trajectories that link the current end-effector pose to the predicted one. 3D Diffuser Actor instead predicts the next 3D keypose for the robot's end-effector alongside the linking trajectory, which is a much harder task than linking two given keyposes. 3D Diffusion Policy [22] also combines 3D scene representations with diffusion objectives but uses 1D point cloud embeddings. We compare against both ChainedDiffuser and 3D Diffusion Policy in our experimental section and show we greatly outperform them.2D and 3D scene representations for robot manipulation End-to-end image-to-action policy models, such as RT-1 [49] , RT-2 [50] , GATO [51] , BC-Z [52] , RT-X [53] , Octo [54] and InstructRL [55] leverage transformer architectures for the direct prediction of 6-DoF end-effector poses from 2D video input. However, this approach comes at the cost of requiring thousands of demonstrations to implicitly model 3D geometry and adapt to variations in the training domains. 3D scene-to-action policies, exemplified by C2F-ARM [13] and PerAct [14] , involve voxelizing the robot's workspace and learning to identify the 3D voxel containing the next end-effector keypose. However, this becomes computationally expensive as resolution requirements increase. Consequently, related approaches resort to either coarse-to-fine voxelization, equivariant networks [12] or efficient attention operations [56] to mitigate computational costs. Act3D [15] foregoes 3D scene voxelization altogether; it instead computes a 3D action map of variable spatial resolution by sampling 3D points in the empty workspace and featurizing them using cross-attentions to the 3D physical scene points. Robotic View Transformer (RVT) [16] re-projects the input RGB-D image to alternative image views, featurizes those and lifts the predictions to 3D to infer 3D locations for the robot's end-effector. 3D Diffuser Actor builds upon works of Act3D [15] from 3D policies and upon the works of [36, 6] from diffusion policies. It uses a tokenized 3D scene representation, similar to [15] , but it is a probabilistic model instead of a deterministic one. It does not sample 3D points and does not infer 3D action maps. It uses diffusion objectives instead of classification or regression objectives used in [15] . In contrast to [40, 36] , it uses 3D scene representations instead of 2D images or low-dimensional states. We compare against both 2D diffusion policies and 3D policies in our experiments and show 3D Diffuser Actor greatly outperforms them. We highlight the differences between our model and related models in Figure 3 in the Appendix, and we refer to Figures 4 and 5 for more architectural details of 3D Diffuser Actor and Act3D. 
method: 
 3D Diffuser Actor is trained to imitate demonstration trajectories of the form of {(o 1 , a 1 ), (o 2 , a 2 ), ...}, accompanied with a task language instruction l, similar to previous works [57, 14, 15, 58] , where o t stands for the visual observation and a t stands for robot action at timestep t. Each observation o t is one or more posed RGB-D images. Each action a t is an end-effector pose and is decomposed into 3D location, rotation and binary (open/close) state:a t = {a loc t ∈ R 3 , a rot t ∈ R 6 , a open t ∈ {0, 1}}.We represent rotations using the 6D rotation representation of [59] for all environments in all our experiments, to avoid the discontinuities of the quaternion representation. We will use the notation τ t = (a loc t:t+T , a rot t:t+T ) to denote the trajectory of 3D locations and rotations at timestep t, of temporal horizon T . Our model, at each timestep t predicts a trajectory τ t and binary states a open t:t+T . The architecture of 3D Diffuser Actor is shown in Figure 1 . It is a conditional diffusion probabilistic model [31, 60] of trajectories given the visual scene and a language instruction; it predicts a whole trajectory τ at once, non autoregressively, through iterative denoising, by inverting a process that gradually adds noise to a sample τ 0 . The diffusion process is associated with a variance schedule {β i ∈ (0, 1)} N i=1 , which defines how much noise is added at each diffusion step. The noisy version of sample τ 0 at diffusion step i can then be written as τ i = √ ᾱi τ 0 + √ 1 -ᾱi ϵ, where ϵ ∼ N (0, 1)𝝉 ! ~𝑁(𝟎, 𝟏) 𝝉 !"# 𝝉 $ 𝝉 # Trajectory prediction … 𝜖 ! "#$ , 𝜖 ! %#& 𝜖 ! "#$ , 𝜖 ! %#& 𝜖 ! "#$ , 𝜖 ! %#& 𝝉 %"Open the drawer" is a sample from a Gaussian distribution (with the same dimensionality as τ 0 ), α i = 1 -β i , and ᾱi = i j=1 α j . 3D Relative Denoising Transformer 3D Diffuser Actor models a learned gradient of the denoising process with a 3D relative transformer ε = ϵ θ (τ i t ; i, o t , l, c t ) that takes as input the noisy trajectory τ i t at timestep t, diffusion step i, and conditioning information from the language instruction l, the visual observation o t and proprioception c t of timestep t, to predict the noise component ε. At each timestep t and diffusion step i, we convert visual observations o t , proprioception c t and noised trajectory estimate τ i t to a set of 3D tokens. Each 3D token is represented by a latent embedding and a 3D position. Our model fuses all 3D tokens using relative 3D attentions, and additionally fuses information from the language instruction using normal attentions, since it is meaningless to define 3D coordinates for language tokens. Next, we describe how we featurize each piece of input (when not ambiguous, we omit the subscript t for clarity).3D tokenization At each diffusion step i, we represent the noisy estimate τ i of the clean trajectory τ 0 as a sequence of 3D trajectory tokens, by mapping each noisy pose a i of τ i , to a latent embedding vector with a MLP, and a 3D position which comes from the noised 3D translation component a loc,i . We featurize each image view using a 2D feature extractor and obtain a corresponding 2D feature map F ∈ R H×W ×c , where c is the number of feature channels and H, W the spatial dimensions, using a pre-trained CLIP ResNet50 2D image encoder [61] . Given the corresponding depth map from that view, we compute the 3D location (X, Y, Z) for each of the H × W feature patches by averaging the depth value inside the patch extent. We map the pixel coordinate and depth value of a patch to a corresponding 3D coordinate using camera intrinsics and extrinsics and the pinhole camera model. This results in a 3D scene token set of cardinality H × W . Each scene token is represented by the corresponding patch feature vector F x,y , the feature vector at the corresponding patch coordinate (x, y), and a 3D position. If more than one views are available, we aggregate 3D scene tokens from each, to obtain the final 3D scene token set. The proprioception c is also a 3D scene token, with a learnable latent representation and the 3D positional embedding corresponding to the end-effector's current 3D location. Lastly, we map the language task instruction to language tokens using a pre-trained CLIP language encoder, following previous works [15] .Our 3D Relative Denoising Transformer applies relative self-attentions among all 3D tokens, and cross-attentions to the language tokens. For the 3D self-attentions, we use the rotary positional embeddings [18] to encode relative positional information in attention layers. The attention weight between query q and key k is written as: e q,k ∝ x T q M(p q -p k )x k , where x q / x k and p q / p k denote the features and 3D positions of the query / key, and M is a matrix function which depends only on the relative positions of the query and key, inspired by recent work in visual correspondence [62, 63] and 3D manipulation [15, 21] . We feed the final trajectory tokens to MLPs to predict: (1) the noise ϵ loc θ (o, l, c, τ i , i) and ϵ rot θ (o, l, c, τ i , i) added to τ 0 's sequence of 3D translations and 3D rotations, respectively, and (2) the end-effector opening f open θ (o, l, c, τ i , i) ∈ [0, 1] T .Training and inference During training, we randomly sample a time step t and a diffusion step i and add noise ϵ = (ϵ loc , ϵ rot ) to a ground-truth trajectory τ 0 t . We use the L1 loss for reconstructing the sequence of 3D locations and 3D rotations. We use binary cross-entropy (BCE) loss to supervise the end-effector opening f open θ , we use the prediction from i=1 at inference time. Our objective reads:L θ = w1∥(ϵ loc θ (o, l, c, τ i , i) -ϵ loc ∥ + w2∥(ϵ rot θ (o, l, c, τ i , i) -ϵ rot ∥ + BCE(f open θ (o, l, c, τ i , i), a open 1:T ),where w 1 , w 2 are hyperparameters estimated using cross-validation. To draw a sample from the learned distribution p θ (τ |o, l, c), we start by drawing a sample τ N ∼ N (0, 1). Then, we progressively denoise the sample by iterated application of ϵ θ N times according to a specified sampling schedule [32, 64] , which terminates with τ 0 sampled from p θ (τ ):τ i-1 = 1 √ α i τ i -β i √ 1-ᾱi ϵ θ (o, l, c, τ i , i) + 1-ᾱi+1 1-ᾱi β i z,where z ∼ N (0, 1) is a random variable of appropriate dimension. Empirically, we found that using separate noise schedulers for a loc and a rot , specifically, scaled-linear and square cosine scheduler, respectively, achieves better performance.At training, we segment demonstration trajectories at detected end-effector keyposes, such as a change in the open/close end-effector state or local extrema of velocity/acceleration, following previous works [57, 14, 15, 55] . We then resample each trajectory segment to have the same length T . During inference, 3D Diffuser Actor can either predict and execute the full trajectory of actions up to the next keypose (including the keypose), or just predict the next keypose and use a sampling-based motion planner to reach it, similar to previous works [14, 27, 15] . Due to space limits, please check Section B.3 for detailed model diagram, Section B.5 for our choice of hyper-parameters, Section B.6 for detailed formulation of denoising diffusion probabilistic models and Section B.7 for discussion of noise schedulers in the Appendix. 
experiments: 
 We test 3D Diffuser Actor in multi-task manipulation on RLBench [19] and CALVIN [20] , two established learning from demonstrations benchmarks, and in the real world.RLBench is built atop the CoppelaSim [65] simulator, where a Franka Panda Robot is used to manipulate the scene. On RLBench, our model and all baselines are trained to predict the next end-effector keypose as opposed to keypose trajectory; all methods employ a low-level motion planner BiRRT [66] , native to RLBench, to reach the predicted robot keypose. We train and evaluate 3D Diffuser Actor on two setups based on the number of available cameras: 1. Multi-view setup, introduced in [14] , that uses a suite of 18 manipulation tasks, each with 2-60 variations, which concern scene variability across object poses, appearance and semantics. There are four RGB-D cameras available, front, wrist, left shoulder and right shoulder. The wrist camera moves during manipulation. Only the front RGB-D camera view is available. We evaluate policies by task completion success rate, which is the proportion of execution trajectories that achieve the goal conditions specified in the language instructions [15, 14] .Baselines All compared methods on RLBench are 3D policies that use depth and camera parameters during featurization of the RGB-D input. We compare against the following: C2F-ARM-BC [13] and PerAct [14] that voxelize the 3D workspace, Hiveformer [27] that featurizes XYZ coordinates aligned with the 2D RGB frames, PolarNet [68] that featurizes a scene 3D point cloud, GNFactor [67] that uses a single RGB-D view and is trained to complete the 3D feature volume, RVT [16] and Act3D [15] , that are the previous SOTA methods on RLBench. We report results for RVT, PolarNet and GNFactor based on their respective papers. Results for CF2-ARM-BC and PerAct are presented as reported in [16] . Results for Hiveformer are copied from [68] . We retrained Act3D on the multi-view setup using the publicly available code, because we found some differences on the train and test splits used in the original paper. We also trained Act3D on the single-view setup to use as additional baseline for the single-view setup, alongside GNFactor. 3D Diffuser Actor and all baselines are trained on the same set of keyposes extracted from expert demonstrations [57] .We show quantitative results for the multi-view setup in Table 1 and for single-view setup in Table 2 . On multi-view, 3D Diffuser Actor achieves an average 81.3% success rate among all 18 tasks, an absolute improvement of +18.1% over Act3D, the previous state-of-the-art. In particular, 3D Diffuser Actor achieves big leaps on long-horizon high-precision tasks with multiple modes, such as stack blocks, stack cups and place cups, which most baselines fail to complete. All baselines use classification or regression losses, our model is the only one to use diffusion for action prediction.On single-view, 3D Diffuser Actor outperforms GNFactor by +46.7% and Act3D by +13.1%. Surprisingly, Act3D also outperforms GNFactor by a big margin. This suggests that the choice of 3D scene representation is more crucial than 3D feature completion. Since Act3D has a similar 3D scene tokenization as 3D Diffuser Actor, this shows the importance of diffusion over alternatives to handle multimodality in prediction, specifically over sampling based 3D action maps and rotation regression.Ablations We consider the following ablative versions of our model: 1. 2D Diffuser Actor, our implementation of 2D Diffusion Policy of [6] . We remove the 3D scene encoding from 3D Diffuser Actor and instead use per-image 2D representations by average-pooling features within each view. We add learnable embeddings to distinguish between different views and fuse them with action estimates, as done in [6] . See Fig. 6 in the Appendix for more details. The CALVIN benchmark is built on top of the PyBullet [69] simulator and involves a Franka Panda Robot arm that manipulates the scene. CALVIN consists of 34 tasks and 4 different environments (A, B, C and D). All environments are equipped with a desk, a sliding door, a drawer, a button that turns on/off an LED, a switch that controls a lightbulb and three different colored blocks (red, blue and pink). These environments differ from each other in the texture of the desk and positions of the objects. CALVIN provides 24 hours of tele-operated unstructured play data, 35% of which are annotated with language descriptions. Each instruction chain includes five language instructions that need to be executed sequentially. We evaluate on the so called zero-shot generalization setup, where models are trained in environments A, B and C and tested in D. We report the success rate and the average number of completed sequential tasks, following previous works [70, 58] . No motion planner is available in CALVIN so all models need to predict robot pose trajectories. We found beneficial to employ more layers of language attention, as language understanding on CALVIN is more challenging. See Fig. 4 in the Appendix for more details.Baselines All methods tested so far in CALVIN are 2D policies, that do not use depth or camera extrinsics. We compare against the hierarchical 2D policies of MCIL [71] , HULC [70] and SuSIE [45] which predict latent features or images of subgoals given a language instruction, which they feed into lower-level subgoal-conditioned policies. They can train the low-level policy on all data available in CALVIN as opposed to the language annotated subset only. We compare against large scale 2D transformer policies of RT-1 [49] , RoboFlamingo [72] and GR-1 [58] which pretrain on large amounts of interaction or observational (video alone) data. We report results for HULC, RoboFlamingo, SuSIE and GR-1 from the respective papers. Results from MCIL are borrowed from [70] . Results from RT-1 are copied from [58] .We also compare against 3D Diffusion Policy [22] and ChainedDiffuser [21] , which we train both on the language annotated training set. We evaluate 3D Diffusion Policy, ChainedDiffuser and 3D Diffuser Actor with the final checkpoints across 3 seeds. We report both the mean and standard deviation of evaluation results. We devise an algorithm to extract keyposes on CALVIN, since prior works do not use keyposes. We define keyposes as those at frames with significant motion change. Both ChainedDiffuser and 3D Diffuser Actor segment the demonstations based on the keyposes. Notably, though keyposes extracted in RLBench have clear structure since they correspond to programmatically labeled 3D waypoints [19, 57] , keyposes extracted in CALVIN are instead noisy and random, as the benchmark consists of human play trajectories.For evaluation, for each task, our model predicts at most 60 trajectories in sequence, one after the other (it conducts 60 inferences per task). Each trajectory is on average 6 action long. We did this to ensure a fair comparison to prior works that predict a maximum of 360 actions in total per task [70, 45, 72] .We show quantitative results in Table 5 : Multi-Task performance on realworld tasks.We validate 3D Diffuser Actor in learning manipulation tasks from real-world demonstrations across 12 tasks. We use a Franka Emika robot equipped with a Azure Kinect RGB-D sensor at a front view. Images are originally captured at 1280×720 resolution and downsampled to a resolution of 256 × 256.During inference, we utilize the BiRRT [66] planner provided by the MoveIt! ROS package [73] to reach the predicted poses. We collect 15 demonstrations per task, most of which naturally contain noise and multiple modes of human behavior. For example, we pick one of two ducks to put in the bowl, we insert the peg into one of two holes and we put one of three grapes in the bowl. We evaluate 10 episodes for each task and report the success rate. We show quantitative results in Table 5 and video results on our project webpage. 3D Diffuser Actor effectively learns real-world manipulation from a handful of demonstrations.We compare the inference time of our model against ChainedDiffuser [21] and 3D Diffusion Policy [22] on CALVIN, using an NVIDIA 2080 Ti graphic card. The inference speed of 3D Diffuser Actor, ChainedDiffuser and 3D Diffusion Policy is 600ms, 1170ms (50 for keypose detection and 1120 for trajectory optimization) and 581ms.Limitations Despite its SOTA performance with large margins over existing methods, our framework currently has the following limitations: 1. It requires camera calibration and depth information, same as all 3D policies. 2. All tasks in RLBench and CALVIN are quasi-static. Extending our method to dynamic tasks and velocity control is a direct avenue of future work. 3. It is on average slower than non-diffusion policies. This can be improved by recent techniques on reducing the inference steps of diffusion models.Please, refer to our Appendix for more experiments and details: Section A.1 for robustness of 3D Diffuser Actor to depth noise, Section A.2 for discussion of failure cases, Section A.3 and A.4 for descriptions of tasks in RLBench and the real world, Section A.5 for descriptions of baselines, Section A.5.1 for implementation details of re-training 3D Diffusion Policy on CALVIN, Section A.6 for keypose discovery on CALVIN. Video results can be found in our supplementary file. 
conclusion: 
 We present 3D Diffuser Actor, a manipulation policy that combines 3D scene representations and action diffusion. Our method sets a new state-of-the-art on RLBench and CALVIN by a large margin over existing 2D and 3D policies, and learns robot control in the real world from a handful of demonstrations. 3D Diffuser Actor builds upon recent progress on 3D tokenized scene representations for robotics and on generative models and shows how their combination is a powerful learning from demonstration method. Our future work includes learning 3D Diffuser Actor policies from suboptimal demonstrations and scaling up training data in simulation and in the real world. 
 
 Appendix: We evaluate the robustness of 3D Diffuser Actor to noisy depth information. Our results are presented in Table 6 . We adopt the single-view setup on RLBench, reporting success rates on 10 tasks using the front camera view. We add Gaussian noise with a variance of 0.01 and 0.03 to the 3D location of each pixel. We evaluate the final checkpoints across 3 seeds with 25 episodes for each task. With mild perturbation, 3D Diffuser Actor can still predict actions precisely, achieving an average success rate of 72.4%. We only observed 6% absolute performance drop. With strong perturbation, 3D Diffuser Actor obtains an average success rate of 50.1%. Even though 3D Diffuser Actor was trained with clean depth maps, it performs reasonably well with noisy depth information. σ = 0 (clean) σ = 0.01 σ = 0.03Table 6 : Multi-Task performance on noisy depth information. Top: We evaluate the robustness of 3D Diffuser Actor to noisy depth information. We adopt the single-view setup in RLBench, reporting success rates on 10 tasks using the front camera view. We add Gaussian noise with a variance of 0.01 and 0.03 to the 3D location of each pixel. We evaluate the final checkpoints across 3 seeds with 25 episodes for each task. 3D Diffuser Actor performs reasonably well with noisy depth information. Bottom: Visualization of the 3D location of each pixel with different levels of noise.We analyze the failure modes of 3D Diffuser Actor on RLBench. We categorize the failure cases into 4 types: 1) pose precision, where predicted end-effector poses are too imprecise to satisfy the success condition, 2) instruction understanding, where the policy fails to understand the language instruction to grasp the target object, 3) long-horizon task completion, where the policy fails to complete an intermediate task of a long-horizon task, and 4) path planning, where the motion planner fails to find an optimal path to reach the target end-effector pose.We conduct the experiment with single-view setup on RLBench, aggregating the failure cases among 10 manipulation tasks. As shown in Fig. 2 , the major failure mode is imprecise prediction of endeffector poses. This failure mode often occurs in manipulation tasks that require high precision of predicted poses, such as stack blocks, open drawer and turn tap. Confusion of language instruction is another major failure mode. This mode often occurs in the scene with multiple objects, where the policy fails to grasp the target object specified in the instruction, such as stack blocks, close jar and push buttons. On RLBench, we deploy a sample-based motion planner to find the path to reach the keypose. The motion planner could sometimes fail to find the path. Lastly, 3D Diffuser Actor might fail to complete an intermediate task, such as open drawer of put item in drawer task.In the following, we discuss ideas on how to mitigate each failure case:• Imprecise pose prediction: e.g., the model reaches the right object but fails to grasp it due to last centimeter errors (misses while executing fine motor control). We expect that aggregating datasets and training the policy on a wider and more diverse set of manipulation tasks would benefit learning finer control. • Language understanding: e.g., the model misunderstands the language instruction and grasps the wrong object. Equiping our model with perception and language understanding modules from modern VLMs, such as explicitly detecting the target objects through a VLM pre-trained on internet-scale data, would drastically improve the cognitive capabilities of our model. • Intermediate tasks: e.g., failing to open the drawer, when the task is to put something inside the drawer. We believe that decomposing a task into simpler ones through high-level planning (e.g., using LLMs) and training reward models for independent subtasks will boost our model's generalization to longer-horizon tasks. • Planner failures: we can predict the full trajectory up to the next keypose, instead of relying to a planner, as we do on CALVIN. We didn't do it on RLBench since the amount of planning errors is small.Figure 2 : Failure cases on RLBench on the setup of GNFactor.We categorize the failure cases into 4 types: 1) precise pose prediction, where predicted end-effector poses are too imprecise to satisfy the success condition, 2) instruction understanding, where the policy fails to understand the language instruction to grasp the target object, 3) long-horizon task completion, where the policy fails to complete an intermediate task of a long-horizon task, and 4) path planning, where the motion planner fails to find an optimal path to reach the target end-effector pose. Imprecise action prediction and confusion of language instruction are two major failure modes on RLBench.We provide an explanation of the RLBench tasks and their success conditions under the multi-view setup for self-completeness. All tasks vary the object pose, appearance and semantics, which are not described in the descriptions below. For more details, please refer to the PerAct paper [14] .1. Open a drawer: The cabinet has three drawers (top, middle and bottom). The agent is successful if the target drawer is opened. The task on average involves three keyposes. 2. Slide a block to a colored zone: There is one block and four zones with different colors (red, blue, pink, and yellow). The end-effector must push the block to the zone with the specified color. On average, the task involves approximately 4.7 keyposes 3. Sweep the dust into a dustpan: There are two dustpans of different sizes (short and tall). The agent needs to sweep the dirt into the specified dustpan. The task on average involves 4.6 keyposes. 4. Take the meat off the grill frame: There is chicken leg or steak. The agent needs to take the meat off the grill frame and put it on the side. The task involves 5 keyposes. 5. Turn on the water tap: The water tap has two sides of handle. The agent needs to rotate the specified handle 90 • . The task involves 2 keyposes. 6. Put a block in the drawer: The cabinet has three drawers (top, middle and bottom). There is a block on the cabinet. The agent needs to open and put the block in the target drawer. The task on average involves 12 keyposes. 7. Close a jar: There are two colored jars. The jar colors are sampled from a set of 20 colors.The agent needs to pick up the lid and screw it in the jar with the specified color. The task involves six keyposes. 8. Drag a block with the stick: There is a block, a stick and four colored zones. The zone colors are sampled from a set of 20 colors. The agent is successful if the block is dragged to the specified colored zone with the stick. The task involves six keyposes. 9. Stack blocks: There are 8 colored blocks and 1 green platform. Each four of the 8 blocks share the same color, while differ from the other. The block colors are sampled from a set of 20 colors. The agent needs to stack N blocks of the specified color on the platform. The task involves 14.6 keyposes.10. Screw a light bulb: There are 2 light bulbs, 2 holders, and 1 lamp stand. The holder colors are sampled from a set of 20 colors. The agent needs to pick up and screw the light bulb in the specified holder. The task involves 7 keyposes. 11. Put the cash in a safe: There is a stack of cash and a safe. The safe has three layers (top, middle and bottom). The agent needs to pick up the cast and put it in the specified layer of the safe. The task involves 5 keyposes. 12. Place a wine bottle on the rack: There is a bottle of wine and a wooden rack. The rack has three slots (left, middle and right). The agent needs to pick up and place the wine at the specified location of the wooden rack. The task involves 5 keyposes. 13. Put groceries in the cardboard: There are 9 YCB objects and a cupboard. The agent needs to grab the specified object and place it in the cupboard. The task involves 5 keyposes. 14. Put a block in the shape sorter: There are 5 blocks of different shapes and a sorter with the corresponding slots. The agent needs to pick up the block with the specified shape and insert it into the slot with the same shape. The task involves 5 keyposes. 15. Push a button: There are 3 buttons, whose colors are sampled from a set of 20 colors. The agent needs to push the colored buttons in the specified sequence. The task involves 3.8 keyposes. 16. Insert a peg: There is 1 square, and 1 spoke platform with three colored spoke. The spoke colors are sampled from a set of 20 colors. The agent needs to pick up the square and put it onto the spoke with the specified color. The task involves 5 keyposes. 17. Stack cups: There are 3 cups. The cup colors are sampled from a set of 20 colors. The agent needs to stack all the other cups on the specified one. The task involves 10 keyposes. 18. Hang cups on the rack: There are 3 mugs and a mug rack. The agent needs to pick up N mugs and place them onto the rack. The task involves 11.5 keyposes.We explain the the real-world tasks and their success conditions in more detail. All tasks take place in a cluttered scene with distractors (random objects that do not participate in the task) which are not mentioned in the descriptions below.1. Close a box: The end-effector needs to move and hit the lid of an open box so that it closes. The agent is successful if the box closes. The task involves two keyposes. 2. Put a duck in a bowl: There are two toy ducks and two bowls. One of the ducks have to be placed in one of the bowls. The task involves four keyposes. 3. Insert a peg vertically into the hole: The agent needs to detect and grasp a peg, then insert it into a hole that is placed on the ground. The task involves four keyposes. 4. Insert a peg horizontally into the torus: The agent needs to detect and grasp a peg, then insert it into a torus that is placed vertically to the ground. The task involves four keyposes. 5. Put a computer mouse on the pad: There two computer mice and one mousepad. The agent needs to pick one mouse and place it on the pad. The task involves four keyposes. 6. Open the pen: The agent needs to detect a pen that is attached vertically to the table, grasp its lid and pull it to open the pen. The task involves three keyposes. 7. Press the stapler: The agent needs to reach and press a stapler. The task involves two keyposes. 8. Put grapes in the bowl: The scene contains three vines of grapes of different color and one bowl. The agent needs to pick one vine and place it in the bowl. The task involves four keyposes. 9. Sort the rectangle: Between two rectangle cubes there is space for one more. The task comprises detecting the rectangle to be moved and placing it between the others. It involves four keyposes. 10. Stack blocks with the same shape: The scene contains of several blacks, some of which have rectangular and some cylindrical shape. The task is to pick the same-shape blocks and stack them on top of the first one. It involves eight keyposes.7. ChainedDiffuser [21] , a combination of Act3D [15] that predicts end-effector poses at key frames and a 3D trajectory predictor that generates intermediate trajectories to reach target end-effector poses. 8. 3D Diffusion Policy [22] , that encodes sparsely sampled point cloud into 3D representations using a PointNeXt [74] encoder. The 3D representations are average pooled into a 1D feature vector. Then, a UNet conditions on the holistic 3D scene representations and predicts the robot end-effector poses. Notably, in its original implementation, 3D Diffusion Policy is only applied to single-task experimental setup and does not condition on language instructions. For fair comparison, we update the architecture to enable language conditioning. See Section A.5.1 for more details.We compare 3D Diffuser Actor against 3D Diffusion Policy on CALVIN. However, in its original implementation, the 3D Diffusion Policy does not condition on language instructions and is applied only in a single-task setup. For a fair comparison, we enabled language conditioning in the 3D Diffusion Policy by updating its architecture with additional point-cloud-to-language cross-attention layers. We used a language encoder similar to 3D Diffuser Actor to encode language instructions into latent embeddings and applied three cross-attention layers among average pooled point-cloud features and all language tokens. To ensure the maximal performance, we do not use the simplified backbone of DP3 (Simple DP3), but the original DP3 architecture.We adopt the common setup in CALVIN, using both the front and wrist camera view. Based on the suggestion of the paper [22] , we crop a 160 × 160 and 68 × 68 bounding box from the depth map of the front and wrist camera, sampling 1024 points within each bounding box. We train 3D Diffusion Policy with a batch size of 5400 and a total epoch of 3000. Otherwise, we use the default hyper-parameters of 3D Diffusion Policy. We set the horizon of action prediction to 4, the number of executed actions to 3, and the number observed timesteps to 2. During inference, we allow the model to predict 360 times, resulting in a maximum temporal horizon of 1080 actions.For RLBench we use the heuristics from [57, 13] : a pose is a keypose if (1) the end-effector state changes (grasp or release) or (2) the velocity's magnitude approaches zero (often at pre-grasp poses or a new phase of a task). For our real-world experiments we maintain the above heuristics and record pre-grasp poses as well as the poses at the beginning of each phase of a task, e.g., when the end-effector is right above an object of interest. We report the number of keyposes per real-world task in this Appendix (Section A.4). Lastly, for CALVIN we adapt the above heuristics to devise a more robust algorithm to discover keyposes. Specifically, we track end-effector state changes and significant changes of motion, i.e. both velocity and acceleration. For reference, we include our Python code for discovering keyposes in CALVIN here:1 """Utility functions for computing keyposes.""" 2 import numpy as np 3 from scipy.signal import argrelextrem 4 5 def motion_changed(trajectories, buffer_size):6"""Select keyposes where motion changes significantly. The chosen poses shall be sparse. 3D Diffuser Actor 80 3D Diffusion Policy [22] 8Table 7 : Single-Task performance on stack-cup task of RLBench.We test 3D Diffusion Policy [22] to our 3D Diffuser Actor on the stack-cup task of RLBench. Under single-task setup, we do not need to condition on language instruction, which matches the original setup of DP3. We trained both methods on 100 episodes and tested on 25 unseen episodes.As shown in Table 7 , 3D Diffuser Actor achieves a success rate of 80%, and 3D Diffusion Policy achieves a success rate of 8%. We believe this experiment clearly highlights the importance of our tokenized 3D scene and action transformer, missing from 3D Diffusion Policy, which encodes a whole 3D point cloud into a single embedding vector.We describe the architectural differences between 3D Diffuser Actor and other 3D policies in Figure 3 . We compare 3D Diffuser Actor with Act3D [15] , PerAct [14] , and 3D Diffusion Policy [22] . PerAct encodes RGB-D images into voxel representations, while 3D Diffusion Policy average pools point cloud representations into a holistic 1D feature vector. Act3D encodes input images with a 2D feature extractor and lifts 2D feature maps to 3D. Act3D determines end-effector poses by iterating a coarse-to-fine classification procedure to identify the XYZ location of the ghost point with the highest classification score. Although our model adopts a similar 3D scene encoder as Act3D, 3D Diffuser Actor uses a relative 3D transformer to denoise end-effector poses.To ensure translation invariance in the denoising transformer, we use the rotary positional embeddings [18] to encode relative positional information in attention layers. The attention between query q i , key k j and value v j is written as: where e i,j = q T i M(p j -p i )k j , p i / p j denote the positions of the query / key, and M is a matrix function which depends only on the relative positions of points p i and p j .EQUATIONWe present a more detailed architecture diagram of our 3D Diffuser Actor in Figure 4a . We also show a variant of 3D Diffuser Actor with enhanced language conditioning in Figure 4b , which achieves SOTA results on CALVIN.The inputs to our network are i) a stream of RGB-D views; ii) a language instruction; iii) proprioception in the form of current end-effector's poses; iv) the current noisy estimates of position and rotation; v) the diffusion step i. The images are encoded into visual tokens using a pretrained 2D backbone. The depth values are used to "lift" the multi-view tokens into a 3D feature cloud. The language is encoded into feature tokens using a language backbone. The proprioception is represented as learnable tokens with known 3D locations in the scene. The noisy estimates are fed to linear layers that map them to high-dimensional vectors. The denoising step is fed to an MLP.The visual tokens cross-attend to the language tokens and get residually updated. The proprioception tokens attend to the visual tokens to contextualize with the scene information. We subsample a number of visual tokens using Farthest Point Sampling (FPS) in order to decrease the computational requirements. The sampled visual tokens, proprioception tokens and noisy position/rotation tokens attend to each other. We modulate the attention using adaptive layer normalization and FiLM [76] . Lastly, the contextualized noisy estimates are fed to MLP to predict the error terms as well as the end-effector's state (open/close).To make the paper self-contained, we present a detailed architecture diagram of Act3D in Figure 5 . Act3D first encodes posed RGB-D images into 3D scene tokens and samples ghost points uniformly from a fixed-radius sphere, centered at the current end-effector position. that gradually adds noise to a sample x. For us, x represents a sequence of 3D translations and 3D rotations for the robot's end-effector. The diffusion process is associated with a variance schedule {β i ∈ (0, 1)} N i=1 , which defines how much noise is added at each time step. The noisy version of sample x at time i can then be written as x i = √ ᾱi x + √ 1 -ᾱi ϵ where ϵ ∼ N (0, 1), is a sample from a Gaussian distribution (with the same dimensionality as x), α i = 1 -β i , and ᾱi = i j=1 α j . The denoising process is modeled by a neural network ε = ϵ θ (x i ; i) that takes as input the noisy sample x i and the noise level i and tries to predict the noise component ϵ.Diffusion models can be easily extended to draw samples from a distribution p(x|c) conditioned on input c, which is added as input to the network ϵ θ . For us, c is the visual scene captured by one or more calibrated RGB-D images, a language instruction, as well as the proprioceptive information. Given a collection of D = {(x i , c i )} N i=1 of end-effector trajectories x i paired with observation and robot proprioceptive context c i , the denoising objective becomes:EQUATIONThis loss corresponds to a reweighted form of the variational lower bound for log p(x|c) [32] .In order to draw a sample from the learned distribution p θ (x|c), we start by drawing a sample x N ∼ N (0, 1). Then, we progressively denoise the sample by iterated application of ϵ θ N times according to a specified sampling schedule [32, 64] , which terminates with x 0 sampled from p θ (x):EQUATIONwhere z ∼ N (0, 1).Existing diffusion policies [21, 6] often deploy the same noise scheduler for estimate of position and rotation. However, we empirically found that selecting a good noise scheduler is critical. We visualize the clean/noised 6D rotation representations as two three-dimensional unit-length vectors 


Decision: Summarizing the weaknesses and strengths pointed out by the reviewers:

Strengths:
- The integration of 3D scene representations with diffusion models is novel
- Strong results on CALVIN and RLBenchmark
- Good ablation analysis
- Well-written
- Experiments are well-executed including extensive comparisons to baselines

Weaknesses:
- While well-written, some minor clarity issues (see reviews for specifics)
- Some architecture choices are not well-justified (e.g., why is 3D diffusion policies only used for CALVIN and not RLBenchmark?)
- Experiments are limited to a parallel gripper
- The discussion of failure cases is rather limited
- The methodology is an incremental integration of existing methods

Overall, all 3 reviewers recommend acceptance (though weakly) and have fairly positive things to say. This appears to be a fairly strong, well-executed paper. However, the authors should take care to address the reviewers's concerns as they point out many.

**Update after rebuttal**: The authors' rebuttal responded directly to the reviewers' concerns, including updating the paper. All 3 reviewers responded to the rebuttal, 2 choosing to maintain their recommendation of weak accept and 1 moving up to strong accept. Considering this, it seems prudent to recommend this paper be accepted.

Reviews: review: title: title: Good contribution but some question/concerns
paper_summary: paper_summary: The paper presents a novel architecture for diffusion policies that aims to better leverage 3D information often available in robotic applications while also allowing for language conditioning. Consequently, the authors opt for a transformer-based architecture that leverages spatial attention mechanisms. The method shows strong performance in experiments in the CALVIN and RLBench benchmarks.
main_review: main_review: Strengths:

The presented method works well, the paper is mostly well-written, and the experiments are well-executed. One can see that a lot of effort went into preparing the paper.

Weaknesses:

I do not see a major weakness in the contribution itself. Instead, I rather have some questions/concerns about some of the results and details contained in the appendix:
* The description of the 2D diffuser policy in lines 236/237 did not allow me to understand its implementation precisely.
* The benefit of the 3D diffuser architecture in single-view setups has not been tested. Table 3 is only concerned with the multi-view setup
* Why did the authors decide to employ 3D Diffusion Policies only in the CALVIN and not the RLBench benchmark? I'm asking because this seems to be the most closely related architecture.
* I could not understand the discussion around the temporal horizon in lines 282-286. Could the authors more clearly state the number of predicted actions per inference and the maximum number of executed actions? The current description mixes the concept of actions and trajectories.
* In line 290/291, the authors state that "allowing 3D Diffuser Actor to have a longer horizon before terminating increases the performance significantly, which suggests that the model learns to retry under failure." a) Can this longer horizon impact the fairness of comparison? b) How does the model perform with a shorter horizon? c) Did you verify that the model indeed retries under failure by inspection of rollouts?
* I disagree with the statement about control frequency. If the delay between observation and action is 600ms, the control frequency is 1.6Hz even if the prediction contains 6 endeffector poses. The simple reason is that the policy cannot react to unforeseen events during the 600ms delay. The number of predicted endeffecor poses is simply a parameterization choice. The policy could also predict 20 endeffector poses and would not be more reactive.
* The robustness evaluations in Appendix A.1 raise the question of whether training with noisy depth measurements could improve the robustness of the policy. Did the authors test such a training scheme?
* The authors should more clearly clarify that they are using a different model for the CALVIN benchmark (only stated clearly in Appendix 4b if I did not miss a statement in the main paper). Furthermore, I would like to see the performance of the "default" architecture in Figure 4a.
questions: questions: Improvements:
* A visualization of the 2D diffuser ablation similar to the visualizations in Figure 4 should be added.
* Please correct the statement about control frequency
* I would like to see ablations of the method in the single-view RLBench benchmark. After all, there may be many robotic tasks with a single view; hence, it is important to consider the architectural choices in this domain.
* I would like to see the performance of the "default" 3D diffuser policy presented in the main paper (in addition to the improved version).
* I would be happy if the authors can address my questions regarding the longer 3d diffuser horizon in the CALVIN benchmark.
* I would like to see an ablation against 3D Diffusion Policy in the stack cups task of the real-world evaluation as it seems to be one of the tasks with which the proposed methods struggles but which does not rely on language (please correct me if I am wrong here).

POST REBUTALL UPDATE:

The authors addressed all my points except for the last one. Hence I increased my score to a strong accept.
limitations: limitations: 3
review_summary: review_summary: As detailed, I like the paper except for some improperly presented findings and lacking ablations. If the authors can provide the missing details and fix the presentation accordingly, I am happy to increase my score to a strong accept.
title: title: Review
paper_summary: paper_summary: The paper presents a novel approach to robot manipulation policies by combining 3D scene representations with diffusion objectives, named 3D Diffuser Actor. This approach has demonstrated significant performance gains over existing methods on benchmarks like RLBench and CALVIN, as well as in real-world applications.
main_review: main_review: # Strengths

1. The integration of 3D scene representations with diffusion models is novel. The authors have successfully demonstrated how this combination can lead to improved policy learning for robotic manipulation tasks.

2. The authors have conducted extensive experiments, including comparisons with current SOTA policies and ablation studies. This thorough validation helps to build confidence in the proposed method's effectiveness. Besides, real robot experiments are also included.

3. The paper is well-structured, with a clear introduction, related work, methodological details, experiments, and conclusions. The figures and tables are informative.

# Weaknesses

1. The authors have shown impressive results on tasks with a parallel gripper. I am curious that whether 3D Diffuser Actor could also perform well on tasks with high-dimensional action space, such as dexterous manipulation tasks used in 3D Diffusion Policy? It would better show the generality of 3D Diffuser Actor.

2. Limited Discussion on Failure Cases: Although the paper briefly mentions failure cases, a more detailed analysis of why the model fails in certain situations and how these failures can be mitigated would be beneficial.
questions: questions: Please see weaknesses.
limitations: limitations: 3
review_summary: review_summary: I would possibly recommend accept since this paper presents strong empirical evaluations on 3D Diffuser Actor.
title: title: Good, well-written paper
paper_summary: paper_summary: The paper presents a novel architecture for diffusion policies operating on point clouds. They compare against a large selection of baselines, and show substantial improvements in success rates.
main_review: main_review: Good, well-written paper with thorough evaluations. The paper is mainly an integration of two approaches: relative 3d transformer which has translationally invariant attention layers, and diffusion policy.

Strengths:
1. Strong results on the RLBench and CALVIN benchmarks demonstrating the importance of both 3d representations and diffusion models for performance.
2. Several ablations including comparison of policies trained from a single vs multiple RGBD views, as well as some analysis of policy robustness to noisy depth maps.
3. Thorough comparison to 6 baselines. 
4. The paper is very clearly written with only minor details missing.
5. The method is likely to become an essential baseline for learning from demonstrations with RGBD observations.

Weaknesses:
1. The action representation feels unnatural for point clouds, see next section.
2. The main reason for a weak rather than a strong accept is that the work is an incremental integration of two existing approaches (relative 3d transformer and diffusion policies) and hence the contribution is not as original.
questions: questions: 1. I find the choice of action representation a bit odd: singling out the discrete open/close action, and having separate noise schedules for the target orientation and position seems like a kludge. Have you considered representing actions as a bunch of 3d keypoints on the end-effector? That is a much more natural action representation for point clouds.

2. Why do you use horizon of length 1 for RLBench experiments? The ability to model trajectories is one of the main benefits of using diffusion, at the very least it would be good to show that your results hold when you predict longer trajectories.

3. Regarding comparison to DP3. Why did you choose to evaluate DP3 on CALVIN instead of RLBench? The DP3 paper doesn't consider language conditioning so it's unclear if this evaluation is meaningful. RLBench seems closer to the evaluations considered in DP3. (Note that this is not important for score given that DP3 was a concurrent work but it's good to clarify this.)

4. I couldn't find information about model sizes in the paper, please add those.
limitations: limitations: 3
review_summary: review_summary: Accept