ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation
Authors: Zhou Xian
Nikolaos Gkanatsios
Theophile Gervet
Tsung-Wei Ke
Katerina Fragkiadaki
Venue: CoRL 2023 Poster (2023)
Keywords: Manipulation
Imitation Learning
Transformers
Diffusion Models

Abstract: We present ChainedDiffuser, a policy architecture that unifies action keypose prediction and trajectory diffusion generation for learning robot manipulation from demonstrations. Our main innovation is to use a global transformer-based action predictor to predict actions at keyframes, a task that requires multi- modal semantic scene understanding, and to use a local trajectory diffuser to predict trajectory segments that connect predicted macro-actions. ChainedDiffuser sets a new record on established manipulation benchmarks, and outperforms both state-of-the-art keypose (macro-action) prediction models that use motion plan- ners for trajectory prediction, and trajectory diffusion policies that do not predict keyframe macro-actions. We conduct experiments in both simulated and real-world environments and demonstrate ChainedDiffuser’s ability to solve a wide range of manipulation tasks involving interactions with diverse objects.

ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation
Zhou Xian", Nikolaos Gkanatsios", Theophile Gervet", Tsung-War Wei Ke, Katerina Fragkiadaki
School of Computer Science
Carnegie Mellon University
(xianzl, ngkanats, tgervet, taungwek, katef facs cmu, edu
chained-diffused github.
Abstract: We present ChainedDiffuserr a policy architecture that unifies action keypose prediction and trajectory diffusion generation for learning robot manipu- lation from demonstrations. Our main innovation to use a global transformer- based action predictor to predict actions at keyframes, a task that requires multi- modal semantic scene understanding, and to use local trajectory diffuser pre- dict trajectory segments that connect predicted macro-actions. ChainedDiffuser sets new record on established manipulation benchmarks, and outperforms both state-of-the-arr keypose (macro-action) prediction models that use motion plan- ners for trajectory prediction, and trajectory diffusion policies that do not predict keyframe macro-actions. We conduct experiments in both simulated and real- world environments and demonstrate ChainedDiffuser's ability solve wide range of manipulation tasks involving interactions with diverse objects.
Keywords: Manipulation, Imitation Learning, Transformers, Diffusion Models
Introduction
While learning manipulation policies from demonstrations is a supervised learning problem, the multimodality and diversity of action trajectories poses significant challenges to machine learning methods. Some tasks, such as placing cup in a cabinet, can be handled by a policy that provides only a desired goal pose for the cup [1, 2, 3], while others, such as wiping off dirt on the floor, necessitate the policy to generate continuous action trajectory [4, 5] for the grasped mop.
One line of manipulation learning methods models action trajectories from demonstrations. These methods either reactively map vision and language to dense temporal actions [6, 7, 8, 5, 9], or
model the input-action compatibility using energy-based models [10. 11, 12. 13]. Despite recent progress, these methods may struggle with multimodal action trajectory distributions, experience training stabilities [14, 13, 15]. Building on successes in diffusion models [16, 17, 18], recent line work proposes to train diffusion- policies [14, 4, 19] for generating action trajectories. These approaches have demonstrated stable training behavior and impressive capability in captur- ing multimodal action trajectory distributions. Yet, they have not yet been tested on long-horizon manipulation tasks.
Another line of works casts the problem of robot manipulation as predicting sequence of discrete end-effector actions on keyframes [1, 20, 12, 21J. This paradigm extracts keyframes from continuous demonstrations and predicts end-effector actions these keyframes [2, 22, 3, 21]. Subsequently, a
low-level path planner connects the predicted keyposes (mucro-actions), and returns full trajectories
Equal contribution
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

that adhere to both environmental and task constraints. Leveraging recent advances in attention- based architectures [23], a number of methods extend keyframe action prediction to 6-DoF language- instructed manipulation tasks [2, 3, 22, 24, 21].
The assumptions behind keyframe prediction hinder its applicability to manipulation tasks that ex- tend beyond pick-and-plas type of actions. Many tasks, such wiping table, opening door while respecting the kinematic constraints, etc., can only be solved via continuous interactions with the environment. Moreover, the dependence on low-level path planning further restricts these meth- ods' capability: while a range of tasks need collision-free trajectories, other tasks, such as object pushing [24, 3, 25], necessitate that the motion planner disregards collision avoidance. Although supervision for this additional reasoning is readily available in simulated datasets [26], real- world human demonstrations typically lack such data, not mention that collision-free motion planning in the real world requires accurate state estimation, which presents its own challenges.
In light of the above, we present ChainedDiffiser, a neural architecture that unifies the two afore- mentioned paradigms. ChainedDiffuser is a policy architecture that takes as input visual signals and, optionally, a language instruction and outputs temporally dense end-effector actions. At a coarse level, it predicts macro-step end-effector actions (which we will call macro-actionss, a high- level task that requires global comprehension of the visual environment and the task complete, with global transformer-batt cal trajectory segments to connect the predicted macro-actions. In comparison to transformer based macro-step step prediction methods [2. 3, 22, 24], our model predicts smooth trajectories to accommo- date tasks that require continuous interactions and collision-free actions. In comparison to diffusion- only trajectory generation methods [14, 15, 4, 19], our hierarchical approach handles long-horizon tasks in a more structured manner and allows different modules to concentrate on the tasks which they excel.
action predictor. Then, low-level trajectory diffuser generates lo-
We test ChainedDiffuser on RLBench [26], an established benchmark for manipulation learning from demonstrations. We evaluate our model across a variety of tasks and scenarios studied in previous literature [22, 24]. ChainedDiffuser sets a new state of the art, and outperforms ablative versions that do not predict macro-actions or use regression or motion planners for keyframe-to- keyframe trajectory prediction. Furthermore, we validate our model in real- world scenarios with a number of long-horizon manipulation tasks, using handful of human demonstrations for training.
2 Related Work
Learning from Demonstrations [27, 28] has been common paradigm for robotics but requires demonstration data collection in the real world [6, 29, 30] or simulation [26, 31, 32]. To improve data efficiency, several approaches learn the policy on top of pre-trained visual representations that ex- ploit large vision-only datasets [33, 34, 35, 36, 37, 38). Orthogonal to this, other approaches abstract every task sequence of subgoals, expressed pick-and-place primitives [1, 2] keyframes [39, 40]. In this case, hand-designed lowelevel controllers are employed to plan the end-effector's motion between intermediate subgoals. While data-efficient, this abstraction does not generalize adequately to scenarios where only few specific trajectories that respect all physical constraints are valid [41], such as manipulations of deformable [42, 43] or articulated [44] objects, motions of closed-chain robotic systems [45, 46], or trajectories through obstacles in a cluttered environment [47]. As result, recent works resort to cemi-manual cost specification for each additional constraint (e.g., collision avoidance, trajectory smoothness [4]). Closer our approach, James and Abbeel [41] learn to score trajectories proposed by either hand-designed or learning based planners. Instead, train scene conditioned diffusion models to generate trajectories that connect predicted keyposes.
Transformers for Robotics Following their success in natural language processing [23, 48, 49] and computer vision [50, 51], numerous recent works use Transformer-basee architectures for robotics and control [52, 53,6 6, 54, 55, 21]. One main motivation is the flexibility of attention for long-horizon prediction when combining information from multiple sensory streams, such visual observations and language instructions [56, 22j. Most related to ours is the stream of multi multi-tasking Transformer-

---

Task
bj Unprojected 3D Feature Cloud
Goal Macro-Action
"Greep the towel and pubse"
Macro Macro-Action
Gamer
Macro Action Predictor
Local Trajectory Difluser
Language Features
4
3D Visual Fentures
00000
Qhopo
Dunoiing
Trajectory Noisa
Goal Trajectory
Language Encoder
Query
Encoder
"M
Relattve 3D Cross Attension
000000
Propriccapioon
Detection Transformer Decoder
Macro-Action
Figure 1: ChainedDiffuser is robot manipulation policy architecture that predicts a set robot keyposes and links them using predicted trajectory segments. It featurizes input multi-view images using pre-trained 2D image backbones and lifts the resulting 2D feature maps to 3D using sensed depth. In (b) we visualize the 3D feature cloud using PCA and keeping the 3 principal components, mapping them RGB. The model then predicts end-effector keyposes using coarse-to-fing attention operations estimate end-effe und-effector's 3D location and regress the robot's 3D orientation, similar to [21] (d). It then links the current end-effector pose to the predicted one with trajectory predicted using diffusion model conditioned on the 3D scene feature cloud and predicted keypose (e).
action map for the
based models, that are trained diverse datasets to achieve higher in-distribution [3. 24] or out- of-distribution generalization [6, 57, 58, 59]. Our model comprises of two attention-based modules, one for macro-step action prediction and one for local trajectory optimization, that can leverage different input modalities and operate over different abstractions.
Diffusion Models [60, 18, 16, 17] learn to approximate the data distribution through an iterative denoising process, and have shown impressive results on both unconditional and conditional image generation [61, 62, 63, 64). In the field of robotics, diffusion models find applications planning [15, 65, 66]. scene re-arrangement [67, 68], controllable motion optimization [69, 70], video gener- ation [71] and imitation learning [14, 19]. Their main advantage is that they can better capture the action trajectory distribution compared to previous generative models. Recent works use diffusion model to predict complete trajectories, often auto-regressively [14, 72). Instead, we use diffusion models to generate local trajectories that are chained together with macro-actions.
ChainedDiffuser
3.1 Overview
The architecture of ChainedDiffuser is illustrated Figure ChainedDiffuser combines macro- action prediction with conditional trajectory diffusion. Its input comprises of visual observations of the environment and natural language description 1 of the task. At each step, ChainedDiffuser predicts macro-action à, using a global policy Tglobal, and then feeds å, together with its current
end-effector state qi low-level local trajectory generator Tlocal(g, à,) to generate dense micro-
actions connecting qe and â shown in Figure 1. Both a, and ge share the same space A

---

{apos: Grot, agriph, ), consisting of the end-effector's 3D position apos, rotation Ortt represented as 4D quaternion, and a binary flag Ogrip indicating whether the gripper is open. For each task, we assume access to a dataset 'D = (2........ .m.......... Sm m expert demonstrations, where S contains the language instructions I, visual observations and end-effector states ạt. for all timesteps in the demonstration.
Input Encoding ChainedDiffuser operates in 3D space to achieve robustness across changing camera viewpoints an important advantage over prior 2D methods which assume fixed camera viewpoints [22, 24, 14]. Compared to prior robotic architectures which rely on voxel-based 3D) representation (e.g.. [3, 40)), ChainedDiffuser employs a point-based representation, that facilitates sparse computation and cireumvents precision loss during voxelization. ChainedDiffuser uses a frozen CLIP [73] encode both the language instruction and the RGB images or into set of language and visual feature tokens respectively. Then, it uses the depth channel information to unproject the 2D image feature tokens into a 3D feature cloud (Figure ((b)), where each visual token has 2D appearance information and 3D positional information. We also encode the proprioception information with a simple MLP.
3.2 Macro-Action Predictor
Our macro-action predictor Tglobal is based on Act3D [21], state-of-the-an macro-action prediction method that uses a point-based transformer that casts end-effector action prediction as 3D action map prediction. We include its main pipeline here for completeness. Act3D samples iteratively 3D point candidates and featurizes them using relative position attentions to a scene 3D feature cloud. Then, a trainable query token Zquery is used to score a pool of N point candidates {P, = (r,, Vr. 2;) )/ in the scene and select a position for next macro-action. The point candidates are first uniformly sampled within the robot's empty workspace and only contain 3D positional information and trainable feature embedding Zpoint- The query token and the point candidates individually attend the concatenation (across the sequence dimension) of language tokens Zins, visual feature tokens Zvia and proprioception token Zrobot (Figure I(d)):
E Equery Attn Zquary, (Zins; Zviss Zrobot))
poinn
Attn Zpoints (2mssy Zvisy Zrobot)
where Attn(z, y) an attention operation [23, 74] where the queries are formed from æ, the keys and values from y. After this comtextualization step, the query token and the point candidates have captured the task and scene information. We take the dot product of the contextualized query em- bedding with all point candidates and select the best-matching point candidate for the position of the predicted macro-action:
âpos (I Mi. 22),
argmax
ŽT query 2point
(3)
Once obtain the best point candidate, we predict the rotation and gripper open flag with simple MLP top of the query:
(āroe, ågrip) AMLP(2quen))
(4)
3.3 Local Trajectory Diffuser
Once we obtain the macro-action à for the current step t, we call upon our diffusion- based local trajectory generator fill up the gap in-between with micro-actions. We model such trajectory gen- eration denoising process [18, 14, 4]: start with drawing sequence of S random Gaussian samples {x_ K.S. } ===1 in the normalized SE(3) space, and then perform K denoising iterations to trans- form the noisy trajectories to a sequence of moise-free waypoints {x215_ Each denoising iteration described by: k-1
x
>x(xx; Tx k)) +0,, 15s58
(5)

---

where is the noise prediction network, &o the denoising step, NN,. the Gaussian noise added at each iteration, and Abo Ti, Th are scalar noise schedule functions dependent on k (Appendix 7.1).
The noise prediction network (Figure (e)) is also an attention-based model that absorbs similar input as the macro-action selector does, i.e., the language instruction RGB-D observations or and current end-effector state qt. but additionally conditions on the goal macro-action å and the denois- ing timestep k. The language tokens Zios+ visual tokens vvss and current end-effector state zobot are featurized similarly to the Macro-Action Selector. We use an MLP to encode the goal macro- action into Zmacro = Mā+( We encode the denoising timestep into Ztime using sinusoidal positional embeddings [23], and encode the the sampled noise using an MLP into a sequence of tokens z We let this sequence iteratively cross-attend to all encoded inputs first:
- = Aunn 2 (Zmss Zviss Zrobots zmacro Zrime),
and then self-attend obtain finalized clarity):
- (note that reuse the same symbol for presentation
こ = Attn 2% 25)
Again, we use relative positional embeddings to encode all tokens' spatial positions. For the tra- jectory noise tokens, we additionally encode each sample's temporal position using sinusoidal positional embeddings. These are added to the respective noise tokens z The contextualized noise sample is then fed into another MLP for noise regression:
cx k k) MLP(2*)
After K denoising steps by substituting Equation 6 into 5, we convert the denoised samples back to the actual micro-actions by unnormalizing them: at-1+s Unnormalize xx"), more implementation and training details, please see the Appendix 3.4.
1 S s S S. For
Noise schedulers We model local trajectory optimization as a discrete-time diffusion process, which we implement using the DDPM sampler [18]. DDPM uses non-parametric time-dependent noise variance scheduler 3e, which defines how much noise is added at each time step. We adopt a scaled linear schedule for the position and a squared cosine schedule for the rotation of each trajectory step.
3.4 Implementation and Training Details
ChainedDiffuser takes input m multi-view RGB-D images of the scene. For experiments in simulation, we m = 3 (left, right, wrist) on the settings of the baselines compare with. For real-world experiments, single front-view camera. Each RGB-D image is 256 x 256 and is encoded to 64 x 64 visual tokens with CLIP''s ResNet50 visual encoder [73]. The demonstration data contains end-effector states for all timesteps. In order to extract macro-actions to supervise the action selection transformer, we use a simple heuristic following previous literature [3, 22, 24]: a timestep is considered be a keyframe containing macro-action if the gripper opens or closes, or if the robot arm not moving (when all joint velocities approach zero). All dense actions present in the demonstration the local trajectory difffuser We resample the dense trajectories between extracted macro-actions to a trajectory of fixed length S = 50. We found in practice, denoising fixed number of micro micro-actions actions leads to more stable training, and works better than learning variable-length trajectory diffusion with predicted trajectory length. We train both the action detector and the trajectory diffuser jointly, using a cross-entropy (CE) loss to supervise the point candidate selection by predicting a probability distribution g over all point candidates in the pool, and mean- -egared error (MSE) losses supervise quaternion, gripper opening and trajectory noise regression:
(with additional front view), depending
use k = 1, with
used to supervise
i+8-1 MSE(co (x S A), cx)
-=
CE(G((P))M) ((P:))))) + MSE(G, 07) +
Σ
CED lec
5

---

Simulated Tasks RLBanch
2aptop
"pot
up
"unplog charger"
"bang the Zrane on the hanger"
"aster the plans"
"wipe the ciet off the deak*
"cake shoes ouz of the box"
"stack the other oups or of the red cup"
b) Real-world
"stack the cups the bzown
rollez spread blue dough"
olose it"
splilled coffee"
crbe"
Figure 2: Simulation and real-world tasks evaluate
where indicates predicted value, à (ārot dgrip): ) k is randomly sampled denoising step, and ca the sampled ground truth noise. In order to speed up training in practice, we train the first 2 terms till convergence, and then add the 3rd term for joint optimization, as opposed to using ground-truth macro-actions for training the trajectory diffuser. This allows the trajectory diffuser to incorporate certain error recovery capability to handle inaccurate macro-action predictions. In addition, at test time, we normalize the predicted quaternion to ensure it respects the normalization constraint before feeding it to the robot. We use batch size of B 24 and AdamW [75] optimizer with learning rate of le 4 for all experiments. Our single-task model is trained for day A100 GPU, and multi-task model is trained for 5 days 4 A100 GPUs.
Control Our control algorithm is closed-loop at the macro-action level, which means the macro- action predictor reasons about the surroundings and predicts actions to handle environment changes. At the low-level, our controller open-loop and follows Cartesian-space end-effector trajectory composed of the predicted micro-actions using position control, with control frequency of 10Hz. For our real-robot setup, low-level PID controller at 1kHz.
the open source frankapy package [76] with ROS, which uses
4 Experiments
We test ChainedDiffuser in various manipulation tasks in both simulated and real- world environ- ments. Our experiments aim to answer the following questions:
How does ChainedDiffuser compare to previous SOTA 2D and 3D manipulation methods?
Is macro-action prediction helpful in guiding trajectory generation?
Does ChainedDiffuser work in the real-world where only single camera and limited num- ber of demonstrations are available?
4.1 Simulation Experiments
We experiments in simulation using RLBench [26], widely adopted manipulation benchmark
with diverse tasks concerning interactions with wide range of objects, as shown in Figure 2. We

---

Table 1: Success rates in 10 single-tasks of the Hiveformer experimental setting.
pick & lift
pick-up
push button
put knife
put money
stack wine
take noney
Meun
target
block
unbrella
Auto [77]
87
78
95
31
62
100
36
23
38
37
55.0
HiveFormer [24]
92
100
82
InstructRL [22]
98
85
100
85
99
100
98
93
90
93.8
ChainedDiffuser (curs)
98
96
100
95
90
100
95.8
follow the same setting used in prior works [22, 24], where each task has multiple variations and contains 100 demonstrations. report success rates each task averaged over 100 unseen test episodes. For baselines, when possible, we use the official numbers reported in their papers.
Baselines We compare ChainedDiffusse with the following baselines:
1. Auto-A [77] and HiveFormer [24], policy learners that operate multi-view 2.5D images and predict actions by offseting detected points in the input images.
2. InstructRL [22], a policy that operates multi-vier 2D images with pre-trained vision and language encoders, and directly predicts 6-DoF end-effector actions.
3. Ac13D [21], policy that predicts keyframe end-effector macro-actions with 3D action detection transformer and relies on low-level motion planner to connect macro-actions,
4. Open-loop trajectory diffusion, which is ChainedDiffuser without the macro-action detec- tor, making it trajectory diffusion model.
5. Act3D+ trajectory regression, which replaces the local trajectory diffuser in ChainedDif- fuser with deterministic trajectory regression
Dataset We consider the following single-task task experimental settings:
10 tasks considered in the Auto-A [77] experimental setup. These tasks are considered by many prior works and this allows us to compare our performance with them.
10 tasks in RLbench we identify to require continuous interaction with the environment, such as wipe_destt where a wiping trajectory is needed to remove the dirt from a desk, and open_fridge where a local trajectory needs to adhere to the kinematic constraint when the robot is grasping the door handle. Most tasks RLBench can be reasonably solved with only macro-action prediction and motion planners. This set of tasks consider highlights the limitation of these approaches.
Results We train single-task ChainedDiffuser and the baselines. For Auto-A, HiveFormer and InstructRL we use the numbers reported in the corresponding papers. We show quantitative results in Tables and 2. ChainedDiffuser consistently achieves better performance than prior methods on all task categories. On the set of challenging tasks for motion planners, ChainedDiffuser gives significant boost of 60% on average. ChainedDiffuser improves upon open-loop trajectory diffusion model, which demonstrates that delegating global macro-action prediction high-level policy to guide local trajectory diffusion helps. Act3D+ trajectory regression struggles where multi-modal trajectories are present in demonstrations, e.g. cup_in wtw.iincccaaa. cabinet where multimodal trajectories exist for grasping the cup and feeding into the cabinet in the training set. This demonstrates that modeling trajectory generation as a multi-step denoising process is advantageous over regression- -aased model, which aligns with conclusions from previous literature [14].
4.2 Real-world Experiments
We conduct experiments with a real-world setup, using Franka Emika Panda robot with a parallel- jaw gripper. We use single Azure Kinect camera collect front-view RGB-D image input. See Ap- pendix 7.4 for more details on our hardware and data collection setup. We design 7 tasks that involve
7

---

Table Success rates on challenging tasks for motion planners.
unplug charger
open box
open fridge
open oven
en shelf
wipe desk
cup cabinet of box
Meun
door
banger
Act3D [21]
48
9
9
19
66
2
34
0
19
21.0
Open-loop trajectory diffusion
37
31.7
Act3D trajectory regression
95
5
95
60
77
68
70
40
67
59.6
ChainedDDiflserr (ours)
95
76
96
68
85
86
92
68
78
80.9
multi-step actions and continuous interactions with the scene are shown Figure 2), collected 10 20 demos for each tasks, and train multi-task ChainedDiffuser for real-world deployment. We refer the reader to supplementary video for qualitative executions of the robot. We eval- uate it on 10 episodes for each task, and re- port success rates Table 3. ChainedDiffuser able to perform reasonably well on most of the tasks, even for tasks with multiple action modes and skills. The most common failure case is caused by noisy depth image: we lever- age point selection for macro-action prediction, which would suffer from incorrect depth esti- mation the real world. This could potentially be resolved by more accurate camera calibra- tion with multi-view camera setup and learning to recover from noisy input, which we leave as our future work.
Task
# Train
Success
put mask in kit
20
6/10
fold and wipe coffee
20
8/10
stack cups spread dough fold and wip beans
15
7/10
15
7/10
20
6/10
put nails box
20
6/10
press stapler
10
10/10
Table Real-world tasks.
4.3 Limitations
Our method currently has the following limitations: 1) Our trajectory diffuser is conditioned on end-effector poses in SE(3) space. It would be ideal to extend it to full joint configuration space for more flexible trajectory prediction. 2) Our model performs closed-loop control on the macro-action level, which restricts its flexibility in highly dynamic environments. That said, our framework can be easily extended with closed-loop re-planning at the micro-action level, making the policy more robust to environment dynamics, which we leave as our future work. Following the standard setting in RLBench, our method assumes access to calibrated cameras. We believe this assumption valid as mobile robots performing household tasks for humans in the future should have cameras attached to the robots, where these cameras can be calibrated when coming out of the factories.
5 Conclusion
We presented ChainedDiffuser, neural policy architecture for learning 6-DoF robot manipulation from demonstrations. Our model achieves competitive performance various task settings, in both simulation and the real-world, Our experiments demonstrate that by unifying both transformer- based macro-action detection and diffusion-based trajectory generation, ChainedDiffisser achieves the best of both families and addresses their respective limitations. ChainedDiffuser outperforms both keyframe prediction methods and trajectory diffusion alone, which justifies their unification our framework. sets a new ofute-of-the-arr in RLbench, and especially improves performance on contact-rich tasks and tasks that involve articulated objects, where methods that rely hand designed planners typically struggle.
6 Acknowledgements
This work is supported by Sony AI, NSF award No 1849287. DARPA Machine Common Sense, Amazon faculty award, and an NSF CAREER award.

---

References
[1] A. Zeng, P. R. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, I. Armstrong, I. Krasin,
D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning, 2020.
[2] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu-
lation. ArXiv, abs/2109.120990 2021.
[3] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMIR, 2023.
[4] J. Urain, N. Funk, G. Chalvatzaki, and Peters. Se (3dddiiffussiof::eld::: Learning cost functions for joint grasp and motion optimization through diffusion. arXiv preprint arXiv:2209.0388 2022.
[5] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108. 03298, 2021.
[6] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068177 2022.
[7] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imita- tion learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628-5635. IEEE, 2018.
[8] M. Sicb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki. Graph-structured visual imita- tion. In Conference on Robot Learning, pages 779.989. PMLR, 2020.
[9] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems, 35:22955-22968, 2022.
[10] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168, PMLR, 2022.
[11] D. Jarrett, I. Bica, and M. van der Schaar. Strictly batch imitation learning by energy-based distribution matching. Advances in Neural Information Processing Systems, 33:7354-7365, 2020.
[12] N. Gkanatsios, A. Jain, Z. Xian, Y. Zhang, C. Atkeson, and K. Fragkiadaki. Energy- based models as zero-shot planners for compositional scene rearrangement. arXiv preprint arXiv:2304. 14391, 2023.
[13] Y. Du, S. Li, J. Tenenbaum, and I. Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316. 2020.
[14] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy:
Visuomotor policy learning via action diffusion. arXiv preprint 23X 2023.
[15] M. Janner, Y. J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint vrXiv:2205.09991 2022.
[16] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv:0010.02502, 2020.
arXiv preprint
[17] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Learning, pages 8162-8171. PMLR, 2021.

---

[18] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[19] M. Reuss, M. Li, X. Jia, and R. Lioutikov. Goal-conditioned imitation learning using score- based diffusion policies. arXiv preprint arXiv:2304.025:2, 2023.
[20] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 4568 4575. IEEE, 2021.
[21] T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3d: Infinite resolution action detec- tion transformer for robotic manipulation. Conference on Robot Learning, 2023.
[22] H. Liu, I.. Lee, K. Lee, and P. Abbeel. Instruction-following agents with jointly pre-trained vision-language models. arXiv preprint arXiv:2210.134311 2022.
[23] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, N. Gomez, E.. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[24] P.-.. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven history-aware policies for robotic manipulations. Conference on Robot Learning, pages 175-187, PMLR, 2023.
[25] H.-Y. F. Tung, Z. Xian, M. Prabhudesai, Lal, and K. Fragkiadaki. 3d-oes: Viewpoint- invariant factorief environment simulators. arXiv preprint arXiv:201 1.06464, 2020.
[26] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-33) -0266,00662200 2020.
[27] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation.
Neural Computation, 3:88-97, 1991.
[28] C. Atkeson and S. Schaal, Robot learning from demonstration, In International Conference on Machine Learning, 1997.
[29] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. H. Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, and K. Hausman. Open-world object manipulation using pre- trained vision-language models. ArXiv, abs/2303.00905,, 2023.
[30] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang. J. Singh, C. Tan, M. Dee, J. Per- alta, B. Ichter, K. Hausman, and Xia. Scaling robot learning with semantically imagined experience. ArXiv, abs/2302. 1550, 2023.
[31] Mu. Z. Ling, F. Xiang. D. Yang, X. Li. S. Tao, Z. Huang, Z. Jia, and H. Su. Maniskill: Gen- eralizable manipulation skill benchmark with large-scale demonstrations, In NeurIPS Datasets and Benchmarks, 2021.
[32] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and I.. Fan. Vima: General robot manipulation with multimodal prompts. ArXiv, abs/2210.03094, 2022.
[33] Xiao, Radosavovie, T. Darrell, and J. Malik. Masked visual pre-training for motor control. ArXiv, abs/2203.06173. 2022.
[34] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. ArXiv, abs/2210.03109 2022.
10

---

[35] Y. Seo, D. Hafner, H. Liu. F. Liu, S. James, K. Lee, and P. Abbeel. Masked world models for visual control. ArXiv, abs/2206.14222 2022.
[36] Y. Seo, J. Kim, S. James, K. Lee, Shin, and P. Abbeel. Multi-view masked world models for visual robotic manipulation. ArXiv, abs/2302.02408, 2023.
[37] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. K. Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning, 2022.
[38] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. In Conference on Robot Learning. 2022.
[39] S. James and A. J. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, PP:1-1, 2021.
[40] S. James, K. Wada, T. Laidlow, and A. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11739-13748, 2022.
[41] S. James and P. Abbeel. Coarse-to-fine q-attention with learned path ranking. ArXiv, Abs/2204.01571, 2022.
[42] Z. Xian, B. Zhu, Z. Xu, H.-Y. Tung, A. Torralba, K. Fragkiadaki, and C. Gan. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. arXiv preprint arXiv:2303.02344. 2023.
[43] Z. Xu, Z. Xian, X. Lin, C. Chi, Z. Huang, C. Gan, and S. Song. Roboninja: Learning an adaptive cutting policy for multi-material objects. arXiv preprint arXiv:2302. 1553, 2023.
[44] H. Geng, 7. Li, Y. Geng, J. Chen, H. Dong, and H. Wang. Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations. ArXiv, abs/2303.16958. 2023.
[45] F. Suárez-Ruiz, X. Zhou, and Q.-C. Pham. Can robots assemble ikea chair? Science Robotics, 3(a7)ee 2018.
[46] Z. Xian, P. Lertkultanon, and Q.-C. Pham. Closed-chain manipulation of large objects by multi-arm robotic systems. IEEE Robotics and Automation Letters, 2(4):1832-1839) 2017.
[47] A. Fishman, A. Murali, C. Eppner, B. N. Peele, B. Boots, and D. Fox. Motion policy networks. In Conference on Robot Learning, 2022.
[48] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.0480s; 2019.
[49] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, Neelakantan, Shyam, Sastry, A. Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[50] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[51] K. He, Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[52] W. Liu, C. Paxton, T. Hermans, and D. Fox. Structformer: Learning spatial structure for language-guu guided semantic rearrangement of novel objects. 2022 International Conference Robotics and Automation (ICRA), pages 6322-6322 6329, 2021.

---

[53] I. Radosavovic, T. Xiao, B. Zhang. T. Darrell, J. Malik, and K. Sreenath. Learning humanoid locomotion with transformers. ArXiv, abs/2303.03381, 2023.
[54] M. Janner, Q. Li, and S. Levine. Reinforcement learning as one big sequence modeling prob- lem. In Neural Information Processing Systems, 2021.
[55] L.. Chen, Lu, A. Rajeswaran, Lee, Grover, M. Laskin, Abbeel, A. Srinivas, and
I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems, 2021.
[56] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint wrXiv:2303.33376 2023.
[57] K.-H. Lee, O. Nachum, M. Yang, L. Y. Lee, D. Freeman, W. Xu, S. Guadarrama, 1. S. Fis- cher, E. Jang, H. Michalewski, and I. Mordatch. Multi-game decision transformers. ArXiv, abs/2205. 15552205.15241,,.1 2022.
[58] M. Xu, Y. Shen, S. Zhang, Y. Lu, D. Zhao, J. B. Tenenbaum, and C. Gan. Prompting decision transformer for few-shot policy generalization. ArXiv, abs/2206. 100ps/220..11999,, 2022.
[59] M. Xu, Lu, Y. Shen, S. Zhang, D. Zhao, and C. Gan. Hyper-decision transformer for efficient online policy adaptation. ArXiv, abs/2304.08487, 2023.
[60] J. N. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ArXiv, 503.03385, 2015.
[61] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. ArXiv, abs/2105.08 (.0.03....3,.... 2021.
[62] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealis- tic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205. 111477 2022.
[63] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthe- sis with latent diffusion models. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2022.
[64] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. ArXiv, his/2204.061255 2022.
[65] A. Ajay, Y. Du, A. Gupta, J. B. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision making?? ArXiv, abs/22 1..22211.1660 15657, 2022.
[66] S. Huang, Z. Wang, P. Li, B. Jia, T. Liu, Y. Zhu, W. Liang, and S.-C. Zhu. Diffusion- based generation, optimization, and planning in 3d scenes, ArXiv, abs/2301.06W) 2023.
[67] I. Kapelyukh, V. Vosylius, and Johns. Dall-e-bot: Introducing web-scale diffusion models to robotics. IEEE Robotics and Automation Letters, 13955-39633 2022.
[68] W. Liu, T. Hermans, S. Chernova, and C. Paxton. Structdiffusion: Object-centric diffusion for semantic rearrangement novel objects. ArXiv, abs/2211 11.2211..00 1.4604, 2022.
[69] Z. Zhong, D. Rempe, D. Xu, Y. Chen, S. Veer, T. Che, B. Ray, and M. Pavone. Guided conditional diffusion for controllable traffic simulation. ArXiv, abs/2210.173666 2022.
[70] C. Jiang, A. Comman, C. Park, B. Sapp. Y. Zhou, and D. Anguelov. Motiondiffuser: Control- lable multi-agent motion prediction using diffusion. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition (CVPR), pages 9644-9653, June 2023.
12

---

[71] Y. Du, M. Yang. B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. ArXiv, abs/2302.0011 11, 2023.
[72] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. 7. Tan, I. Momennejad, K. Hofmann, and S. Devlin. Imitating human behaviour with diffusion models, 2023.
[73] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 8748-8763 8763. PMLR, 2021.
[74] Lu, D. Batra, Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic repre- sentations for rision-and-language tasks. In Neural Information Processing Systems, 2019.
[75] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017.
arXiv preprint
[76] K. Zhang, M. Sharma, J. Liang, and O. Kroemer. A modular robotic arm control stack for research: Franka-interface and frankapy. arXiv preprint arXiv:201.002398, 2020.
[77] Liu, S. James, A. Davison, and E. Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.0309. 2022.
[78] Y. Li and T. Harada. Lepard: Learning partial point cloud matching in rigid and deformable scenes. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5544-5554 2022.
[79] J. Su, Y. Lu. S. Pan, A. Murtadha, B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint 104.0944 2021.
13

---

7 Appendix
7.1 Noise schedulers for Local Trajectory Diffuser
We model local trajectory optimization as discrete-time diffusion process, which we implement using the DDPM sampler [18]. DDPM uses a re---armmeric time-dependent noise variance sched- uler SR. which defines how much noise is added at each time step. We adopt a scaled linear schedule for the position and a squared cosine schedule for the rotation of each trajectory step.
1ak-19k 2k-1
Vor - xx-1) 2 +
1 - ūk-1
(xk - co(
(8)
āk
1 ā
1 Ox
By defining 0% 1 3k, and ai 11 ag we can now obtain the analytical form of Ag. Yks Ok in Equation 5 as follows:
1
A
V 04k
0kk
7h
V1 αik
(10)
ac++1
0
3k
(11)
Okk
where ls is the diffusion denoising timestep.
7.2 Act3D Background and Implementation Details
Act3D is a language-conditioned end-effector 6-DoF keypose predictor that learns 3D perceptual representations of arbitrary spatial resolution via recurrent coarse-to-fine 3D point sampling and featurization. Act3D featurizes multi-view RGB images with pre-trained 2D backbone and lifts them in 3D using depth to obtain a multi-scale 3D scene feature cloud. It then iteratively predicts 3D foci of attention in the empty 3D workspace, samples 3D point grids their vicinity, and featurizes the sampled 3D points using relative cross-attention the physical scene feature cloud, language tokens, and proprioception. Act3D detects the 3D point that corresponds to the next best end-effector position using detection Transformer head, and regresses the rotation, end-effector opening, and planner collision avoidance from the decoder's parametric query.
We extract two feature maps per 256 256 input image view: 32 x 32 coarse visual tokens and 64% x 64 fine visual tokens. We use three ghost point sampling stages: first across the entire workspace (roughly 1 meter cube), then in 16 centimeter diameter ball, and finally in centimeter diameter ball. The coarsest ghost points attend to global coarse scene feature cloud (32 x 32 Neam coarse visual tokens) whereas finer ghost points attend to local fine scene feature cloud (the closest 32 x 32 x Team out of the total 64 64 Mcam fine visual tokens). During training, we sample 1000 ghost points in total split equally across the three stages. At inference time, we trade-off extra performance additional compute by sampling more ghost points than the model ever saw at training time (20, 000). We use layers of cross-attention and an embedding size 60 for single-task experiments and 120 for multi-task experiments. Training samples augmented with random crops of RGB-D images and +45.0 yaw rotation perturbations (only in the real world as this degrades performance in simulation).
7.3 Simulation Setup in RLBench
The RLBench simulation environment uses Franka Panda robotic arm on table-top setting. We consider m camera inputs: left shoulder, right shoulder, wrist, and front, as shown in Figure The wrist camera is attached the robot's end-effector and moves together with the robot. The other 3 are static. To ensure fair comparison, when comparing with PerAct, use all 4 cameras following PerAct setting, and use the first cameras when compared with other baselines.
14

---

Environment
Right
Wrist
Left
Front
Figure 3: Simulation setup.
7.4 Real-world Setup
Our real-robot setup contains real Franka Panda robotic arm equipped with parallel jaw gripper, as shown Figure 4. We use a single Azure Kinect sensor to provide RGB-D input signal from the front view 30Hz. The image input is of resolution 1280 720, and we crop and downsample it to 256 x 256 before feed- ing it to our model. We calibrate the extrinsics of the camera with respect to the robot base us- ing the easy handeye' ROS package. Our full model generates dense trajectories, thus we use low-level motion planners. We collect 6-DoF human demonstrations by tele-operating the robot using SpaceMouse? 30Hz, follow- ing [14]. We use the same strategy for keyframe extraction as in simulation. Our real-world multi-task policy is trained on 4 A100 GPUs for 3 days. Inference is done on a desktop with single RTX4090 GPU, running Ubuntu 20.04 and ROS Noetic. For robot control, we use the open-source frankapy' package to send real- time position-control commands to the robot.
Franka Emika
Azure Kinect
Figure 4: Real-world setup.
7.5 Discussion contribution versus Act3D
The main contribution of Act3D is lightweight point-based neural architecture that casts keyframe/macro-action prediction as feature point selection with adaptive spatial computation, which brings higher accuracy than previous voxel-based or 2D-based architectures at a lower com- putation cost. It advances the previous SOTA but still falls under the paradigm of keyframe predic- tion. ChainedDiffuser, on the other hand, presents a simple yet crucial idea: the unification of two existing popular paradigms in robotic imitation learning: keyframe abstraction and learnable
http:///twa..aoeoooo////o/oc//tp 'https://github http:///hh...oo//mmmmmmmm/////ua
15

---

Table 4; Success rate comparison with randomized camera viewpoint.
pick lift
pick-up cup
puch button
reach
slide
stack
take
take unbrolla
Mean
knifa
money
target
block
wine
money
IliveFormer
26
98
98
45
77
62.2
.26.2
ChatnedDiffuser
72
91
97
78
96
32
90
92
91
82.7
-1311
diffusion- trajectory generation. The two frameworks have been proven successful in the past, but each presents their own drawbacks. ChainedDiffuser is built on top of Act3D, but the way it unifies both frameworks is independent of the macro-action prediction method, and Act3D can be easily swapped with any keyframe prediction approach later if needed. The detailed pipeline pro- posed and evaluated in our paper a realization of this unification idea, and proves effectiveness as single method that's applicable to a wider range of robotic manipulation tasks.
7.6 Robustness 2D and 3D Methods under Varying Camera Viewpoints
In order better understand how much helps to use 3D information ChainedDiffuser compared with prior 2D methods, conducted experiments to evaluate the robustness of ChainedDiffuser, InstructRL and HiveFormer under varying camera viewpoint. At test time, randomly perturb the two shoulder cameras by [20, 30] degrees, while keeping the same camera look-at point. InstructRL. completely fails on all the tasks and yields unreasonable actions, as it operates purely on 2D inputs and directly regresses action outputs. fails to handle distribution shift image input due to changing camera viewpoint. HiveFormer uses depth image and 2.5D architecture: selects the highest-score score pixel and uses its corresponding depth to compute 3D actions, thus presenting certain robustness. We report numbers of HiveFormer and ChainedDiffussr in Table 6. Our model still performs reasonably well when the cameras perturbed, achieving the smallest performance drop, showing desirable viewpoint-invan invrrianinv This is an advantage of reasoning directly in 3D.
Whit
Global Fine Feature Cloud
Average (x,yzz) of Token Pixels
Feature Pyramid Network
Figid
Frozen Backbone
with 3x1 12xxx228 tokens
Global Coarse Feature Cloud
with 3x32x32 tokens
keep only two feature maps coarse size 32x32 fine of size 128x128
input images of size 256x256
feature maps of size 128x128,, 64x64, 32x32, 16x16
Figure 5: Scene Feature Cloud Generation. We encode each image independently with pre- trained and frozen 2D vision backbone to get multi-scale feature maps, pass these feature maps through feature pyramid network and retain only two: coarse feature map (at granularity that lets ghost points attend to all tokens within GPU memory) and a fine feature map (as spatially precise afforded by input images and the backbone). We lift visual tokens from these two feature maps for each image to 3D scene feature clouds by averaging the 3D positions pixels in each 2D visual token.
7.7 Further Architecture Details
We explain here the 3D visual backbone in more details, including the visual and language encoding, 3D projection, and relative spatial encoding.
16

---

Visual and language encoder Our visual encoder maps mieti-view RGB-D images into a multi- scale 3D scene feature cloud. We use a large-scale pre-trained 2D feature extractor followed by a feature pyramid network to extract multi-scale visual tokens for each camera view. Our input RGB-D, so each pixel is associated with depth value. After featurizing the image, we obtain feature map whose spatial resolution lower than the original image. We associate every super-pixel (2D grid location) in this feature map depth value, by averaging the depth values of the image pixels that correspond to this super-pixel, i.e., the receptive field. Then we "lift" this 2D feature vector to 3D using the pinhole camera equation and the camera intrinsics, as shown in Figure 5. Each visual token in 3D uses the mean 3D position of all the 2.D pixels as its 3D position.
The language encoder featurizes instructions with large-scale pre-trained language encoder. We the CLIP ResNet50 [73] visual encoder and language encoders exploit their common vision- language feature space for interpreting instructions and referential grounding. Our pre-trained visual and language encoders are frozen during training.
Relative 3D cross-attentions ChainedDiffuser uses relative 3D positional encodings proposed in [78, 79] to incorporate translational invariance. We featurize each of the 3D point candidates and a parametric query (used to select via inner-product one the point candidate as the next best end- effector position the decoder) independently through cross-attentions the multi-scale feature cloud, language tokens, and proprioception. Our cross-attentions use relative 3D position information and are implemented efficiently with rotary positional embeddings [79]. Given point (x. 2) E R and its feature x E R the rotary position encoding function PE is defined as:
scene
coos ane
sin 8 ers ar 0
sin 2
(E(p, x) M(p)x=
x, M
y
dok coe yfe
y8,
Mayc_
0
crs A sin 20t
sin 28 29k
where 0g
{1.... d/6). The product two positionally encoded features
10000 1772,
PE(p...,) PE(P, Xy) x M(p.)* M(P,)x, = x M(P, P,)x,
which depends only the relative positions of points p; and Pj.
17


Reviews:
[{"review_id": "rzv9az6kJM", "reviewer": "Reviewer_JWec", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents ChainedDiffuser, a new policy architecture for learning language-conditioned robot manipulation tasks via imitation learning. The architecture predicts high-level actions (e.g. a next end effector pose for the arm to reach) based on RGBD input and language conditioning and uses a diffusion model, conditioned on the high-level action, to generate a dense trajectory to reach it. The experiments demonstrate strong performance on the RLBench benchmark, and on some real world tasks. A comparison to ablations also highlights the value of both the high-level and low-level components.", "summary_of_recommendation": "While the experimental results in the paper appear to be strong, the paper would benefit from some additional discussion and comparisons. The paper should also revise the writing to improve the clarity.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the points made in the weaknesses section -- in particular the comments on experimental evaluation and clarity.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n\nThe paper presents convincing empirical results that show the benefits of the proposed approach and clear improvements over baselines. \n\n**Weaknesses**\n\nSummary: The paper could benefit from some more analysis and discussion on when the proposed method is better than alternatives. The writing also needs some major improvement to improve the clarity of the method, and some assumptions could be more clearly stated.\n\nThe paper could benefit from some more analysis and discussion on when the proposed method is better than alternatives:\n\n- From Tables 1 and 2, InstructRL also appears to be a strong baseline, and it does not use 3D information. While Act3D is also a strong baseline, it could be seen as an ablation, since several parts of the architecture are shared with this method. It would be great to have some further comparison and analysis against InstructRL in order to understand situations in which (1) the 3D information helps and (2) the overall method performs better than InstructRL. It might also be useful to add InstructRL as a row in Table 4.\n\n- It would also be nice to understand whether Act3D is critical, or if other high-level action detector architectures could be used, especially because Act3D is a concurrent CoRL submission and appears to advance the state of the art itself. This would help decouple the value of this hierarchical policy architecture from the specific instantiation of the high-level.\n\n- The comparison against \"Regression\" in Table 4 indeed shows the value of using the diffusion model for trajectory prediction. However, the gap is quite small on most tasks and large on a select handful. More analysis of what task characteristics lend themselves to modeling the low-level trajectory generation with diffusion would be valuable.\n\nThe writing also needs some major improvement to improve the clarity of the method:\n\n- Section 3.2 is not very clearly explained, and the diagram in Fig 1e is also not straightforward. For example, proprioception is feeding into the cross attention from the bottom of the diagram, while the other inputs feed in from the top. In the text, it isn't clear what Z_query (tilde) and Z_point (tilde) correspond to, and where dot product comparisons are taking place. Instead, having a clear set of equations and variables that describe how inputs map to outputs, instead of pure text descriptions, would be a lot more straightforward to follow.\n\n- Similarly, section 3.3 could also be more clear. For example, the text above equation (2) suggests both cross-attention and self-attention operations are being applied simulataneously, which does not make much sense, and equation (3) has the same variable appearing three times. Perhaps equation (2) and (3) are describing the evolution of a particular variable (e.g. a forward pass through some operations) but it is not clear right now.\n\n- More details on the way 2D CLIP vision tokens are unprojected into 3D would be helpful. The point-based representation appears to be a crucial part of the method, so clear and precise operations (with equations) would be helpful (at the very least in the appendix).\n\n- Overall, the writing in section 3 could be improved by deferring some details (such as relative positional encodings) to the appendix, and instead focusing more on the core operations that describe how the input is transformed into the output prediction.\n\n- Improving the figures would also be beneficial. Consider adding more figures of the model architecture and splitting the dense overview figure (Figure 1) into more pieces. Some of the results figures (e.g. Tables) could be compressed and/or moved to the appendix to make room for more figures that outline the approach more clearly, and provide intuition.\n\n- The method requires a calibrated camera - this should probably be stated clearly as a limitation / assumption in the paper. It might also be useful to explicitly state the assumptions on the data contained in the demonstrations.\n\nSome additional general comments follow:\n\n- L2 loss for quaternion prediction (as mentioned above equation 5) seems like it could be problematic. How is the network output enforced to be a valid quaternion?\n\n- Reference 6 and 10 are repeated.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "T6NpHgGGWS", "reviewer": "Reviewer_zryn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The presented paper introduces a hierarchical method to solve language-conditioned tasks. For a given natural language-based task description and multi-view images of the scene, a 3D feature cloud is constructed and used to condition a macro-action policy to find the goal action (end-effector pose). This segment is from Act3D, their concurrent submission. Next, the goal state and the current state, along with the scene and task embeddings, are used to condition a trajectory diffuser to sample intermediate micro-actions for obtaining the whole sequence of poses (contribution). The method proposes goal pose (Diffuser) to action-only diffusion (Diffusion Policy) for fixed number of intermediate poses between current and goal state. The proposed method is validated over RLBench tasks in simulation and real-world deployments. Comparison is conducted with relevant baselines in their corresponding benchmarks. Ablation results are shown for some tasks to show the relevance of each component.", "summary_of_recommendation": "The paper is built over Act3D, where a classical motion planner between the current and predicted goal state is replaced by a diffusion-based planner. While such a framework helps marginally in most of the tasks, certain challenging tasks have been solved well by the unified approach. There is minor discrepancy in how the authors are training the diffusion model jointly with their macro-action model and how the control loop is working. The evaluation is very extensive and appropriate comparison results have been shown.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. How do you actually achieve this “We train both the action detector and the trajectory diffuser jointly” using Eq. (5)? Aren’t you using score matching objective for training diffusion score network? Eq (5) seems to be the training objective for regression planner.\n\n2. The limitation mentions that the control is closed-loop w.r.t. the macro-actions (i.e. goal). This is confusing as no clarifications on control is provided in the implementation section. Are the micro-actions open-loop for 50 poses? Please add an algorithm to describe the control loop. \n\n3. In Table 1, is chained-diffuser not using a learned collision avoidance classifier for the low-level motion planner? \n\n4. Are the language instructions same for a particular task? How do they change during test time?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Act3D proposes most of the segment in the proposed approach, from obtaining the language task and 3D feature cloud embeddings to getting the goal state which leads to the successful completion of the task. This is also evident from the success metric values of Act3D that a simple planner is sufficient for most tasks.\n\nWhen that is not the case, the authors show that a learning-based planner is beneficial and show that the performance of a diffusion-based planner is better as compared to a regression-based action sequence fitting (also established by prior work: Diffusion Policy).\n\nIf we consider Act3D as prior work, then the proposed method has limited novelty. This is more concerning as the architecture for obtaining all the used embeddings is also similar to Act3D. However, the unified framework is an efficient approach for solving complex manipulation problems in 3D. The span of environments used for validation of the proposed approach is convincing. Further, the ablation on the importance of the unified framework as compared to using individual segments justifies the combination.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "DL0-4HIFU6q", "reviewer": "Reviewer_BFkn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes ChainedDiffuser, unifying transformer-based macro action prediction and diffusion-based trajectory generation, allowing it to leverage their respective strengths. Namely transformer-based macro action prediction are good at long horizon planning but struggle with continuous interactions and smooth trajectory generation; diffuser-based trajectory generations perform well in local smooth trajectory generation but struggle with long horizon multi-task learning. ChainedDiffuser thus uses a transformer-based macro-action prediction, and creates smooth micro-actions to fill in the gap between macro-actions with a diffusion-based trajectory generator.\n\nThey test their method on RLBench, and demonstrate state-of-the-art performance in a variety of challenging settings. They show that ChainedDiffuser outperforms transformer-based only methods and trajectory diffusion alone, justifying the unification of the two frameworks.", "summary_of_recommendation": "The paper is technically sound, well written and potentially of high impact. Its contributions are somewhat novel. The claimed results are better than previous state-of-the-art. The recommendation is weak accept, but could be upgraded to a strong accept if the authors point out more clearly if and how significant the novelty is, despite the great technical quality of the paper.\n\n ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "One remaining question is about the extent of the novelty presented in unifying two frameworks in itself,  regardless of how good the resulting method and technical contributions are. It would be appreciated to see a more clear statement of the author's view of this.\n\nMoreover, as pointed out already in the limitations, ChainedDiffuser is not very flexible in highly dynamic environments, but since it's mentioned it would have been sound to see some experiments on this for completeness.\n\n ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is overall well explained, the work is of high technical quality and significant importance. The method is reasonably novel. \n\nThe main strength of the paper is that it overcomes the limitations of methods based only on transformers or diffusion by cleverly combining them, achieving state-of-the-art on RLBench, significantly improving performance on contact-rich tasks. The method is also very well justified and the experiments are sound.\n\nThe fact that their concurrent work Act3D performs competitively in most benchmarks to ChainDiffusers, and is submitted at the same time begs the question of whether the two should have been submitted together, with Act3D simply being a baseline to further show the improvements and add to the novelty of ChainDiffusers.\n\nSome of the weaknesses are well highlighted in the limitations section, the most significant one being that the method performs closed-loop control on the macro-actions, which could make it brittle in dynamic environment.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents ChainedDiffuser, a new policy architecture for learning language-conditioned robot manipulation tasks.  The proposed method overcomes the limitations of methods based only on transformers or diffusion by cleverly combining them, achieving state-of-the-art on RLBench. Reviewers agree that the method is well justified, clearly described, and the experiments are sound. It is an important contribution to the field. However, the reviewers also raise some concerns about the comparison with concurrent work such as Act3D. Since it is a concurrent work, the AC thinks a comparison is not required for submission, but it will be nice to include a discussion and potential comparison in the final version.
Decision: Accept (Poster)