RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
Authors: Songming Liu
Lingxuan Wu
Bangguo Li
Hengkai Tan
Huayu Chen
Zhengyi Wang
Ke Xu
Hang Su
Jun Zhu
Venue: ICLR 2025 Poster (2025)
Keywords: robot learning
diffusion models
foundation models
bimanual manipulation
Primary area: applications to robotics, autonomy, planning

Abstract: Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and the scarcity of training data. In this paper, we present the Robotics Diffusion Transformer (RDT), a pioneering diffusion foundation model for bimanual manipulation. RDT builds on diffusion models to effectively represent multi-modality, with innovative designs of a scalable Transformer to deal with the heterogeneity of multi-modal inputs and to capture the nonlinearity and high frequency of robotic data. To address data scarcity, we further introduce a Physically Interpretable Unified Action Space, which can unify the action representations of various robots while preserving the physical meanings of original actions, facilitating learning transferrable physical knowledge. With these designs, we managed to pre-train RDT on the largest collection of multi-robot datasets to date and scaled it up to $1.2$B parameters, which is the largest diffusion-based foundation model for robotic manipulation. We finally fine-tuned RDT on a self-created multi-task bimanual dataset with over $6$K+ episodes to refine its manipulation capabilities. Experiments on real robots demonstrate that RDT significantly outperforms existing methods. It exhibits zero-shot generalization to unseen objects and scenes, understands and follows language instructions, learns new skills with just 1$\sim$5 demonstrations, and effectively handles complex, dexterous tasks. We refer to https://rdt-robotics.github.io/rdt-robotics/ for the code and videos.

Published as a conference paper at ICLR 2025
RDT-1B: A DIFFUSION FOUNDATION MODEL FOR BIMANUAL MANIPULATION
Songming Liu, Lingxuan Wu", Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang. Ke Xu, Hang Su Jun Zhu
1 'Department of Computer Science & Technology, Institute for AI, BNRist Center, Tsinghua-Bosie Joint ML Center, THBI Lab, Tsinghua University
ABSTRACT
Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and the scarcity of training data. In this paper, we present the Robotics Diffusion Transformer (RDT), a pioneering diffusion foundation model for bimanual manipulation. RDT builds diffusion models to effectively represent multi-modality, with innovative designs of scalable Transformer deal with the heterogeneity of multi-modal inputs and to capture the nonlinearity and high frequency of robotic data. To address data scarcity, we further introduce a Physically Interpretable Unified Action Space, which can unify the action representations of various robots while preserving the physical meanings of original actions, facilitating learning transferrable physical knowledge. With these designs, managed to pre-train RDT on the largest collection of multi-robot datasets to date and scaled it up to 1.2B parameters, which is the largest diffusion- based foundation model for robotic manipulation. We finally tneettuned RDT on self-created multi-task bimanual dataset with over 6K+ episodes to refine its manipulation capabilities. Experiments demonstrate that RDT significantly outperforms existing methods. It exhibits zero- shot generalization to unseen objects and scenes, understands and follows language instructions, learns new skills with just I~5 demonstrations, and effectively handles complex, dexterous tasks. We refer the project page for the code and videos.
real robots
INTRODUCTION
Bimanual manipulation is essential for robots to accomplish real-world tasks (Edsinger & Kemp, 2007). For practical applications, useful manipulation policy should be able to generalize to unseen scenarios, such as unseen objects and scenes. However, current approaches either depend on task-specific primitives (Mirrazavi Salehian et al., 2017; Rakita et al., 2019; Grannen et al., 2023a) or are limited to small-scale model, data and simple tasks (Krebs et al., 2021; Franzese et al., 2023; Grannen et al., 2023b; Zhao et al., 2023: Grotz al., 2024: Liu al., 2024), thereby exhibiting only narrow generalization and failing in complex tasks. Following the success in natural language processing (Achiam et al., 2023: Touvron et al., 2023) and computer vision (Radford al., 2021; Kirillov et al., 2023), one promising direction to enable generalizable behaviors is to develop a foundation model through imitation learning on large-scale datasets.
Developing bimanual manipulation foundation models confronts the dual challenges of data scarcity and architectural limitations. The prohibitive costs of dual-arm systems create severe data scarcity (Sharma et al., 2018: Collaboration et al., 2023), fundamentally conflicting with the data- hungry nature of foundation models. Inspired by recent attempts in unimanual manipulation (Brohan al., 2023; Kim et al., 2024). mitigate this through cross-robot pretraining: leveraging multi-roba datasets for pre- pre-training followed by target-robot fine-tuning, amplifying data volume by three orders of magnitude extract transferable physical priors. However, two interconnected technical barriers emerge. First, the doubled action space induces multi-modal action distributions (Li, 2006; Jia et al., 2024) (see Fig. 2b for illustrative example) that demand expressiveness capability beyond current
"qqual contribution; 'Corresoondngg authors at (suhangss, dcszj) @tsinghua.cdu.cn

---

Published as a conference paper at ICLR 2025
RDT-1B
Datasets
Robotics Duffusion Trassoomer Langaage- Visuomoto Policy
Pre-Training
Tno-Tuning
boodiie Doniiig
Multiple Robots
Dataset Created by Ounseves
Target Dual-Arm
68.2%
Pre: Tralzed Sentch Basolines
Datases
Rabot
Diffuusion Transformer
T5
SigLiP
RDT
Training Boost 33.4%
IM Fly 6m
1,6%
Unitted cbon Space
HUT
ACT OpeNTZ
Dexterity
Few-Shot Learning
Instruction Following
Zero-Shot GehhoGGett Generalizability
Unens
Time
Robot walks its dog
Handover' & Fold Shorts
Pour water specified level
Unseen Objects & Soepes
Figure 1: Overview of Robotics Diffusion Transformer with 1B-Parameters (RDT-1B), a language-condlitonee visuomotor policy for bimanual manipulation.with state-of-the-art generaliz- ability to unseen scenarios (See App. H for metric calculation details).
methods (Zhao et al., 2023; Brohan et al., 2023; Kim et al., 2024), while simultaneously requiring scalabilly for stable large-scale training on multimodal data (text, vision, actions). Beyond architec- tural constraints, physical and action space variations across robots introduce data heterogeneity that risks negative transfer (Pan & Yang, 2009). Existing solutions either discard robots with structural inconsistencies or retain only cross-robot invariant features (Brohan al., 2023; Ghosh et al., 2023; Shah et al., 2023a), sacrificing valuable data diversity essential for generalization.
In this paper, introduce the Robotics Diffusion Transformer (RDT), the largest bimanual manipula- tion foundation model with strong generalizability. RDT employs diffusion transformer (DiT) as its scalable backbone (Peebles & Xie, 2023), with special designs for language-conditione bimanual manipulation. For expressiveness, RDT excels in capturing the full modalities of bimanual actions from massive data by using the capacity of diffusion models represent complex distributions (Sohn et al., 2015; Ho et al., 2020). For scalability, we harness the Transformer backbone and carefully de- encoding to eliminate the heterogeneity of various modalities. Moreover, robotic
sign the data is differed significantly from images and videos with temporal and spatial continuity (Chen et al., 2019; Liang al., 2022). To characterize its inherent nonlinear dynamics (de Wit et al., 2012), high-frequency changes (Ghosh al., 2023), and the unstable numerical range, make important modifications to the original DiT structure, including MLP decoding, improved normalization, and alternate injection of conditions (see Fig. 4 for their importance). To further enable training RDT on heterogeneous data, we propose the Physically Interpretable Unified Action Space, a unified action format for various robots with gripper arms. This innovative format mitigates potential conflicts between different robots while retaining the physical meanings of the original actions, which can promote the model to learn generalizable physical knowledge across diverse robotic datasets.
With the above designs, we managed to pre-train the RDT model on the largest collection of multi- robot datasets date (Collaboration al., 2023; Walke et al., 2023; Fang al., 2023; Kumar et al., 2024) and scale it up to 1.2B parameters, which is the largest diffusion-basel pre-trained model for robotic manipulation. To further enhance its bimanual manipulation capabilities, we fine-tuned the RDT on self-collected multi-task bimanual dataset comprising over 6K+ trajectories, which is one of the most extensive bimanual datasets. In our experiments, we have comprehensively evaluated RDT against strong baselines in both bimanual manipulation and robotic foundation models. Results show that RDT achieves state-of-the-art performance, outperforming baselines by achieving an improvement of 56% in success rates across a wide spectrum of challenging tasks. In particular, RDT has exceptional zero-shot and fhw-shot (1 ~ shots) generalizability to unseen objects, scenes, instructions, and even skills. RDT is also capable of accomplishing tasks requiring fine- aie-ggrined operations, such as controlling robot dog with a joystick. Finally, ablation studies show that diffusion modeling, large model size, and large data size all contribute to superior performance.
2

---

Published as a conference paper at ICLR 2025
Hape Camara
Erigger
1000
4 lellgrigper oith on:ge
Left Caseo
Exodo
Lettorpp
Mode2
Mode3
Action Distribution
(a) Target dual-arm robot
(b) Illustration of multi- modaltty
Figure 2: (a) Schematic diagram the ALOHA dual-arm robot. (b) A toy example of grasping a cube. Compared with unimanual manipulation, bimanual manipulation has more possible action modes, leading to stronger multi modaiityddlityy.... Colors from light to dark indicate that time goes forward.
2 RELATED WORK
Learning-based Bimanual Manipulation. One substantial challenge in learning bimanual manipu- lation policy is the high dimensionality of the action space, which exacerbates the data scarcity (Zoll- ner et al., 2004; Smith et al., 2012; Lioutikov et al., 2016; Stepputtis et al., 2022) and the multi-modal behavior (Colomé & Torras, 2018; 2020; Figueroa & Billard, 2017; Sharma et al., 2018; Xie et al., 2020; Franzese et al., 2023). Some works have developed more cost-effective interfaces for data collection (Zhao al., 2023; Aldaco et al., 2024), but they are limited to specific hardware configura- tions and still insufficient to bridge the data gap for generalizable policy. Others attempt to reduce data requirements by introducing inductive biases, such as distinguishing two arms for stabilization and functionality (Grannen et al., 2023b), parameterizing movement primitives (Batinica et al., 2017; Amadio al., 2019; Chitnis et al., 2020; Franzese et al., 2023), or using voxel representations (Grotz et al., 2024; Liu et al., 2024). These methods use strong priors or simplified modeling, which successfully reduce the action space, but at the cost of reduced scope of application and inability to express the multi-modaliin of bimanual behaviors (Pearce et al., 2023).
Foundation Models for Robotics. Foundation models have shown immense promise in enabling generalizable behaviors by training multi task "generalist" models (Brohan al., 2022; 2023; Ghosh et al., 2023; Kim et al., 2024) on large multi-task robot datasets (Collaboration et al., 2023; Brohan et al., 2022: Fang et al., 2023). Most studies adapt large vision-
models directly predict action (Brohan et al., 2022; Driess et al., 2023; Brohan et al., 2023; Collaboration et al., 2023; Kim al., 2024). While demonstrating generalization to new objects and tasks, they face issues with quantization errors and uncoordinated behaviors (Pearce et al., 2023) when applied to bimanual manipulation. It's largely due to their discretization of action spaces. To enhance precision, diffusion models have been used for continuous control (Ho et al., 2020; Chi et al., 2023; Pearce et al., 2023; Ghosh al., 2023). Ghosh al. (2023) pre-train a Transformer-basee diffusion policy on subset of Open X- X-Embodiment (Collaboration et al., 2023) dataset (25 datasets), with up to 93M parameters.
3 PROBLEM FORMULATION AND CHALLENGES
We start by formulating the task and elaborating on the challenges. To evaluate the model on the hardware, we choose the ALOHA dual-arm robot our target robot since it is one of the most representative dual-arm robots and is suitable for collecting human demonstration data via teleoperation (Zhao al., 2023: Fu al., 2024; Aldaco al.. 2024). Fig. 2a shows schematic diagram of the target robot, which consists of two arms with grippers and three cameras. Note that our setting and foundation model are generic to any dual-arm gripper robot.
We consider the concrete task of innguage-conditionn fundamental in robotics and has great value in real-world scenarios such as household (Stepputtis et al., 2020; Brohan et al., 2022; Zhao et al., 2023). Formally, given a language instruction f, the policy is presented with an observation o, at time E N+; and then it produces an action a, to control two robot arms to achieve the goal specified by L. The observation is represented as a triple o, = (X,-Tin - 11 Zt, c), where X, X,-Tmg+1:t+ 1:4++ := (X,- +m: I..... X,) is the RGB observation history of size T Ting. 2; the low-dimensional proprioception of the robot, and is the control frequency. The action a; is usually subset of the desired proprioception 2/+1
bimanual manipulation with vision, which is
F.g., 2, may include the gripper position time t, and a be the target gripper position step
3

---

Published as a conference paper at ICLR 2025
Unified ction Space
www-Dimensional Inputs
Image Inputs Xe-a
Propelo 2, Nonay Art. RoteT,
Embeed
St Retats
Exteror X?
Right/Wissx
LaltWWistx
Duff Time Stepe
SingleAAm RSP
Actionn Space
SigUIP
MLP
Concet.
MLP
Takono with
Kl.,
Al.,
Melli-Di Pos. Erib,
XT
X
DIT Black with Cross- meriien
Allemvtively
Dual A.m Joints
Toloms Mark
MLP
TS-XXL
O uttputs
Language Inputs
"Insert the lemon slice on the peper cup into goblet rim."
Denplsad dasers
Figure 3: RDT framework. Heterogeneous action spaces of various robots are embedded into unified action space for multi-robot training. Inputs: proprioception 2;, noisy action chunk ar:+TT. control frequency and diffusion time step k; acting as denoising inputs; image inputs (m = 2 and X = (x+, x. X denotes set of images from exterior, right-wrist, and left wrist cameras) and language inputs, acting as conditions. Outputs: denoised action chunk O.:+T.
A specific task in bimanual manipulation typically consists of multiple elements: a skill (e.g., verbs like "pick", "wipe", or "open"), an object (e.g., nouns like "bottle", "table", or "door"), scene (i.e. the environment in which the task takes place), and modality describing how the skill is performed (e.g., adverbials like "pick the bottle with the left hand"). When encountering a new task, a practical policy is required to generalize to unseen elements in the task, which is particularly challenging for previous rule-based methods (Mirrazavi Salehian et al., 2017; Rakita et al., 2019; Grannen et al., 2023a) as well as barning-bassed methods that are limited to either small models and data or simple tasks, as discussed in Sec. 2.
We aim to train foundation model policy via imitation learning to achieve generalizability. However, the available data for a specific dual-arm robot is particularly scarce (< 10K trajectories) due to high hardware costs, far from the common requirement train foundation model. To address this, we propose to employ pre-training and fine-tuning pipeline (Radford et al., 2018) to take advantage of data from multiple robots by drawing inspiration from recent advances in unimanual manipulation (Ghosh et al., 2023; Collaboration et al., 2023; Kim et al., 2024). In this manner, we would expand the data size by three orders of magnitude. Specifically, we first pre-train the model on a large-scale multi-robot dataset Ppr (mostly single-arm) and then fine-tune on dataset of the target robot Dn- We denote the dataset by D. = {(E()) o0 (.)) | < T(9, S ii N). where ( the length of the i-th trajectory and N is the number of trajectories. Moreover, is worth emphasizing that our goal is to use multi-robot data to enhance the model's generalizability in bimanual manipulation ruther than developing a cross-embodiment model for various robots. There are two main challenges to developing such a foundation model with multi-robot data:
How to design a powerful architecture? A generalizable foundation model ne- cessitates powerful architecture. This requirement encompasses two primary aspects. Firstly, the architecture must possess sufficient expressiveness to capture the multi- modality in the action distribution. Fig. 2b illustrates toy example where the robot attempts to grasp cube. We can that there are many modes to finish this task, in contrast to unimanual manipulation, where only one robot arm controlled. When collecting demonstrations, the human operator may randomly pick one of them, leading to multi-modality in the collected action data. Secondly, scalability is necessary for such an architecture. As foundation model, it should effectively process heterogeneous inputs from various modalities (text, images, actions, etc.) while being scalable to train stably on large datasets.
Challenge 1:
Challenge 2: How to train on heterogeneous data? Training on multi-robot data presents unique challenge of data heterogeneity. The physical structure and the action space can vary greatly across different robots. Previous attempts either restrict themselves to a subset of robots with similar action spaces (Yang et al.. 2023; Ghosh et al., 2023; Kim et al., 2024) or only retain a subset of inputs
unseen means that certain element has not appeared in the training data.
4

---

Published as a conference paper at ICLR 2025
sharing the same structure (Collaboration et al., 2023; Yang et al., 2024), at the cost of losing lot of information. remains largely under- addressed on how to train models on such heterogeneous data.
4 ROBOTICS DIFFUSION TRANSFORMER
We now present Robotics Diffusion Transformer (RDT), as illustrated in Fig. In Sec. 4.1, we present the diffusion model and the corresponding architecture to address Challenge 1. In Sec. 4.2, we resolve Challenge by proposing a physically interpretable unified action space to unify various robot action spaces and enable multi-robot pre-training. We also collect comprehensive multi-task bimanual dataset for fine-tuning to improve the bimanual manipulation capabilities of RDT.
4.1 RDT MODEL
Diffusion Modeling. Due multi-modality, given the language instruction É and observation ot there may be many possible actions a, to proceed with the task. The policy will learn the "average" of action modes if we model it as deterministic mapping (e, o.) + a, and regress the tuples of (E, or, aq) in the training data. This may result in out-of of-ooiiriibuution actions, such as the arithmetic mean of multiple modes, which can be completely infeasible (Pearce et al., 2023). Instead, we choose to model the continuous conditional distribution p(a, f, or). As discussed in Sec. 2, among various approaches, diffusion models excel in both expressiveness and sampling quality, but can be slow to sample nigh-dimensional data (e.g.. images). I Luckily, for our settings, the drawback is minor since that a, has a much lower dimension than images, which requires only minimal sampling overhead. This has made diffusion models an ideal choice for policy as in Chi et al. (2023).
Nevertheless, employing diffusion models for robotic tasks faces unique challenges since the in- herent properties of robotic physics quantities (i.e., the action and proprioception) are different from image/video data. Image and video data, while high-dimensional, often exhibit a degree of temporal and spatial continuity (Chen et al., 2019; Liang et al., 2022), with changes between frames typically being incremental. In contrast, robotic physics quantities are characterized by its nonlinear dynamics (de Wit et al., 2012) and the potential for high-frequency changes stemming from the physical interactions, such as collision, constraints, and material properties like damping. Moreover, the quantities also feature an unstable numerical range, probably due to extreme values caused by unreliable sensors. This underscores the necessity of adapting current diffusion models to effectively capture the instability and nonlinearity of robot data. Next, we will first elaborate on diffusion formulation and then introduce our design of architecture to resolve these challenges.
When making decision with diffusion policies, we first sample a totally noisy action at ~N I) and then perform K E N+ denoising steps to denoise it to clean action sample at from p(a. o:):
a Vak-130 1gt
at (1-""))
a²= ā a
o k=
where {a [a*]K=n, 1k=l K [k=1 are scalar coefficients pre-defined by a noise schedule (Nichol & Dhariwal, 2021). Here, g* := 1 1-1-a at and a := II= a' NN I) if k > 1, else a 1, 0. However, af is intractable before sampling is finished. We opt to use learnable denoising network fe with parameters to estimate the clean sample from a noisy one: at + (f, or, k:). To train such a network, we will minimize the following mean-squared error (MSE) of denoising:
(()) = MSE acs (a: So((, or, Vata, + v vI-a"e,
where k Uniform( (fI,..., KJ), ~ NO I), and (L, or, a.) is sampled from our training dataset. Later in this paper, we will denote noisy action inputs by à, == vā + v1 ãe, in which the prefer to predict sequence of
superscript of k is dropped for simplicity. Besides, in practice, actions, i.c., an action chunk, in one shot to encourage temporal consistency (Chi et al., 2023) and to alleviate error accumulation over time by reducing number of decisions in task (Zhao et al., 2023). Specifically, we model p parr+4z le, ot), where au == (a.,..., a++T, -1) is an action chunk and T, denotes the chunk size (Zhao et al., 2023). We provide detailed discussion in App. A.
We now present the design of the architecture, including the encoding of modtl inputs and the network structure of fa, while details are deferred to App. B.
5

---

Published as a conference paper at ICLR 2025
Encoding of Heterogeneous Multi-Modal Inputs. The heterogeneity of multi- inputs is reflected in the structure; that is, the format and number of dimensions of each modality are significantly different. This has posed challenges for mult-modal training. To address this, we encode these diverse modalities into a unified latent space. Below are the encoding methods:
now-Dimensional Inputs are low-dimensional vectors that represent physical quantities of the robot, including the proprioception, the action chunk, and the control frequency. To encode them, we use MLPs (with Fourier features (Tancik et al., 2020)), which can effectively capture the high-frequency changes in wow-dimensional spaces.
Image Inputs are high-dimensional and contain rich spatial and semantic information. To extract compact representations, we use an image- segge-text-alle text-aligned pre-trained vision encoder, SigLIP (Zhai et al., 2023). We fix its weights during training to save GPU memory.
Language Inputs are of varying length and highly abstract, posing integration challenges due to their complexity and ambiguity. To encode them, we use pre-trained Transformer-based language model, XX-XXL (Raffel et al., 2020). We also fix its weights during training save GPU memory.
Besides, heterogeneity also manifests in both cross cross-modal modal and intra-modd information density. First, modalities differ inherently in information capacity (e-g.. visual inputs typically yield more tokens than text). Second, substantial variance exists within modalities, as seen in robotic perception where exterior cameras capture broader scenes versus the limited views from wrist cameras (Fig. 3). This discrepancy risks shortcut learning: focusing on exterior views while neglecting details from wrist cameras. To promote balanced multimodal integration, we implement stochastic independent masking across modalities during encoding, preventing overreliance on specific inputs.
Network Structure of fe- We choose Transformer as the scalable backbone network (Bao et al., 2023; Peebles & Xie, 2023) and make the following three key modifications from Diffusion Transfomer (DiT) by considering the characteristics of our robotic problem:
10
RDI
10"
107
QKNorm & RMSNorm. The unstable numerical range of the inputting robotic physical quantities can lead to problems such as gradient instability and numerical overflow, especially when training large foundation models. To solve this problem, add QKNorm (Henry et al., 2020) to avoid numerical instability when calculating attention. Besides, we also note that our problem be considered as a time series forecasting task, and the centering operation in the original DiTs' LayerNorm could cause token shift and attention shift, thus destroying the symmetry of the time series (Huang et al., 2024). Therefore, we replace LayerNorm with RMSNorm (Zhang & Sennrich, 2019) without a centering operation. Fig. 4a shows that large-scale pre-training tends be very unstable or even explode without this modification.
10"
10"
20K
40K
60K
Training Steps
(a) Loss w/o QKN RMSN
100
KDT
Stcoeess Rate (%)
12
Destority
Pollowing
Task
(b) Task w/o MLP or ACI
MLP Decoder. To improve the approximation capability for nonlinear robot actions, replace the final linear decoder with a nonlinear MLP decoder as a projection from the latent space back to the physical space. As empirically shown in Fig. 4b, without this design, RDT cannot effectively capture nonlinear dynamics and thus loses the ability to accomplish dexterous tasks that require delicate operations.
Figure (a) Unstable loss curve during training without QKNorm & RMSNorm. (b) Success rates of RDT (w/o MLP Decoder or w/o ACT) in tasks of Robot Dog (walk straight sub-task) and Pour Water-I-Ild (correct amount sub- task). See Fig. 5 for task defini- tions. All the models are with- out pre-training in this experiment due to resource constraints.
Alternating Condition Injection (ACI). In our model. image and language inputs serve as conditions, which are high-dimensiooal and variable in length, contrasting with the class label conditions in traditional DiT's (Peebles & Xie, 2023). These informative conditions are challenging to compress into a single token, making the original adaptive layer norm approach unsuitable. Therefore, we employ cross-attention to accommodate conditions of varying lengths avoiding the information loss in further compression. Besides, we further analyze that, given that image tokens usually much more than text tokens, simultancous injection of both modalities tends to overshadow text-related information, thus impairing the capability of the instruction following (see Fig. 4b for quantitative results). To

---

Published as a conference paper at ICLR 2025
mitigate this issue, we strategically alternate between injecting image and text tokens in successive layers' cross-attention rather than injecting both in every layer.
4.2 DATA
Training on Heterogeneous Multi-Robot Data. To enable training on hetcrogeneous multi-robot data, we need unified action space shared among various robots provide unified format for multi-robot actions. The mapping from the original action space of robot to the unified action space should be physically interpretable, and each dimension of the space should have clear physical meaning. This can encourage the model to learn shared physical laws from different robot data, thereby improving the efficiency of learning from data of different robots (Shah et al., 2023a).
The design of the space consists of two steps. Firstly, for each robot, we can use single space to accommodate both its proprioception & and action az. This is because a is usually subset of the desired 2++1 (de Wit et al., 2012; Kouvaritakis & Cannon, 2016), and thus the space of 2; naturally contains the space of ag. Secondly, we design a unified space that encompasses all the main physical quantities of most robots with gripper arms. As illustrated in the left side of Fig. 3, we embed the action space of a robot into this unified space by filling each element of the original action vector into the corresponding position of the unified action space vector according to its physical meaning, with the remaining positions being padded. The specific definition the space is given in App. C.
With this unified space, we are able to pre-train RDT on data from almost all modern robots with gripper arms, and greatly expand the data scale towards the requirement for a foundation model. Specifically, our collection of pre-training datasets includes 46 datasets of various robots, with total size of 1M+ trajectories and 21TB. More details and preprocessing are deferred to App. D.
Collecting Comprehensive Multi-Tass Bimanual Dataset. Though having been pre-trained on large-scale datasets, RDT could still need help to zero-shot generalize to the target dual-arm robot due to the embodiment gap. To bridge the gap, we need to collect a multi-task bimanual dataset on the target robot for fine-tuning, Recent advances in large language models (Ziegler et al., 2019; Brown et al., 2020; Touvron et al., 2023) have shown that high-quality fine-tuning datasets are crucial for model performance. We ensure the high quality of our dataset from three aspects: (1) Regarding quantity,
have collected 6K+ trajectories, making dataset one of the largest bimanual datasets nowadays; (2) Regarding comprehensiveness, we consider 300+ challenging tasks, covering most manipulation task types, from pick-and-place to plugging cables, even including writing math equations; (3) Regarding diversity, we prepare 100+ objects with rigid and non-rigid bodies of various sizes and textures and 15+ different rooms with different lighting conditions. Besides, we further utilize GPT-4-Tume (Achiam et al., 2023) to rewrite
instructions to increase text
diversity. For more information, we refer to Fig. 6 and App. E.
5 EXPERIMENTS
We aim to answer the following questions through real-robot experiments: Q1: Can RDT zero-shot generalize to unseen objects and scenes? Q2: How effective is RDT's zero-shot instruction-following capability for unseen modalities? Q3: Can RDT facilitate few-shot learning for previously unseen skills? Q4: Is RDT capable of completing tasks that require delicate operations? and Q5: Are large model sizes, extensive data, and diffusion modeling helpful for RDT's performance?
5.1 EXPERIMENT SETUPS
Tasks. We select 7 challenging tasks to evaluate the generalizability and capabilities of RDT from different dimensions, including complex scenarios that the model may encounter in real-world tasks, such as various unseen elements and dexterous manipulation. An illustration of the dimension of each task is given in Table 1 while detailed definitions and visualizations are provided in Fig. 5.
Data. We use the pre- pre-training and fine-tuning datasets in See. 4.2. We now list the number of demos related to each task in our fine-tuning dataset. Wash Cup: 133 demos for seen cups combined and 0 demos for unseen cups; Pour Water: 350 demos for seen rooms combined and 0 demos for unseen
7

---

Published as a conference paper at ICLR 2025
1.2
Wash Cup: "Fill the mag with water from the faucet and it into the sink. "The is randomized up to 10cm. The robot needs to Pick Up Cup (fI), Tur On Faucet (H22), Get Water (H3, to ensure that the water falls inlo the cup), Pour Out Water (64), and Place Back Up (15), The last image shows seen and unseen cups (from left right).
Init.
water from bottle muy. " bottle mnug randomized up lu needs Pick Up Bottle (HI), Pour Water (12), and Place Back Bottle (H3). The last shows and three (b-d).
init.
Left Hand
Right Hand
Pour Water-L-1/3 (-R-2/3); "Pour water from the boatle into the mug until about one-third (two-thirdss) with the left (right) hond." In the loft-hand task, the bottle initially placed to the left of the mug, and vice versa. The bottle and mug are randomized up to 6em.
Init
"Pick up the black marker the right pur into the packaging the left. The marker and box randomized up 3em. The robot needs Pick Up Pon (#1), Switch Hand (#2), Drop Pen (#3), and ensure can Fall into Box (#4).
Handover:
Trit.
Shorts:
"Fold basketball shorts rectangle. shorts are randomized up 3em wrinkles randomized.
Striight
Robot Dog: "Push the left joystick, forward to make the robot dog walk struight forward." The remote control randomized up dem and the robut dog up to 50cm. The robot needs to Grab Remnote (#1) and Push Joystick (42) as straight as possible to make the robot dog walk in straight line as shown the last two images.
Figure 5: Task definitions and visualizations. For challenging tasks, we describe their language instruction, randomization, and definitions of each sub-task. For Pour Water-L-1/3 and Pour Water-R- 2/3, we show the resulting water levels in two images.
rooms; Pour Water-L-1/3 & Pour Water-R-2/3: 18 demos for the water level of little, 19 demos for half, and 19 demos for full; Handover: 5 demos; Fold Shorts: demo; Robot Dog: 68 demos.
Model Training and Inference. We scale the size of RDT to 1.2B parameters, establishing it as the currently largest diffusion-based robotic foundation model. The model pre-trained on 48 H100 80GB GPUs for month, giving total of 1M training iteration steps. It takes three days to fine-tune this model using the same GPUs for 130K steps. We defer further details to App. F, including the running platform, design choices, and data augmentation techniques. For real-time inference, adopt 3PM-Solver-s (Lu et al., 2022), a recent sampling accelerator of diffusion models. It can reduce the diffusion steps required to sample action chunk from 100 steps to 5 steps, achieving action chunk inference frequency of Hz (action chunks per second) and an average action inference frequeney of 381 Hz (actions per second) on the target robot's onboard RTX 4090 24GB GPU.
Baselines. To comprehensively evaluate RDT, we consider the most advanced basclines in robotic foundation models and bimanual manipulation, including Action Chunking with Transformers
8

---

Published as a conference paper at ICLR 2025
Table 1:
Dimensions when designing tasks. For Pour Water-L-1/3 and Pour Water-R-225 only the water levels of little, half (i.e., 1/2), and full are seen in training instructions. For Handover and Fold Shorts, the dataset only contains demos and 1 demo of the skill, respectively. For Robot Dog, requires delicate operations, as slight angle when pushing joysticks can make the robot dog deviate.
TASK NAME
DIMENSION
EXPLANATION
Wash Cup
Unseen Object (Q1)
To wash one seen and two unseen cups with the faucet
Pour Water
Unseen Socne (Q1)
To pour water into the cup in three unseen rooms
Pour W.1en-I.-1/3
Instruction Following (22)
To pour water into the with the left hand until one-third full
Pour Water-R-2/3
Instruction Following (Q2)
To pour water into the cup with the right hand until two-thirds full
Handover
S-Shot Learning (Q3)
To the marker the box, where handover needed due to far distance
Fold Shorts
1-Shot Learning (Q3)
To fold the shorts in horizontally
Robot Dog
Dexterity (Q4)
To push the joystick straight to control the robot dog to walk a straight line
(ACT) (Zhao et al., 2023). OpenVLA (Kim et al., 2024), and Octo (Ghosh et al., 2023). ACT is state-of- the-art -tate-of-the-art
method in bimanual manipulation, which uses VAE to model the action distribution. Open VLA is the largest open-source foundation model (7B), employing the discretization modeling, Octo a diffusion-basse foundation model, and its largest version has only 93M parameters.
Metric and Hardware. We employ the success rate as our main metric, which is calculated by dividing successful trials by total trials. Wash Cup is tested with 8 trials for each cup (one seen cup, two unseen cups, 24 trials in total). Pour Water is tested with trials for each room (three unseen rooms, 24 trials in total). Pour Water-L--133 and Pour Water-R-20 Handover, Fold Shorts, and Robot Dog are tested with 25 trials each. All the tests are performed on the ALOHA dual-arm robot (see App. G for hardware configurations). Experimental details, such as the implementation and hyper-parameters. are elaborated in App. H.
tested with 8 trials each.
Ablation Study. Answering Q5, we have con- ducted ablation studies on the model size, pre- training, and the modeling method to understand their importance. We consider the variants of: RDT (ours): the original RDT. RDT (regress): RDT without diffusion modeling. It models the deterministic mapping (e, o,) + a, RDT (small): RDT without large parameters. It has only 166M parameters. RDT (scratch): RDT without pre-training. It is trained from scratch during fine-tuning. In Table 2, we evaluate these variants in terms of three dimensions of gener- alizability. Table 7 provides a comparison of different variants of RDT as well as baselines.
Table 2: Ablation study results. Here are the success
rates (%) of the original RDT and its three variants in tasks Wash Cup (unseen cup 2, total success rate), Pour Water (unseen room 3, total success rate), and Pour Water-L-I1 (correct amount sub-task). All the models except RDT (scratch) pre-trained before fine-tuning.
VARIANT
UNSEEN
UNSEEN SCENE
INSTRUCTION
NAME
OBJECT
FOLLOWING
RDT (regress)
12.5
50
12.5
RDT (small)
37.5
62.5
RDT (scrutch)
25
62.5
RDT (ours)
50
62.5
100
5.2 RESULTS ANALYSIS
From the results in Table 3, we can see that RDT consistently outperforms other baselines. This is because RDT employs diffusion with a powerful network architecture to model the distribution of multi-modal actions accurately, while discretization and VAE lack accuracy and expressiveness. respectively. Besides, the large number of parameters after large-scale pre-training provides a lot of prior knowledge, which significantly improves the generalizability. Here is a detailed analysis:
Q1 & Q2: RDT can zhoo generalize to unseen objects, scenes, and modalities. In Wash Cup and Pour Water, RDT can still achieve high success rate on unseen scenarios, and its performance is not much different from that on seen ones. In contrast, the other baselines cannot even complete the entire task. In Pour Water-L-1/3 and Pour Water-R-2/33 from the third row of Fig. 5 or Fig. 10 (zoomed-in version), how much water to pour and closely follows the instruction through its actions, even though it has never seen words like "one-third" or "two-thirds". It is precisely because of large-scale pre- pre-training that RDT has seen a large number of diverse objects, scenes, and instructions, leading to such strong zero-shot generalization.
can find that RDT understands precisely which hand to manipulate and

---

Published as a conference paper at ICLR 2025
Table Quantitative results. We report success rates (%) of ACT, VLe RDT (from scratch, no pre-trained), and RDT (ours, pre-trained) for 7 tasks. Sub-columns in each sub-task cell repre- sent different elements (objects, instructions, scenes). ACT is not tnaguage-condtionee and thus unavailable for instruction following. RDT (ours) consistently outperforms others.
Wash Cup: seen cup
unseen cup
unseen cup- (Unseen Object)
Up Cup
Tum
Water
Pou: Out Water
Place Back Cup
Total
ACT
50
12.5
37.5
01
D
0
0
0
0
0
37.5
0
0
0
OpenVIA
0
0
0
0
0
0
0
0
0
0
Octo
RIST (scratch)
12.5
12.5
37.5
12.5
0
0
RDT tours)
87.5
87.5
50
62.5
75
50
50
75
50
87.5
75
50
87.5
62.5
50
50
75
50
Pour Latce-L-1/3 Pour Water-R- R-2/3 (lostruction Following)
Pick Up Bottle
Pour Water
Place Back Bottle
Total
Correct Hand
Correct Amsoun
VLA
Octo
0
0
0
0
0
0
RDT (scratch)
100
75
75
25
62.5
25
62.5
25
100
75
62.5
12.5
RDT (ours)
100
87.5
100
87.5
100
87.5
100
87.5
100
87.5
100
75
Pour Water: unseen room unseen romm unseen room
(Unseen Scene)
Fold Shorts (1-Shot)
Pick Up Rottle
Pour Water
Place Back Bottle
Total
Total
ACT
25
87,5
25
0
50
12,5
37,5
12.5
0
37,5
12,5
D
OpenVILA
0
0
0
0
0
0
0
0
0
D
Octo
12.5
12.5
12.5
0
12.5
RDT (scratic)
62.5
25
87,5
25
25
75
40
RITT (ours)
62.5
100
62.5
62.5
100
62.5
62.5
100
62.5
62.5
100
62.5
68
Handover (S-Shot)
Robot Dog (Dexterity)
Pick Up
Pon
Switch Hund
Drop
Pon
Full ino
Bert
Total
Remete
Gmb
Joystick
Puch
Toaa
Stmight
Walk
ACT
0
0
88
32
32
32
0
D
Octo
D
100
4
RDT (scratch)
88
32
24
16
16
100
64
64
32
RITT (ours)
100
56
56
40
40
100
76
76
48
Q3: RDT can learn new skills using only a few shots. In Handover and Fold Shorts, RDT has learned new and complex skills of handover and folding through few-shot learning, whose action patterns are very different from known skills, while the success rate of others is almost zero. Such improvement is also due to large-scale pre-training. Few-shot learning can help RDT quickly adapt to new working environments, which is of great significance for practical applications.
Q4: RDT can handle dexterous tasks. In Robot Dog, RDT accurately controls the angle when pushing the joystick, while others have caused the robot dog to deviate. This is because diffusion, with our powerful network architecture, can model the distribution of multi-modal and nonlinear actions so that the action precision can meet the requirements of dexterous tasks. We also note that the joystick and the remote control are both black, making the joystick not visually apparent. It probably makes ACT prone to failure. In contrast, large-scale pre-training has made RDT learn a better vision- Maaiuag representation of the joystick concept, improving the recognition capability.
Q5: Large model size, extensive data, and diffusion are all essential factors for our excellence. In Table 2, there is serious performance drop without any of these factors, demonstrating the necessity of our contributions. In particular, RDT (scratch) performs poorly on unseen objects and scenes, indicating that the knowledge from pre-training is critical for generalization.
CONCLUSION
In this paper, we tackled the challenges of data scareity and increased manipulation complexity in generalizable bimanual manipulation by developing the Robotics Diffusion Transformer (RDT), a diffusion-based foundation model for language-conditionel visuomotor imitation learning. Our model was pre- traiera on an extensive multi-robot dataset and fine-tuned on a self-collected bimanual dataset. We further introduce Physically Interpretable Unified Action Space to unify action representations across different robots, enhancing robustness and transferability. Outperforming existing methods, RDT not only demonstrates significant improvements in dexterous bimanual capability and instruction following but also achieves remarkable performance in few-shot learning and zero-shot generalization to unseen objects and scenes.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by NSFC Projects (Nos. 92248303, 92370124, 62350080, 62276149, 92270001). BNRist (BNR2022CC0006) Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J. Zhu was also supported by the XPlorer Prize.
ETHICS STATEMENT
All the data used in this research comes from open-source and well-documented datasets, and strictly follow all applicable licensing and usage guidelines. Our finetuning dataset is collected by the authors of this paper along with volunteers.
While RDT is a model trained for scalable, Iinuagg--oondtioonl visuomotor policy learning and tested the ALOHA dual-arm robot, emphasize that any harmful of our model is neither intended nor encouraged, and we encourage responsible deployment on real-world robots.
REPRODUCIBILIT STATEMENT
To reproduce our pre-training and fine- tuning processes, we have provided the code the repository. We also include instructions for downloading the dataset, how to use the training code, and guide for deploying on a real machine in the README file. We have fully open-sourced all our code, model weights, and fine-tuning datasets. We refer the project page for more information.
Please refer to App. D for pre-training dataset details, App. for fine-tuning dataset details, App. for RDT training details, App. for hardware details, and App. H for experimental details and implementation of baselines.
REFERENCES
TensorFlow Datasets, collection of ready-to-use datasets.
cn/datasets. 23
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: www.w.tensr.f.... org/. Software available from tensorflow.org. 23
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 0rXh::2003.087744 2023. 1, 7,
Stavros Adam, coomatios-Aggelos Alexandropoulos, Panos M Pardalos, and Michael Vrahatis. No free lunch theorem: A review. Approximation and optimization: Algorithms,, complexity and applications, pp. 57-82, 2019. 25
Jorge Aldaco, Travis Armstrong, Robert Baruch, Jeff Bingham, Sanky Chan, Kenneth Draper, Debidatta Dwibedi, Chelsea Finn, Pete Florence, Spencer Goodrich, et al. Aloha 2: An enhanced low-cost hardware for bimanual teleoperation. arXiv preprint arXiv:2405.022292, 2024. 3
Fabio Amadio, Adrià Colomé, and Carme Torras. Exploiting symmetries in reinforcement learning of bimanual robotic tasks. IEEE Robotics and Automation Letters, 4(2):1838-188) 1845, 2019.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 22669-22679, 2023. 6, 19

---

Published as a conference paper at ICLR 2025
Aleksandar Batinica, Bojan Nemec, Aleš Ude, Mirko Raković, and Andrej Gams. Compliant movement primitives in a bimanual setting. In 2017 JEEE-RA 17th International Conference on Humanoid Robotics (Humanoids). pp. 365-371. IEEE, 2017. 3
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Hydra: Hybrid robot actions for imitation learning. arxiv, 2023. 22
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint wrXiv:2212.0688) 2022. 3, 19, 20, 22
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. R1-2: Vision Vision-language-action models transfer web knowledge to robotic control. arXiv preprint ar81v:2307.15818, 2023. 1, 3, 19
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020 7
Lawrence Yunliang Chen, Simeon Adehola, and Ken Goldberg. Berkeley UR5 demonstration dataset.
https://sites.godtt www.ssoooogolooooooooooo www.///iww/bbrccllyy-r/////o/ooo 22
Lili Chen, Shikhar Bahl, and Deepak Pathak. Playfusion: Skill acquisition via diffusion from language-annotann play. In CoRL, 2023. 22
Xin Chen, Aming Wu, and Yahong Han. Capturing the spatio-temporal continuity for video semantic segmentation. IET Image Processing, 13(14)28113-2820 2019. 2,5
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023. 3, 5, 19, 21, 22
Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, and Abhinav Gupta. Efficient bimanual manip- ulation using learned task schemas. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 1149- 1155. IEEE, 2020. 3
Open X-Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew Wang, Andrey Kolobov, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Beomjoon Kim, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu, Charlotte Le, Chelsea Finn, Chen Wang. Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Driess, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns. Ethan Foster, Fangchen Liu, Federico Ceola, Fei Xia, Feiyu Zhao, Felipe Vieira Frujeri, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guangwen Yang. Guanzhi Wang, Hao Su, Hao-Shu Fang, Haochen Shi, Henghui Bao, Heni Ben Amor, Henrik Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jachyung Kim, Jaimyn Drake, Jan Peters, Jan Schneider, Jasmine Hsu, Jeannette Bohg. Jeffrey Bingham. Jeffrey Wu, Jensen Gao, Jiaheng Hu, Jiajun Wu, Jialin Wu, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jingyun Yang, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kaiyuan Wang, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Lin, Kevin Zhang, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Lawrence Yunliang Chen, Lerrel Pinto. Li Fei-Fei, Liam Tan, Linxi "Jim" Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina,
12

---

Published as a conference paper at ICLR 2025
Mateo Guaman Castro, Max Spero, Maximilian Du Michael Ahn, Michael C. Yip, Mingtong
Zhang, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim. Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Ning Liu, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R Sanketi, Patrick "Tree" Miller, Patrick Yin, Paul Wohlhart, Peng Xu, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Quan Vuong. Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Mart iarttin-Marttii in, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Ruohan Zhang, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Shan Lin, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davehev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiangyu Chen, Xiaolong Wang. Xinghao Zhu, Xinyang Geng, Xiyuan Liu, Xu Liangwei, Xuanlin Li, Yansong Pang, Yao Lu, Yecheng Jason Ma, Yejin Kim, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Yilin Wu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yongqiang Dou, Yoonyoung Cho. Youngwoon Lee, Yuchen Cui, Yue Cao. Yuch-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang. Yunfan Jiang, Yunshuang Li, Yunzhu Li. Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zhuo Xu, Zichen Jeff Cui, Zichen Zhang, Zipeng Fu, and Zipeng Lin. Open X- X-Embodiment: Robotic learning datasets and RT-X models. https: //wrx/aaavvvv///////aa//aa wtggabs/////aooaaoorr/....s....r /2taab/////// 221 08864, 2023. 1, 2, 3, 4, 5, 21, 23
Adria Colomé and Carme Torras Dimensionality reduction for dynamic movement primitives and application to bimanual manipulation of clothes. IEEE Transactions on Robotics, 34(3)x602-615. 2018. 3
Adrià Colomé and Carme Torras. Reinforcement learning of bimamual robot skills. Springer, 2020. 3
Shivin Dass, Jullian Yapeter, Jesse Zhang. Jiahui Zhang, Karl Pertsch, Stefanos Nikolaidis, and Joseph J. Lim. Clvr jaco play dataset, 2023. URL https: http:p:://gttti http://ggttuuub..c///////aaaaaa/ wwtp:////ccccccwa//c/ccce/ 22
jaco_ppay_dataaee
Carlos Canudas de Wit, Bruno Siciliano, and Georges Bastin. Theory of robot control. Springer
Science& & Business Media, 2012. 2, 7,
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprini arXiv:2303.00 03378, 2023. 3
Aaron Edsinger and Charles C Kemp. Two arms are better than one: A behavior based control system for assistive bimanual manipulation. In Recent Progress Robotics: Viable Robotic Service to Human: An Edition of the Selected Papers from the 13th International Conference on Advanced Robotics, pp. 345-355. Springer, 2007.
Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023. 2, 3, 20, 21, 22
Niko Sünderhauf Federico Ceola, Krishan Rana. Lhmanip: A dataset for long horizon manipulation tasks., 2023. 22
Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Finetuning offline world models in the real world. arXiv preprint arXiv:2310.16029, 2023. 22
Nadia Figueroa and Aude Billard. Learning complex manipulation tasks from heterogeneous and unstructured demonstrations. In Proceedings of Workshop on Synergies between Learning and Interaction, 2017.
Giovanni Franzese, Leandro de Souza Rosa, Tim Verburg, Luka Peternel, and Jens Kober. Interactive imitation learning of bimanual movement primitives. IEEE/ASME Transactions on Mechatronics, 2023. 1,3
13

---

Published as a conference paper at ICLR 2025
Zipeng Fu. Tony Z Zhao, and Chelsea Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. arXiv preprint arXiv:2401.02117, 2024. 3, 21, 22, 24, 25
Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, al. Octo: An open-source generalist robot policy, 2023. 2, 3, 9, 21, 25
Jennifer Grannen, Yilin Wu, Suneel Belkhale, and Dorsa Sadigh. Learning bimanual scooping policies for food acquisition. In Conference on Robot Learning, pp. 1510-1519. PMLR, 2023a. 1,
Jennifer Gramnen, Yilin Wu, Brandon Vu, and Dorsa Sadigh. Stabilize to act: Learning to coordinate for bimanual manipulation. In Conference Robot Learning, pp. 563-576. PMLR, 2023b. 1,
Markus Grotz, Mohit Shridhar, Tamim Asfour, and Dieter Fox. Peract2: perceiver actor framework for bimanual manipulation tasks. arXiv preprint arXiv:2407.000275
2024. 1,
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In International Conference on Learning Representations, 2023. 22
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
1.Xiv:1606.08415. 2016. 25
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint arXiv:2010.04245 2020. 6, 19
Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In Robotics: Science and Systems, 2023. 22
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1866800--851,, 2020. 2,
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
100.:260669685. 2021. 26
Nan Huang, Christian Kümmerle, and Xiang Zhang. Unitnorm: Rethinking normalization for transformers in time series. arXiv preprint arXiv:2405. 15903, 2024.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference on Robot Learning, 2021. URL ht https: //pp////epeeeeeeeee/ wwt.:eet//ooooeee (d38kbp23ttGGX 22
Xiaogang Jia, Denis Blessing, Xinkai Jiang, Moritz Reuss, Atalay Donat, Rudolf Lioutikov, and Gerhard Neumann. Towards diverse behaviors: A benchmark for imitation learning with human demonstrations. arXiv preprint 2402 14606, 2024.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 1806. 10293, 2018. 22
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint 1rXiv:2403.12945, 2024. 20,
Minchan Kim, Junhyek Han, Jachyung Kim, and Beomjoon Kim. Pre-and post-contact policy decomposition for nonppeehe manipulation with zero-shot sim-to-real transfer. 2023. 22
14

---

Published as a conference paper at ICLR 2025
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision- aation-laguage-acttoo model. arXiv preprint arXiv:2406. 09246, 2024. 1, 2, 3, 4, 9, 19, 25, 26
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yea Lo, al. Segment anything. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Basil Kouvaritakis and Mark Cannon. Model predictive control. Switzerland: Springer International Publishing, 38:13-56, 2016. 7
Franziska Krebs, Andre Meixner, Isabel Patzer, and Tamim Asfour. The kit bimanual manipulation dataset. In 2020 JEEE-RA 20th International Conference on Humanoid Robots (Humanoids), pp. 499-506. IEEE, 2021.
Vikash Kumar, Rutav Shah, Gaoyue Zhou, Vincent Moens, Vittorio Caggiano, Abhishek Gupta, and Aravind Rajeswaran. Robohive: A unified framework for robot learning. Advances in Neural Information Processing Systems, 36, 2024. 2, 21, 22
Michelle A Lec, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 IEEE International Conference Robotics and Automation (ICRA), 2019. URL https: http://aww.w..raaaaa/////aaa 22
Weiwei Li. Optimal control for biological movement systems. University California, San Diego, 2006. 1
Hanwen Liang, Niamul Quader, Zhixiang Chi, Lizhe Chen, Peng Dai, Juwei Lu, and Yang Wang. Self- supervised spatiotemporal representation learning by exploiting video continuity. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1564-1573, 2022. 2,5
Rudolf Lioutikov, Oliver Kroemer, Guilherme Maeda, and Jan Peters. Learning manipulation by sequencing motor primitives with a two-armed robot. In Intelligent Autonomous Systems 13: Proceedings of the 13th International Conference IAS- 13, pp. 1601-1611. Springer, 2016.
Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. Robot learning on the job: fluman-in-the-loge autonomy and learning during deployment. In Robotics: Science and Systems (RSS), 2023. 22
I Liu, Chun Arthur, Sicheng He, Daniel Seita, and Gaurav Sukhatme. Voxact-b: Voxel-based acting and stabilizing policy for bimanual manipulation. arXiv preprint arXiv:2407.04152 2024. 1,3
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European Conference on Computer Vision, pp. 531-548. Springer, 2022. 19
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX11:2211.01095. 2022. 8, 25
Jianlan Luo, Charles Xu, Xinyang Geng, Gilbert Feng, Kuan Fang. Liam Tan, Stefan Schaal, and Sergey Levine. Multi-stage cable routing through hierarchical imitation learning. arXiv pre-print, 2023. URL https: /ttp/aaavvv////////a///ee////waa org/ abs 2307.33..00 08927. 22
Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel, and Sergey Levine. Fmb: functional manipulation benchmark for generalizable robotic learning, 2024. URL https: wxxii.ii.. otg/abbs. /to/////411. 2401.08552 08553. 22
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Anmstrong, and Pete Florence. Interactive language: Talking to robots in real time, 2022. URL https: / a rxiv. www..org//ssoo... /211 0..21....66664 07. 22
15

---

Published as a conference paper at ICLR 2025
Tatsuya Matsushima, Hiroki Furuta, Yusuke Iwasawa, and Yutaka Matsuo. Weblab xarm dataset, 2023. 22
Oier Mees, Lukas Hermann, Erick Rosete-Best and Wolfram Burgard. Calvin: A benchmark for policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L). 7(3)77327-7334, 2022. 22
Seyed Sina Mirrazavi Salehian, Nadia Barbara Figueroa Fernandez, and Aude Billard. Dynamical system-based motion planning for multi-arm systems: Reaching for moving objects. In IJCAI'77 Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 4914-4918, 2017. 1,4
Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference Robot Learning (CoRL.), 2022. 22
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffuusion probabilistic models. In International conference machine learning, pp. 8162-8171, PMLR, 2021.
Jihoon Oh, Naoaki Kanazawa, and Kento Kawaharazuka. X-embodiment u-tokyo pr2 datasets, 2023.
URL httph::///// ptttps://githute con/
22
Takayuki Osa. Motion planning by learning the solution manifold in trajectory optimization. The International Journal of Robotics Research, 41(3):291-31, 2022. 22
Abhishek Padalkar, Gabriel Quere, Antonin Raffin, João Silvério, and Freek Stulp. A guided reinforcement learning approach using shared control templates for learning manipulation skills in the real world. Research square preprint rs-3289569/11, 2023. 22
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-135) 2009. 2
Jyothish Pari, Nur Muhammad Shafiullah, Sridhar Pandian Arunachalam, and Lerrel Pinto. The surprising effectiveness of representation learning for visual imitation, 2021. 22
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perfoperformmnce deep learning library. Advances in neural information processing systems, 32, 2019. 23
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. The Eleventh International Conference on Learning Representationss 2023. 3,5
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023. 2, 6, 19
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. 4
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- PMLR, 2021.1
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In CoRL, 2022. 22
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http://jml r, www.lr.rrg/gpppee wwwppaperr///aoooooooooop..earar hampers/v21/200 074 html, 1, 6, 19, 25
16

---

Published as a conference paper at ICLR 2025
Daniel Rakita, Bilge Mutlu, Michael Gleicher, and Laura M Hiatt. Shared control- based bimanual robot manipulation. Science Robotics, x(30))aaaw095, 2019. 1,4
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020. 23
RethinkRobotics. Sawyer performing table top manipulation. https://github.cm com/ httphhiinnnoooiisssssswweeeeeeee
22
Erick Rosete-Beant Beess, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task agnostic offline reinforcement learning. 2022. 22
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JML. Workshop and Conference Proceedings, 2011. 19
Saumya Saxena, Mohit Sharma, and Oliver Kroemer. Multi-resolution sensing for real-time control with vision-language models. In 7th Annual Conference on Robot Learning, 2023. URL https: //opppnevvee. net/ forum? 2idimmidddWyyyy d-WuBvW------ IIDUA. 22
Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrll Pinto. On bringing robots home. arXiv preprint (tti::311..60099 2023. 22
Dhruv Shah, Ajay Sridhar, Arjun Bhorkar, Noriaki Hirose, and Sergey Levine. Gnm: A general navigation model to drive any robot. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7226-7233. IEEE, 2023a. 2.7
Rutav Shah, Roberto Martín-Martín and Yuke Zhu. MUTEX: Learning unified policies from multimodal task specifications. In 7th Annual Conference on Robot Learning, 2023b. URL https: /cppneevee.eee.eeooooeeee..oeeee net forrmormmm// wedidddddwwwaaaaaaae 22
Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In Conference on robot learning, pp. 906-915. PMLR, 2018. 3,
Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, and Jiajun Wu. Robocook: Long-horizon clasto-plastic object manipulation with diverse tools. arXiv preprint MKKi::2306.1444/4 2023. 22
Christian Smith, Yiannis Karayiannidis, Lazaros Nalpantidis, Xavi Gratal, Peng Qi, Dimos V Dimarogonas, and Danica Kragic. Dual arm manipulation a survey. Robotics and Autonomous systems, 60(10):1340-135: 2012. 3
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015. 2
Simon Stepputtis, Joseph Campbell, Mariano Phiclipp. Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditionn imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-13150, 2020. 3
Simon Stepputtis, Maryam Bandari, Stefan Schaal, and Heni Ben Amor. system for imitation learn- ing of contact-rich bimanual manipulation policies. In 2022 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11810-11817. IEEE, 2022.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Kridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems. 33:7537-7547, 2020. 6
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ama Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023. 1,7
17

---

Published as a conference paper at ICLR 2025
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Caamenn-Estuuch Quan Vuong, Andre He, Vivek Myers, Kuan Fang. Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference Robot Learning (CoRL.,, 2023. 2, 21, 22
Zhiqiang Wang. Hao Zheng, Yunshuang Nie, Wenjun Xu, Qingwei Wang, Hua Ye, Zhe Li, Kaidong Zhang, Xuewen Cheng, Wanxi Dong, et al. All robots in one: new standard and unified dataset for versatile, general-puryy purpose embodied agents. arXiv preprint arXiv:2408. 10899, 2024. 21
Lingxuan Wu, Xiao Yang, Yinpeng Dong, Liuwei Xie, Hang Su, and Jun Zhu. Embodied ac- tive defense: Leveraging recurrent feedback to counter adversarial patches. arXiv preprint 04XYv:2404.00540, 2024. 21
Fan Xie, Alexander Chowdhury, M De Paolis Kaluza, Linfeng Zhao, Lawson Wong, and Rose Yu. Deep imitation learning for bimanual robotic manipulation. Advances in neural information processing systems, 33:2327-2337, 2020. 3
Ge Yan, Kris Wu, and Xiaolong Wang. uesd kitchens Dataset. August 2023. 22
Jonathan Yang, Dorsa Sadigh, and Chelsea Finn. Polybot: Training one policy across robots while embracing variability. arXiv preprint arXiv:2307.03779. 2023,
Jonathan Yang. Catherine Glossop. Arjun Bhorkar, Dhruv Shah, Quan Vuong, Chelsea Finn, Dorsa Sadigh, and Sergey Levine. Pushing the limits cross-embodiment learning for manipulation and navigation. arXiv preprint arXiv:2402. 19432, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- vre-traniin, 2023. 6, 19, 25
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019. 6, 19
Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 2023. 1, 2, 5, 9, 19, 21, 22, 25
Gaoyue Zhou, Victoria Dean, Mohan Kumar Srirama, Aravind Rajeswaran, Jyothish Pari, Kyle Hatch, Aryan Jain, Tianhe Yu, Pieter Abbeel, Lerrel Pinto, Chelsea Finn, and Abhinav Gupta. Train offline, test online: A real robot learning benchmark. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023. 22
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In 20/9 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5738-5746, 2019. doi: 10.1 109//CVP.2019.000588 21
Xinghao Zhu, Ran Tian, Chenfeng Xu, Mingyu Ding, Wei Zhan, and Masayoshi Tomizuka. Fanuc manipulation: A dataset for learning-based manipulation with fanuc mate 200id robot. 2023. 22
Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. 6th Annual Conference Robot Learning (CoRL., 2022a. 22
Yifeng Zhu, Peter Stone, and Yuke Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEFE Robotics and Aulomation Letters, 7(2):4126-41333 2022b. 22
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1909,v
2019. 7
R Zollner, Tamim Asfour, and Rüdiger Dillmann. Programming by demonstration: dual-arm manipulation tasks for humanoid robots. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IRES)IEEE Cat. No. 04CH37566), volume pp. 479 484. IEEE, 2004.
18

---

Published as a conference paper at ICLR 2025
A ACTION CHUNKING TECHNIQUE
In practice, we find that the errors in action prediction accumulate as the number of historical decisions increases due to the imperfection of the learned policy. This may cause the robot drift out of the training distribution, reaching hard-to-recover states (Ross et al., 2011). To alleviate this, we prefer to predict multiple actions in one shot, thereby reducing the total number of decisions in a trajectory. In this way, we model p(a 14, oz), where d2.++T, denotes the chunk size (Zhao et al., 2023). To adapt Eq. and Eq. 2 to this context, we could simply replace ae by at.t+7.. Besides, according to Chi et al. (2023), action chunking is also helpful for improving temporal consistency. It can better consider the coherence of previous and subsequent actions when making decisions and may avoid sudden changes in actions that may cause damage to the robot.
(at..... ... Q++T.-1] is action chunk and To
B ARCHITECTURE DETAILS
Encoding of Multi-Modal Inputs. Encoding details are outlined below:
Low-Dimensional Inputs. The proprioception &, and the noisy action chunk dz.+T, are first embedded into the unified action space. This space is used to unify the representation of and aes+T, across various robots, which is elaborated in Sec. 4.2. Then, they are encoded into the token space by shared MLP since they have similar physical meanings. Such continuous encoding can avoid precision loss in contrast to discretized encoding (Brohan et al., 2022; 2023; Kim et al., 2024). For frequency c well the diffusion time step k; we encode them into the token space through two MLPs, respectively. Afterward, all of them are concatenated together in the length direction to achieve in in-context conditioning (Peebles & Xie, 2023; Bao et al., 2023), resulting in an input token sequence of length 1 + Ta + 1 + 1. Finally, position embeddings are added to distinguish different modalities and to inject temporal information in ā.:+T..
Image Inputs. We encode the RGB images by a frozen SigLIP (Zhai et al.. 2023) and utilize an additional MLP to project the output to the token space. To enhance the model's ability to distinguish images based on viewpoint and time steps. we extend traditional sinusoidal positional embeddings multi-dimensiinal grids, as shown on the right side of Fig. 3. This modification integrates -patial-temporal information, enabling the model to capture the relationships between input images. Specifically, we adopt the implementation by Liu et al. (2022), employing grid dimensions of (Timg. Neam; Npatch. D). Here, Neam represents the number of cameras, set to three in our configuration, and Npach indicates the number of patches into which each image is divided by the ViT-based Image Encoder and D denotes the embedding dimension.
Language Inputs. Language instruction is encoded by a frozen T5-XXL (Raffel et al., 2020), and an MLP is used to project the output to the token space. When calculating attention for language tokens, we apply the language attention mask to mask out the pad tokens appended during batching.
During training, each input from various modalities is independently masked with a probability of 10%.
Network Structure of fe. After encoding. we feed the tokens of the low low-dimensional inputs into the main network, which is adjusted from Diffusion Transformers (DiTs) with Cross-Attentin (Peebles & Xie, 2023) due to their high scalability. For better training stability, we add QKNorm (Henry et al., 2020) into each attention layer and replace each LayerNorm with RMSNorm (Zhang & Sennrich, 2019). In each DIT block's cross-attention layer, we alternately inject language and image tokens rather than simultancously inject both, avoiding the issue of token imbalance between the two modalities. After L DiT blocks, normalize the output and project it back to the action space via an MLP decoder.
C PHYSICALLY INTERPRETABLE UNIFIED ACTION SPACE
As mentioned in Sec. 4.2, embed the actions of various robots into one unified space that includes all the main physical quantities of robots. This unified action space has a dimensionality of 128. Table 4 describes each element of the vector in this unified action space. For specific robot, each
19

---

Published as a conference paper at ICLR 2025
element of the raw action vector is filled into the corresponding position of the unified action vector according to its physical meanings, with the remaining positions being padded.
Index Range
Element Index
Mapped Physical Quantity
[0, 10)
0-9
Right arm joint positions
[10, 15)
10-14
Right gripper joint positions Right arm joint velocities Right gripper joint velocities Right end effector positions
[15, 25)
15-24
[25, 30)
25-29
[30, 33)
30-32
[33, 39)
33-38
Right end effector 6D pose Right end effector velocities
[39, 42)
39-41
[42, 45)
42-44
Right end effector angular velocities Reserved
[45, 50)
45-49
[50, 60)
50-59
Left arm joint positions
[60, 65)
60-64
Left gripper joint positions
[65, 75)
65-74
Left arm joint velocities
[75, B0)
75-79
Left gripper joint velocities Left end effector positions
[80, 83)
80-82
[83, 89)
83-88
Left end effector 6D pose Left end effector velocities
[89, 92)
89-91
[92, 95)
92-94
Left end effector angular velocities
[95, 100)
95-99
Reserved
[100, 102)
100-101
Base linear velocities
[102, 103)
102
Base angular velocities Reserved
[103, 128)
103-127
Table 4: Description of the unified action space vector. For single-arm robot cases, its arm is mapped to the "right" arm. For robot arm with only DoF, its joint positions will be filled in the first 6 of the 10 corresponding positions. The same is true for other physical quantities.
D PRE-TRAINIA DATASETS
Our pre- pre-training dataset collection includes 46 datasets, with a total scale of 1M+ trajectories and 21TB, making it the largest pre-training vee-training collection of robotics datasets to date. Table presents the complete list of our pre-training datasets and their sampling weights. We assign an initial weight of /N, each dataset with size N, and adjust it according the diversity and quality of each dataset. Compared to linear weighting, this approach prevents excessive sampling of large datasets while ensuring smaller datasets are adequately sampled, thus enhancing the diversity of pre-training samples in each mini-batch. During the pre- pre-training stage, we further observed and adjusted the weights of different datasets based on their intermediate loss results. We increased the weights of those slow-convergent datasets.
Main Datasets.
We list some main datasets as follows:
RT-1 Dataset (Brohan et al., 2022) a large diversve dataset including 130K trajectories with multiple tasks, objects and environments. It is collected across 13 different embodiments, each equipping a single exterior RGB camera. The action space includes the 6D end effector (EEF), gripper open, and base displacement with control frequency of 3Hz.
DROID (Khazatsky et al., 2024) is large-scale multi-task dataset with 76K trajectories and 564 scenes. It is collected via teleoperating Franka Panda 7-DoF Robot Arm, with both wrist and exterior RGB-D cameras. The action space includes 7-DoF joint positions and gripper width, while the proprioception additionally includes the 6I) EEF with control frequency of 15Hz.
RH20T (Fang et al., 2023) a comprehensive dataset covering 1 110K trajectories and 140 tasks. It includes four different robotic embodiments and three different camera views, sampled at frequency of 10Hz. It also includes both long and short tasks. Its state space a mix of 6-DoF and 7-DoF joint positions, and it features third-person perspective RGB-D camera.
20

---

Published as a conference paper at ICLR 2025
Mobile ALOHA Dataset (Fu et al., 2024) is bimanual dataset containing IK+ trajectories col- lected by the Mobile ALOHA robot. Its state space includes base movements and 14-dimensional joint positions of both hands, along with three four first-person perspective cameras. Some of its data includes wide-ranging perspective changes and base movements, which were originally suitable for imitation learning.
Other Datasets. The other data come from RH20T (Fang et al., 2023), RoboSet (Kumar et al., 2024), BridgeData V2 (Walke et al., 2023), and Open X-Embodiment (Collaboration et al., 2023). Most of them feature different robotic morphology and camera observation, enhancing both heterogeneity and variety of our pretraining datasets.
Data Cleaning. Repetitive episodes and episodes of failure are excluded to ensure the quality of the pre-training datasets. We remove blank images, exclude erroneously recorded velocities, and filter out overly short trajectories. Overlength trajectories will be downsampled to avoid unfairness.
Preprocessing Multi-Modal Ohservation/Action Inputs. of each modality:
We describe the preprocessing details
Language Instruction L. We perform a simple cleaning on the raw text, such removing illegal characters and extra spaces, capitalizing the beginning of sentences, and adding a period at the end of sentences. We leave the text variable-length.
RGB Images Tim We employ fixed-length image input strategy. We fix the image input order and format for all robots, with a total of three views: static exterior view, a right-wrist view, and a left-wrist view, deemed sufficient for the requirements of most bimanual tasks. We treat a single-arm robot's wrist camera as the right-wrist one and pad the unavailable views with the background color. When fed into the model, each image padded into a square and resized to 384 x 384, keeping its origin aspect ratio. Besides, we choose Timg = 2 since history length of two adequate for most situations, striking a balance between efficiency and performance (Ghosh et al., 2023; Wu et al., 2024). Finally, we can write the image inputs as X == (( X²- X (X) x,,x;)))
Proprioception z, and Action Chunk a.a+T. We roughly align the scales of various datasets by unifying the units of physical quantities (m, rad, m/s, rad/s, etc) rather than strictly normalizing to [-1, 1] or (0, 1) as in prior work (Chi et al., 2023; Ghosh et al., 2023). For example, "1 (m)" different datasets corresponds the same real-world length. Rescaling the physical quantities will destroy such shared properties and thus impair the model's ability to transfer across robots. We also employ the 6D representation (Zhou al., 2019) for the EEF rotation to overcome the gimbal lock issue.
Before choosing T. 64, we have referred to the previous ablation studies by Zhao al. (2023) and balanced between the performance and computational overhead. Besides, historical proprioceptions z,, are excluded to prevent the model from learning shortcuts using the low-dimensional inputs only and thus sticking to fixed motion patterns. Instead, we encourage the model learn generalizable decision-making structures from high-dimensional image features.
Control Frequency c. addressing the challenge posed by differing control frequencies across datasets, we feed the control frequency into the model, allowing the model to take this variation into account when making decisions.
E FINE-TUNING DATASET
Our fine-tuning dataset is created using Mobile ALOHA robot (Fu et al., 2024), including 300+ tasks, 6K+ trajectories, and 3M+ frames. It is also one of the largest open-source multi-task bimanual robot datasets to date. Fig. 6 gives summary of this dataset. We have borrowed 3 tasks (140 episodes in total) from the open-source Songling dataset (Wang et al., 2024).
Multi-Modal Features. We collect the dataset with three RGB cameras positioned at the front and on the left and right grippers. We record dual-arm 6-DoF joint positions and velocities, along with the gripper angles. We manually annotated instructions for each task. To further augment our instructions and align them with the pre- pre-training datasets, we utilize GPT-4-Turbo (Achiam et al., 2023) to generate 100 expanded instructions and one simplified instruction for each task. This multi-moda information further enhances the richness and quality of our dataset.
21

---

Published as a conference paper at ICLR 2025
Pre-Training Dataset
Sample Percentage (%)
RT-1 Dataset (Brohan et al., 2022)
9.00
TACO Dataset Beas al., 2022)
1.99
JACO Play Dataset (Dass et al., 2023)
1.10
Cable Routing Dataset (Luo et al., 2023)
0.27
NYU Door Opening (Pari et al., 2021)
0.33
Viola (Zhu et al., 2022a)
0,40
Berkeley UR5 (Chen et al.)
1.06
TOTO (Zhou et al., 2023)
1.06
Kuka (Kalashnikov et al., 2018)
1.66
Language Table (Lynch al., 2022)
3.32
Columbia Cairlab Pusht Real (Chi et al., 2023)
0,40
Stanford Kuka Multimodal Dataset (Lee et al., 2019)
1.83
Stanford Hydra Dataset (Belkhale al., 2023)
0.80
Austin Buds Dataset (Zhu et al., 2022b)
0.23
Maniskill Dataset (Gu al., 2023)
5.78
Furniture Bench Dataset (Heo et al., 2023)
2.36
UCSD Kitchen Dataset (Yan al., 2023)
0,40
UCSD Pick And Place Dataset (Feng et al., 2023)
1.23
Austin Sailor Dataset (Nasiriany al., 2022)
0,50
Austin Sirius Dataset (Liu et al., 2023)
0.80
BC Z (Jang et al., 2021)
6.91
UTokyo PR2 Opening Fridge (Oh al., 2023)
0,30
UTokyo PR2 Tabletop Manipulation (Oh al., 2023)
0,50
UTokyo Xarm Pick And Place (Matsushima et al., 2023)
0.33
UTokyo Xarm Bimanual (Matsushima et al., 2023)
0.03
Berkeley MVP (Radosavovic al., 2022)
0.73
Berkeley RPT (Radosavovic et al., 2022)
1.00
KAIST Nonprehensile (Kim al., 2023)
0.46
Tokyo U LSMO (Osa, 2022)
0.23
DLR Sara Grid Clamp (Padalkar et al., 2023)
0.03
Robocook (Shi et al., 2023)
1.66
Imperialcolllge Sawyer Wrist Cam (Rethinkkobottcs)
0.43
lamlab CMU Pickup Insert (Saxena et al., 2023)
0.83
UTAuutii Mutex (Shah et al., 2023b)
1.29
Fanue Manipulation (Zhu et al., 2023)
0,66
Play Fusion (Chen et al., 2023)
0,80
DROID (Khazatsky et al., 2024)
10.06
FMB (Luo et al., 2024)
1.39
Dobb-E (Shafiullah et al., 2023)
1.20
QUT Dexterous Manipulation (Federico Ceola, 2023)
0.46
Aloha Dataset (Zhao al., 2023)
4.98
Mobile Aloha Dataset (Fu et al., 2024)
4.98
RoboSet (Kumar al., 2024)
4.48
RH20T (Fang et al., 2023)
10.99
Calvin Dataset (Mees et al., 2022) BridgeData V2 (Walke et al., 2023)
3.32
7.44
Table The pre-training datasets and their corresponding weights.
Diverse Objects and Scenes. Our dataset includes diverse tasks and scenes, encompassing more than 300 tasks, including skills such as picking up. inserting, writing, pushing, and pulling. features 100+ objects with rigid and non-rigid bodies of various sizes and textures. We collect the dataset 15+ scenes and introduce randomness during data collection for each task, such as varying the initial positions of objects and robots. To further increase diversity, added random lighting conditions. For instance, pouring water was performed under both normal lighting and changing color conditions. These measures further enhance the diversity of dataset.
22

---

Published as a conference paper at ICLR 2025
Overview
Diverse Objects & Scenes
A Augmented Instructions
100+ Diverse Oijects
15+ Divorse Scones
Bimanual
6.Dof Joint Pos. & Vel.
3 Camera Views
200+
Tasks
Various Lighting
Pick the icc the desk Filled
Homan Annotation:
Egisodes
pour ior the goblet, finally put the scoup back on the table.
Franss
Challenging Tasks
Bimanual
Comprehensioe
Desterity
Expanded Anotation:
scoop resting on the desk, which to filled with ice cubes qently transfer the ice cubes the goblet without spilling, then precisely place scoop in its original position on the desk.
Canefully grasp
Apalltleea Selees
wetk viaile
Geelog
Simpliified Amnotation:
Pour ice cibes, from the ios spcop into the goblet
Diclactin Miz
Da Earegutke
Allle Aill kell
Ply dhrging
Unao Eaup
Figure 6: Line-Tuning dataset. Our dataset includes the following key features: (1) Diverse Objects and Scenes. Our dataset contains objects with different properties manipulated in different scenes and conditions. (2) Challenging Tasks. Our dataset incorporates dexterous manipulation, language and vision comprehension, and bimanual tasks. (3) Multi-Modal Features. Our dataset is annotated with rich multi-modal data, including V-eiew RGB cameras, joint information, and augmented instructions.
Challenging Tasks. Various challenging tasks are also considered, encompassing dexterous manipulations, such as unscrewing the cap from a plastic bottle, and comprehension tasks, such as spelling "love" with letter blocks. Furthermore, the dataset includes tasks that integrate both dexterity and comprehension, such solving mathematical equations the whiteboard. Addition- ally, our dataset incorporates bimanual tasks, such as inserting the charging cable into the phone. These complex, high-quality tasks further enhance the model's downstream comprehensibility and generalizability.
F RDT TRAINING DETAILS
Platform. We Pytorch (Paszke et al., 2019) and DeepSpeed (Rasley al., 2020) to facilitate parallel training and employ producer-consumer framework with TensorFlow Dataset (TFD) for fast data loading. Since most of the datasets in the Open X- X-Embodiment (Collaboration al., 2023) are stored in the form of TF FeRoddd, we convert all pre-training datasets into TFRecord for storage. pre-training, we use the producer process to decompress the data from TFRecord and store it in a buffer on the hard disk. At the same time, we use the consumer process to read data from the buffer in a disorderly order and feed it to the model training. This not only decouples the TensorFlow (Abadi al., 2015) and PyTorch environments but also alleviates the training performance loss caused by the small size of the shuffling buffer in the memory. In the fine-tuning stage, since the dataset is relatively small, we additionally implement data reading pipeline using the HDF5 dataset for storage.
Padding Action and Proprioception. To embed a specific robot action into the 128-dimensional unified action space, we need to pad unavailable action elements. The usual practice is to pad with value or a specific value. But "0" actually has a physical meaning. For example, a speed of "0" generally represents stillness relative to the ground. This may confuse the model: Does "0" represent stillness or a filler value? To solve this problem, we concatenate the action and proprioception with a 0-1 vector indicating whether each dimension is padded before encoding them into the token space, resulting in -i6-dimmnsio vector. This can supplement the missing availability information and eliminate confusion.
23

---

Published as a conference paper at ICLR 2025
Inspecting Training Process. During training, for every fixed period. conduct diffusion sampling and compare the sampled actions with the ground truth of the training dataset. Empirically, discover general positive correlation between the Mcan Squared Error (MSE) of the two and the performance of deployment on the robot. This observation allows us to monitor the model's training progress easily. When this MSE converges, we can generally stop training. We note that overly low MSE may also mean overfitting.
Data Augmentation. Overfitting is a common challenge training large neural models. particularly in the fine-tuning phase. We utilize data augmentation techniques to resolve it. We perform image augmentation, including color jittering and image corruption, and add Gaussian noise to the input proprioception with a signal-to-noise ratio (SNR) of 40dB. We also use GPT-4-Turbo to augment and expand the language instructions (Refer to Sec. E for more details on the instruction augmentation).
Some Fine- Tuning Details. During fine-tuning, removed static part at the beginning of each episode, which might be caused by the operator not reacting after the recording started. Our language instructions are sampled from the original manually annotated instruction, the expanded instructions, and the simplified instruction with a probability of one-third. When the expanded instructions are drawn, we evenly sample one from the 100 expanded instructions corresponding the task. We did not apply Classifier-Free Guidance (CFG) because we found that this did not improve the performance of the model but instead brought the unstable robot behavior.
G HARDWARE DETAILS
Parameter
Value
DoF
7 x 2 14
control DESSS
Size
1080 x 700 x 1140
Arm weight
4.2 kg
Arm Payload
3000 (peak)
1500 g (valid)
Arm reach
600 mm mm 653 mm J1: #154, J2: 1655 J3: -3: J4: +106
Arm repeatability
kyptop
Arm working radius Joint motion range
J5: =75 J6: =100 0-80 mm 10 NM
Gripper range Gripper max force
AGLE
Figure 7: Hardware features.
Table 6: Technical specifications.
We provide detailed overview of the hardware configuration of our target dual-arm robot. Our model is deployed and evaluated on the Cobot Mobile ALOHA, robot using the Mobile ALOHA system design (Fu et al., 2024) and manufactured by agilex.ai. ai. The key features of the robot are illustrated in Fig. 7 It is equipped with wrist cameras, front camera, laptop, and onboard battery. The robot's technical specifications are listed in Table 6. is important to note we used the "mobile" ALOHA only to facilitate transportation and testing between various scenes and did not use its autonomous mobility feature during any training or inference stages. Our tasks are still static bimanual manipulation tasks.
H EXPERIMENT DETAILS
The general performance in Fig. I of each method is calculated in three steps. Firstly, we calculate the success rate of a method each task. We take an average of the total success rate and any additional requirement, i.e., the average of the values in the Total column and all columns to its right Table 3. For example, in the Pour Later-L-1/3, we take the average of Total, Correct Hand, and Correct Amount. Secondly, calculate the success rate of
Calculation of Total Performance.
24

---

Published as a conference paper at ICLR 2025
Table Comparision of different baselines. We compare baselines as well as different variants of our model in terms of model size, data size, and modeling scheme.
METHOD NAME
LARGE MODEL
LARGE MULTI-ROBOT DATA
MODELING
ACT (Zhao et al., 2023)
X
X
VAB
OLe (Kim et al., 2024)
/
Discretization
Octo (Ghosh et al., 2023)
X
Diffusion
RDT (scratch)
Diffusion
RDT (small)
X /
Diffusion
RDT (regress)
Regression
RDT (ours)
Diffusion
each dimension of Unseen Object, Unseen Scene, Instruction Following, Few-Shot Learning. and Dexterity by averaging all the tasks this dimension (see Table for the correspondence). Lastly, average the success rates of all the dimensions obtain the overall result.
Implementation and Hyper-Parameters of RDT. We list the details of the multi-modal encoders in Table and the model parameter in Table 9. The image history size is Timg 2, the action chunk size is To 64, the language token space dimension is 4096, the image token space dimension is 1152, and the token space dimension of RDT is 2048. use adaptors to align each modality's s token dimension to 2048. And all adaptors for multi-modal encoders are with GeLU activation (Hendrycks & Gimpel, 2016).
We use the AdamW optimizer (Adam et al., 2019) with a constant learning rate scheduler and papee--arameters Table 10 in the pre-training and fine-tuning stages. The model is pre-trained and finetuned 48 H100 80GB GPUs for 1M steps and 130K steps, respectively. Due to schedul- ing reasons, we did not start fine-tund mee-tunin from the 1M pre-trained checkpoint but chose the 500K checkpoint. During the training stage, (i.e., squaredcos_cap.
the DDPM scheduler with glide cosine scheduler and a step number of 1000. During the sampling stage, we utilize the DPM-Solver++ (Lu al., 2022) with a glide cosine scheduler and sampling step number of 5. During fine-tuning, we also filter out episodes with length lower than 32 and down-sample those with length higher than 2048 to 2048.
Modality
Encoder
Trainable
Adaptor
Language Image Action
TS-XXL (Raffel et al., 2020)
N
2 2-layers MLP
SigLIP (Zhai et al., 2023)
N
2-layers MLP
3 3-layers MLP
Table 8: Encoder configurations of RDT.
Model
Layers
Hidden size
Heads
#Params
RDT-1B
28
2048
32
1.2B
Table 9: Model configurations for RDT.
Implementation and Hlype-PParameter of ACT. We directly employed the same architecture and hyper-parameters of ACT that in the original paper (Fu al., 2024), except for the hyper- syper-parameters in Table 11. We trained ACT with 90% of the 6K fine-tuning episodes for 8000 epochs (about 8 days total), while the remaining 10% is treated as the validation set. We took the checkpoint epoch 5413 as the final outcome, according to the best performance in the validation set.
Implementation and Hyper-Parameters of Open We adopt the official implementation (https: / httpb//tgttttttttt/// com http///pnvlll/aaaaaaa and flagship pre-trained model and checkpoint
25

---

Published as a conference paper at ICLR 2025
Hyper-Paarameter
Value
Batch Size
32 x4x
Learning Rate
10-4
Mixed Precision
bf1 6
Warm-Up Steps
500
31
0.9
82
0.999
Weight Decay
1 x 10-2
1 x 10-8
Table 10: for both pre-training and fine-tuning RDT.
Hyper-Parameter
Value
Batch Size
80x4
Learning Rate
9 x 10-5
Learning Rate for Backbone
4 x 10-5
Table 11: Adapetd hyper-parameters ACT,
https://huggiige http:///wgg..aaccc.ooooo///a//// co//open/la/openvl... For each task evaluation, we further fine-tune the officially pre-trained OpenVLA with all the task-relevant demonstrations (~ 100 episodes) from the fine-tuning dataset to facilitate convergence and train the model to around 95%
action token accuracy suggested by Kim et al. (2024) (htips: 22002222222)) Additionally, we experimented with both full-parameter tuning and LoRA methods using the entire dataset but did not achieve sufficient action token accuracy (approximately 60%) for deployment upon convergence (see Fig. 8). According real-robot testing, such non-convergent checkpoints exhibit completely static random behaviors in the deployment.
thub. com/ conn/ppevlll.
openvla/iss www.la/sssus/////
Concretely, we adhere to the same hyper-parameters claimed in Kim et al. (2024) for fine-tuning via LoRA (Hu et al., 2021) as detailed in Table 12.
Figure 8: The accuracy of action token prediction fluctuates rather than converges with the number of training steps when fine-tuning OpenVLA with the full fine-tuning dataset.
Implementation and Haper-Parameters of Octo. We utilize the official implementation avail- able at https:tp/// /httbubbtbttttggg com/ octo models/octt and the most comprehensive pre-trained model, octo-base 1..5. hosted https: http:://hhggggaa/////ee//// coorciillll -elelk-beeeeee werkeles octo-base-1.5, We follow the officially recommended practices for fine-tuning a bimanual robot, detailed in https: /tll:://gittt github..hhheeeeegg wttp://oooo/ooooeeeeeeeeeeeeeeee 02 02_ finetune new wwwp_obbervatae aariiiinnnntttiitiee ion.ion.a py, employing a hall-maraeeter approach. Addi- tionally, we have incorporated extra image tokenizer process images from the right-wrist camera,
ooto/ /blol ob mwin//xamppmeeeemmmmmmm
26

---

Published as a conference paper at ICLR 2025
Hyper-Paramette
Value
Batch Size
16x8
Learning Rate
10-
Lora Rank
32
Image Augmentation
True
Table 12: Hyper-parameters of fine-tuning Open'VIA for bimanual manipulations.
enhancing the system's manipulation capabilities. Furthermore, by integrating image augmentation during the fine-tuning process, we enhance the performance upon deployment in real-world robotics. We replicate the wrist image tokenizer from the pre-trained model to initialize the right-wrist image tokenizer. Similar to VLA, we only fine-tune octo with the task-relevant demonstrations for each evaluation tasks, for we do not observe sufficient test MSE (approximately 10 10-1) for deployment upon convergence (Fig. 9). Coneretely, we apply the default in Table 13:
with variations listed
Figure 9: The test MSE of action prediction fluctuates rather than converges with the number of training steps when fine-tuning Octo with the full fine-tuning dataset.
Hyper-Parameter
Value
Action Head Type Batch Size Action Chunk Size
DiffusionAction
8x8
8
RandomBrrightness (0. 1)
randomContrast (0.9, 9, 1.1)
Image Augmentation
RandomSaturationn 9, 1.1)
RandomHue 05)
Table 13: Hyper-parameters of fine-tuning Octo for bimanual manipulations.
MORE RESULTS
We further provide a zoom-in view for water-level across 8 trails in instruction inslrwction-folloiing evaluation in Fig. 10.
27

---

Published as a conference paper at ICLR 2025
Left Hand, 1/3 Water Level
Right Hand, 2/3 Water Lovel
Figure 10: Visulization of the resulting water levels across 8 trails in Pour Water-L-1/3 and Pour Water-R-2/3. Left: The water level completed by RDT in each trial is extremely close to the ground- truth 1/3 standard. Right: RDT made one mistake in pouring (empty cup) and mistake in water level, but the other trials were in roughly good agreement with 2/3.
28


Reviews:
[{"review_id": "Ob7nrUH11g", "reviewer": "Reviewer_VxiE", "summary": "This paper develops a 1.2B-parameter robotics foundation model that is trained and evaluated on real robot data for bimanual manipulation. The model is trained with imitation learning: (i) pretrained on 1M trajectories combining available datatsets collected for different robots, and (ii) fine-tuned on a self-collected dataset with 6k demonstrations for a Mobile Aloha robot. The model adopts a diffusion transformer (DiT) architecture that takes multi-modal inputs (images, language, etc.) and generate action chunks with multimodal distribution. \n\nThe model is evaluated on 7 real robot tasks against mainstream baselines. The comparison shows that the model can: (i) generalize zero-shot to novel objects, scenes and language, (ii) learn new skills with few data, (iii) accomplish dexterous tasks. Ablation studies show that larger model and pretraining with large data significantly boost the performance.", "questions": "- The model uses frozen vision encoders. Does it mean existing pretrained visual representation is sufficient for robot manipulation? I wonder if the authors spot any cases where pretrained visual encoder is insufficient and lead to unsatisfactory performance.\n- In pretraining on heterogeneous data with varied control frequency, the model takes control frequency as conditioning. I wonder how this strategy works in practice - does the model learns policy prior corresponding to different control frequency?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper presents a complete and remarkable research work that pushes forward the boundary of large-scale robot learning. \n- The model is developed on top of the diffusion transformer with a unified action space, which allows large-scale pretraining on heterogeneous robot data to boost the performance\n- The authors collect the largest robot dataset for bimanual manipulation with comprehensive task coverage for fine-tuning the model\n- The experiments show that the advantage of the model from a foundation model aspect: generalization, few-shot learning, and scaling behavior", "weaknesses": "- While the paper demonstrates that the foundation model is allows zero-shot and few-shot generalization, and can achieve dexterous manipulation, each of these characteristics is only validated on ~one task and may be insufficient. Evaluations on more tasks and existing benchmark tasks will complete the results.\n- It seems that the baselines are not trained on the complete fine-tuning dataset. This doesn't form an apple-to-apple comparison.\n- The writing of the paper has room for improvement. Some of the sentences are too long, which prevent reading of the paper smoothly.\n- Mistakes in citing papers. For example, it seems that \"Xie et al. 2020\" is wrongly used as the reference of DiT in line 83 and 319.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mUTlgV1KDx", "reviewer": "Reviewer_fucs", "summary": "This paper presents an effort toward building a foundation model for bimanual manipulation. It proposes techniques to unify the action space, enabling training on a very large robot dataset. The authors also scale up the model to 1.2B parameters, making it the largest diffusion transformer model for robotics. In this process, the authors identify several key elements to improve training stability and performance. The resulting model achieves good zero-shot generalization performance on unseen and complex tasks.", "questions": "My questions are mainly also discussed in the weakness. Here is an summary:\n1. Given that the diffusion transformer already contains multiple nonlinear layers, why does adding an additional layer improve performance? Have you conducted ablation studies to support this choice?\n2. In the original UNet diffusion policy, the last layer is a Conv1dBlock. Have you compared the performance of your MLP block with this alternative?\n3. What is the the MLP block performance gains on other tasks?\n4. Can you provide more insight into how the model design specifically addresses high frequency change?\n5. What are the performance gains from using all data through this unified action space compared to previous methods that only use robots with similar action spaces or retain subsets of inputs with a common structure?\n6. How does the proposed padding strategy improve performance compared to zero padding? Could you provide comparative results?\n7. How do these normalization layers integrate into the transformer architecture, and how do they address instability? Have you tested different configurations of these layers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper demonstrates strong performance in scaling up robotics models. It presents several interesting components that improve training stability and performance.\n\nThe unified action space, and especially the padding technique, is interesting.\n\nThe paper shows capabilities on several challenging real-world bimanual manipulation tasks.", "weaknesses": "Several claims are not very precise and not very clear. For example, the authors mention the nonlinearity and high frequency of robotic data. While it is true that the data is nonlinear, how does the proposed method tackle this challenge? The authors argue that changing the last linear layer to an MLP block solves this problem and brings significant performance improvements. While the performance is impressive, I think this requires more careful ablation experiments. Firstly, the entire diffusion transformer is already highly nonlinear due to stacking multiple layers, so why does adding more layers help in this case? Secondly, in the original UNet diffusion policy, the last layer is a Conv1dBlock; have the authors compared with that? Lastly, why is it only evaluated on the dexterity task?\n\nIt is also unclear how the model design choices address the “high frequency of robotic data.” Given that these two claims are highlighted in the abstract as the main challenges, I believe they require more careful analysis and discussion.\n\nAs the unified action representation is a major contribution of this paper, there should also be more analysis of this aspect. For example, what are the performance gains from using all the data because of the unified action space, compared to previous methods that use “robots with similar action spaces (Yang et al., 2023; Ghosh et al., 2023; Kim et al., 2024) or retain only a subset of inputs sharing the same structure (Collaboration et al., 2023; Yang et al., 2024)”? Additionally, what is the performance gain of the proposed padding strategy compared with padding with all zeros?\n\nThe authors argue the necessity of using RMSNorm and QKNorm, but they only show the loss without them (Figure 4(a)), which provides very little information on how effective the proposed approach is and whether it addresses the instability issue. It also does not mention how to integrate these normalization layers within the transformer.\n\nScaling up the models and data is certainly attractive, and the paper shows impressive results. However, most of the analysis are “binary”, which means the results are either with or without. Showing more datapoints (model with different size, using different percentage of the dataset) will present more insights to the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hVv5ngWUBM", "reviewer": "Reviewer_kiFc", "summary": "This paper presents the Robotics Diffusion Transformer (RDT), a large-scale diffusion-based model for bimanual robotics. RDT introduces a Physically Interpretable Unified Action Space to standardize actions across robots, enhancing generalization, and uses diffusion models to handle complex, continuous action distributions. Pre-trained on the largest multi-robot dataset and fine-tuned on a custom bimanual dataset, RDT demonstrates strong zero-shot and few-shot generalization, outperforming existing methods in dexterous, real-world tasks.", "questions": "- Could the authors elaborate on why diffusion models were specifically chosen over other generative methods like VAEs or GANs for this task? While diffusion models show high expressiveness, a comparison or rationale would clarify their unique benefits in bimanual manipulation.\n\n- The Physically Interpretable Unified Action Space is innovative, but how does it handle robots with vastly different kinematics or action constraints?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper introduces a novel application of diffusion models to bimanual manipulation, addressing the high-dimensional, multi-modal action space through a Physically Interpretable Unified Action Space. This approach is a creative extension of diffusion models in robotics, particularly for dual-arm coordination, a challenging domain with limited prior work.\n\n- The model is rigorously tested, with comprehensive experiments demonstrating superior performance over existing baselines. The use of the largest multi-robot dataset and a specialized bimanual dataset for fine-tuning enhances the validity of results and supports the model's effectiveness.\n\n- This work contributes substantially to the field by advancing foundation models for robotic manipulation. RDT’s capabilities for zero-shot generalization, few-shot learning, and instruction following mark a significant step towards adaptable and scalable robotic models, with promising implications for real-world applications.", "weaknesses": "- The paper introduces a Physically Interpretable Unified Action Space for handling data heterogeneity, but additional details on potential limitations or failure cases during training with highly diverse data would be beneficial. This could include examples where action standardization might lead to loss of unique features across robots.\n\n- Although the experiments show impressive results, expanding the evaluation to more varied and complex real-world tasks (beyond the 6,000-episode dataset) and more hardwares(beyond ALOHA) could further validate RDT's robustness.\n\n- The paper proposes several innovative multi-modal encodings (e.g., masking, cross-attention) but lacks ablation studies on these design choices. Showing how each component contributes to performance could clarify their impact on handling visual and language-conditioned tasks effectively.\n\n- There's a typo at L76, it should be \"data\" instead of \"date\" .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tm8zqqzLIV", "reviewer": "Reviewer_dG58", "summary": "This paper proposes a unified action representation to align different robots, facilitating pre-training on diverse robot datasets for bimanual manipulation. Additionally, it introduces a diffusion transformer-based architecture with several modifications for enhancing policy learning, and scaling up with large datasets.", "questions": "1. \"Does the RDT-1B fine-tuning use the entire self-collected dataset, and is it not fine-tuned separately for each task in the evaluation?\"\n2. \"Regarding the computation of success rate for the 'wash cup' task with 'seen cup1': the success rate (SR) for 'get water' is 50, for 'pour water' is 87.5, and for 'place back cup' is also 87.5, yet the overall SR is listed as 50. Since the 'get water' subtask has an SR of 50, and the following subtasks have SRs below 100, how is the total SR calculated as 50?\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "A foundational model for bimanual manipulation is absent in the current community, which is an important direction; aligning different robot embodiments is also a crucial question for pre-training on large-scale datasets. The proposed action representation is simple yet effective for pre-training on diverse robot datasets.", "weaknesses": "Compared to the robot datasets used for pre-training the baseline, such as OpenVLA, this paper appears to use a more diverse set of datasets, including additional bimanual manipulation datasets like ALOHA and Mobile ALOHA, contributing nearly 10% of the total datasets. Fine-tuning a baseline pre-trained on single-arm datasets for a bimanual manipulation setting may result in poor performance on bimanual tasks, making it difficult to demonstrate that using the diffusion transformer architecture is superior to using a large language model as the pre-training backbone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)