RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

## Abstract

Abstract Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and the scarcity of training data. In this paper, we present the Robotics Diffusion Transformer (RDT), a pioneering diffusion foundation model for bimanual manipulation. RDT builds on diffusion models to effectively represent multi-modality, with innovative designs of a scalable Transformer to deal with the heterogeneity of multi-modal inputs and to capture the nonlinearity and high frequency of robotic data. To address data scarcity, we further introduce a Physically Interpretable Unified Action Space, which can unify the action representations of various robots while preserving the physical meanings of original actions, facilitating learning transferrable physical knowledge. With these designs, we managed to pre-train RDT on the largest collection of multi-robot datasets to date and scaled it up to 1.2 1.2 1.2 1.2 B parameters, which is the largest diffusion-based foundation model for robotic manipulation. We finally fine-tuned RDT on a self-created multi-task bimanual dataset with over 6 6 6 6 K+ episodes to refine its manipulation capabilities. Experiments on real robots demonstrate that RDT significantly outperforms existing methods. It exhibits zero-shot generalization to unseen objects and scenes, understands and follows language instructions, learns new skills with just 1 ∼ similar-to \sim ∼ 5 demonstrations, and effectively handles complex, dexterous tasks.
We refer to the project page for the code and videos.

## 1 Introduction

Bimanual manipulation is essential for robots to accomplish real-world tasks (Edsinger & Kemp, ). For practical applications, a useful manipulation policy should be able to generalize to unseen scenarios, such as unseen objects and scenes. However, current approaches either depend on task-specific primitives (Mirrazavi Salehian et al., ; Rakita et al., ; Grannen et al., ) or are limited to small-scale model, data and simple tasks (Krebs et al., ; Franzese et al., ; Grannen et al., ; Zhao et al., ; Grotz et al., ; Liu et al., ), thereby exhibiting only narrow generalization and failing in complex tasks. Following the success in natural language processing (Achiam et al., ; Touvron et al., ) and computer vision (Radford et al., ; Kirillov et al., ), one promising direction to enable generalizable behaviors is to develop a foundation model through imitation learning on large-scale datasets.

However, it is highly non-trivial to develop a bimanual manipulation foundation model. One main reason is that the accessible data for a specific dual-arm robot is significantly scarce (Sharma et al., ; Collaboration et al., ) due to high hardware costs, undermining the data-intensive requirements of training foundational models. Inspired by recent attempts in unimanual manipulation (Brohan et al., ; Kim et al., ), we seek to first pre-train on extensive multi-robot datasets and then fine-tune on the small dataset collected on the target dual-arm robot. This can help us to scale the data size up to three orders of magnitude, having the potential to learn transferrable physics knowledge from datasets of other robots. Nevertheless,
there are two key technical challenges. First, a generalizable foundation model requires a highly capable architecture in terms of both *expressiveness* and *scalabilty*. The dimension of the action space in bimanual manipulation is twice that in unimanual manipulation, bringing a higher degree of *multi-modality* in the distribution of feasible actions (Li, ; Jia et al., ), as illustrated in Fig. 2(b). Accordingly, the model must be expressive enough to capture the multi-modality in action distributions. Previous methods (Zhao et al., ; Brohan et al., ; Kim et al., ) typically fail to meet this standard, leading to unsatisfactory performance. Besides, the architecture needs to effectively process inputs from different modalities, including text, images, and actions. It must be scalable to stably train on large-scale robotic data. Second, date heterogeneity, which is caused by variations in the physical structure and the action space definition across different robots, can lead to negative transfer and impeding policy generalization during training on multi-robot data (Pan & Yang, ). Existing approaches either discard robots with differing action spaces or retain only the parts of the data whose structure is constant across the robot, at the cost of losing valuable data (Brohan et al., ; Ghosh et al., ; Shah et al., ).

Figure: Figure 1: Overview of Robotics Diffusion Transformer with 1B-Parameters (RDT-1B), a language-conditioned visuomotor policy for bimanual manipulation,with state-of-the-art generalizability to unseen scenarios (See App. H for metric calculation details).
Refer to caption: x1.png

In this paper, we introduce the *Robotics Diffusion Transformer (RDT)*, the largest bimanual manipulation foundation model with strong generalizability. RDT employs Diffusion Transformers (DiTs) as its scalable backbone network (Xie et al., ), with special designs for language-conditioned bimanual manipulation with vision. For expressiveness, RDT excels in capturing the full modalities of bimanual actions from massive data by leveraging diffusion models’ capacity for modeling complex distributions (Sohn et al., ; Ho et al., ). For scalability, we harness the Transformer backbone and carefully design the multi-modal encoding to eliminate the heterogeneity of various modalities. To characterize the nonlinear dynamics (de Wit et al., ), high-frequency changes (Ghosh et al., ), and unstable numerical range inherent in robotic data, significantly differing from images and videos with temporal and spatial continuity (Chen et al., ; Liang et al., ), we make important modifications to the original DiT structure, including MLP decoding, improved normalization, and alternating condition injecting (See Fig. 4 for their importance). To further enable training RDT on heterogeneous data, we propose the *Physically Interpretable Unified Action Space*, a unified action format for various robots with gripper arms. This innovative format mitigates potential conflicts between different robots while retaining the physical meanings of the original actions, which can promote the model to learn generalizable physical knowledge across diverse robotic datasets.

With the above designs, we managed to pre-train the RDT model on the largest collection of multi-robot datasets to date (Collaboration et al., ; Walke et al., ; Fang et al., ; Kumar et al., ) and scale it up to $1.2$B parameters, which is the largest diffusion-based pre-trained model for robotic manipulation. To further enhance its bimanual manipulation capabilities, we fine-tuned the RDT on a self-collected multi-task bimanual dataset comprising over $6$K+ trajectories, which is one of the most extensive bimanual datasets. In our experiments, we have comprehensively evaluated RDT against strong baselines in both bimanual manipulation and robotic foundation models. Results show that RDT achieves state-of-the-art performance, outperforming baselines by achieving an improvement of $56\%$ in success rates across a wide spectrum of challenging tasks. In particular, RDT has exceptional zero-shot and few-shot ($1\sim 5$ shots) generalizability to unseen objects, scenes, instructions, and even skills. RDT is also capable of accomplishing tasks requiring fine-grained operations, such as controlling a robot dog with a joystick. Finally, ablation studies show that diffusion modeling, large model size, and large data size all contribute to superior performance.

## 2 Related Work

Learning-based Bimanual Manipulation.
One substantial challenge in learning a bimanual manipulation policy is the high dimensionality of the action space, which exacerbates the data scarsity (Zollner et al., ; Smith et al., ; Lioutikov et al., ; Stepputtis et al., ) and the multi-modal behavior (Colomé & Torras, ; ; Figueroa & Billard, ; Sharma et al., ; Xie et al., ; Franzese et al., ). Some works have developed more cost-effective interfaces for data collection (Zhao et al., ; Aldaco et al., ), but they are limited to specific hardware configurations and still insufficient to bridge the data gap for a generalizable policy.
Others attempt to reduce data requirements by introducing inductive biases, such as distinguishing two arms for stabilization and functionality (Grannen et al., ), parameterizing movement primitives (Batinica et al., ; Amadio et al., ; Chitnis et al., ; Franzese et al., ), or using voxel representations (Grotz et al., ; Liu et al., ). These methods use strong priors or simplified modeling, which successfully reduce the action space, but at the cost of a reduced scope of application and failing to express the multi-modality of bimanual behaviors (Pearce et al., ).

Foundation Models for Robotics. Foundation models have shown immense promise in enabling generalizable behaviors by training multi-task “generalist” models (Brohan et al., ; ; Ghosh et al., ; Kim et al., ) on large multi-task robot datasets (Collaboration et al., ; Brohan et al., ; Fang et al., ). Most studies adapt large vision-language models to directly predict action (Brohan et al., ; Driess et al., ; Brohan et al., ; Collaboration et al., ; Kim et al., ). While demonstrating generalization to new objects and tasks, they face issues with quantization errors and uncoordinated behaviors (Pearce et al., ) when applied to bimanual manipulation, largely due to their discretization of action spaces. To enhance precision, diffusion models have been used for continuous control (Ho et al., ; Chi et al., ; Pearce et al., ; Ghosh et al., ). Ghosh et al. () pre-train a Transformer-based diffusion policy on a subset of Open X-Embodiment (Collaboration et al., ) dataset ($25$ datasets), with up to $93$M parameters.

## 3 Problem Formulation and Challenges

We start by formulating the task and elaborating on the challenges. To evaluate the model on the hardware, we choose the ALOHA dual-arm robot as our target robot since it is one of the most representative dual-arm robots and is suitable for collecting human demonstration data via teleoperation (Zhao et al., ; Fu et al., ; Aldaco et al., ). Fig. 2(a) shows a schematic diagram of the target robot, which consists of two arms with grippers and three cameras. Note that our setting and foundation model are generic to any dual-arm gripper robot.

Figure: (a) Target dual-arm robot
Refer to caption: x2.png

We consider the concrete task of language-conditioned bimanual manipulation with vision, which is fundamental in robotics and has great value in real-world scenarios such as household (Stepputtis et al., ; Brohan et al., ; Zhao et al., ). Formally, given a language instruction $\ell$, the policy is presented with an observation ${\bm{o}}_{t}$ at time $t\in\mathbb{N}^{+}$; and then it produces an action ${\bm{a}}_{t}$ to control the *two robot arms* to achieve the goal specified by $\ell$. The observation is represented as a triple ${\bm{o}}_{t}:=({\bm{X}}_{t-T_{\text{img}}+1:t+1},{\bm{z}}_{t},c)$, where ${\bm{X}}_{t-T_{\text{img}}+1:t+1}:=({\bm{X}}_{t-T_{\text{img}}+1},\dots,{\bm{X
}}_{t})$ is the RGB observation history of size $T_{\text{img}}$, ${\bm{z}}_{t}$ is the low-dimensional proprioception of the robot, and $c$ is the control frequency. The action ${\bm{a}}_{t}$ is usually a subset of the desired proprioception ${\bm{z}}_{t+1}$(^1^11E.g., ${\bm{z}}_{t}$ may include the gripper position at time $t$, and ${\bm{a}}_{t}$ can be the target gripper position at step $t+1$.).

A specific task in bimanual manipulation typically consists of multiple elements: a *skill* (e.g., verbs like “pick” or “wipe”), an *object* (e.g., nouns like “bottle” or “table”), a *scene* (i.e. the environment in which the task takes place), and a *modality* describing how the skill is performed (e.g., adverbials like “pick the bottle with the left hand”). Given a new task, a practical policy is required to generalize to unseen elements that have not appeared in training data. This is challenging for previous rule-based methods as well as learning-based methods with small models/data, as discussed in Section 2.

We aim to train a foundation model policy via imitation learning to achieve generalizability. However, the available data for a specific dual-arm robot is particularly scarce ($<10$K trajectories) due to high hardware costs, far from the common requirement to train a foundation model. To address this, we propose to employ a pre-training and fine-tuning pipeline (Radford et al., ) to take advantage of data from multiple robots by drawing inspiration from recent advances in unimanual manipulation (Ghosh et al., ; Collaboration et al., ; Kim et al., ). In this manner, we would expand the data size by three orders of magnitude. Specifically, we first pre-train the model on a large-scale multi-robot dataset $\mathcal{D}_{\text{pre}}$ (mostly single-arm) and then fine-tune on a dataset of the target robot $\mathcal{D}_{\text{ft}}$. We denote the dataset by $\mathcal{D}_{\cdot}=\{(\ell^{(i)},{\bm{o}}^{(i)}_{t},{\bm{a}}^{(i)}_{t})\mid 0
\leq t<T^{(i)},1\leq i\leq N\}$, where $T^{(i)}$ is the length of the $i$-th trajectory and $N$ is the number of trajectories. Moreover, it is worth emphasizing that our goal is to use multi-robot data to enhance the model’s generalizability in bimanual manipulation *rather than* developing a cross-embodiment model for various robots. There are two main challenges to developing such a foundation model with multi-robot data:

Figure: Figure 3: RDT framework. Heterogeneous action spaces of various robots are embedded into a unified action space for multi-robot training. Inputs: proprioception ${\bm{z}}_{t}$, noisy action chunk $\tilde{{\bm{a}}}_{t:t+T_{a}}$, control frequency $c$, and diffusion time step $k$, acting as denoising inputs; image inputs ($T_{\text{img}}=2$ and ${\bm{X}}_{\cdot}=\{{\bm{X}}_{\cdot}^{1},{\bm{X}}_{\cdot}^{2},{\bm{X}}_{\cdot}^ {3}\}$ denotes a set of images from exterior, right-wrist, and left wrist cameras) and language inputs, acting as conditions. Outputs: denoised action chunk ${\bm{a}}_{t:t+T_{a}}$.
Refer to caption: x4.png

Challenge 1: How to design a powerful architecture?
A generalizable foundation model necessitates a powerful architecture. This requirement encompasses two primary aspects. Firstly, the architecture must possess sufficient *expressiveness* to capture the multi-modality in the action distribution. Fig. 2(b) illustrates a toy example where the robot attempts to grasp a cube. We can see that there are many modes to finish this task, in contrast to unimanual manipulation, where only one robot arm is controlled. When collecting demonstrations, the human operator may randomly pick one of them, leading to multi-modality in the collected action data. Secondly, *scalability* is necessary for such an architecture. As a foundation model, it should effectively process heterogeneous inputs from various modalities (text, images, actions, etc.) while being scalable to train stably on large datasets.

Challenge 2: How to train on heterogeneous data?
Training on multi-robot data presents a unique challenge of data heterogeneity. The physical structure and the action space can vary greatly across different robots. Previous attempts either restrict themselves to a subset of robots with similar action spaces (Yang et al., ; Ghosh et al., ; Kim et al., ) or only retain a subset of inputs sharing the same structure (Collaboration et al., ; Yang et al., ), at the cost of losing a lot of information. It remains largely under-addressed on how to train models on such heterogeneous data.

## 4 Robotics Diffusion Transformer

We now present Robotics Diffusion Transformer (RDT), as illustrated in Fig. 3. In Sec. 4.1, we present the diffusion model and the corresponding architecture to address Challenge 1. In Sec. 4.2, we resolve Challenge 2 by proposing a physically interpretable unified action space to unify various robot action spaces and enable multi-robot pre-training. We also collect a comprehensive multi-task bimanual dataset for fine-tuning to improve the bimanual manipulation capabilities of RDT.

### 4.1 RDT Model

#### Diffusion Modeling.

Due to multi-modality, given the language instruction $\ell$ and observation ${\bm{o}}_{t}$, there may be many possible actions ${\bm{a}}_{t}$ to proceed with the task. The policy will learn the “average” of action modes if we model it as a deterministic mapping $(\ell,{\bm{o}}_{t})\mapsto{\bm{a}}_{t}$ and regress the tuples of $(\ell,{\bm{o}}_{t},{\bm{a}}_{t})$ in the training data. This may result in out-of-distribution actions, such as the arithmetic mean of multiple modes, which can be completely infeasible (Pearce et al., ). Instead, we choose to model the continuous conditional distribution $p({\bm{a}}_{t}|\ell,{\bm{o}}_{t})$.
As discussed in Section 2, among various approaches,
diffusion models excel in both expressiveness and sampling quality, but can be slow to sample a high-dimensional data (e.g., images). Luckily, for our settings, the drawback is minor since that ${\bm{a}}_{t}$ has a much lower dimension than images, which requires only minimal sampling overhead. This has made diffusion models an ideal choice for policy as in Chi et al. ().

Nevertheless, employing diffusion models for robotic tasks faces unique challenges since the inherent properties of robotic physics quantities (i.e., the action and proprioception) are different from image/video data. Image and video data, while high-dimensional, often exhibit a degree of temporal and spatial continuity (Chen et al., ; Liang et al., ), with changes between frames typically being incremental. In contrast, robotic physics quantities are characterized by its nonlinear dynamics (de Wit et al., ) and the potential for high-frequency changes stemming from the physical interactions, such as collision, constraints, and material properties like damping. Moreover, the quantities also feature an unstable numerical range, probably due to extreme values caused by unreliable sensors. This underscores the necessity of adapting current diffusion models to effectively capture the instability and nonlinearity of robot data. Next, we will first elaborate on diffusion formulation and then introduce our design of architecture to resolve these challenges.

When making a decision with diffusion policies, we first sample a totally noisy action ${\bm{a}}_{t}^{K}\sim\mathcal{N}(\bm{0},{\bm{I}})$ and then perform $K\in\mathbb{N}^{+}$ denoising steps to denoise it to a clean action sample ${\bm{a}}_{t}^{0}$ from $p({\bm{a}}_{t}|\ell,{\bm{o}}_{t})$:

$$ ${\bm{a}}_{t}^{k-1}=\frac{\sqrt{\bar{\alpha}^{k-1}}\beta^{k}}{1-\bar{\alpha}^{k }}{\bm{a}}_{t}^{0}+\frac{\sqrt{\alpha^{k}}(1-\bar{\alpha}^{k-1})}{1-\bar{ \alpha}^{k}}{\bm{a}}_{t}^{k}+\sigma^{k}{\bm{z}},\quad k=K,\dots,1,$ (1) $$

where $\{\alpha^{k}\}_{k=1}^{K},\{\sigma^{k}\}_{k=1}^{K}$ are scalar coefficients pre-defined by a noise schedule (Nichol & Dhariwal, ). Here, $\beta^{k}:=1-\alpha^{k}$, and $\bar{\alpha}^{k-1}:=\prod_{i=1}^{k-1}\alpha^{i},{\bm{z}}\sim\mathcal{N}(\bm{0}
,{\bm{I}})$ if $k>1$, else $\bar{\alpha}^{k-1}=1,{\bm{z}}=\bm{0}$. However, ${\bm{a}}_{t}^{0}$ is intractable before sampling is finished. We opt to use a learnable denoising network $f_{\bm{\theta}}$ with parameters $\bm{\theta}$ to estimate the clean sample from a noisy one: ${\bm{a}}_{t}^{0}\leftarrow f_{\bm{\theta}}(\ell,{\bm{o}}_{t},{\bm{a}}_{t}^{k},k)$. To train such a network, we will minimize the following mean-squared error (MSE) of denoising:

$$ $\mathcal{L}(\bm{\theta}):=\mathrm{MSE}\left({\bm{a}}_{t},f_{\bm{\theta}}(\ell, {\bm{o}}_{t},\sqrt{\bar{\alpha}^{k}}{\bm{a}}_{t}+\sqrt{1-\bar{\alpha}^{k}}\bm{ \epsilon},k)\right),$ (2) $$

where $k\sim\mathrm{Uniform}(\{1,\dots,K\})$, $\bm{\epsilon}\sim\mathcal{N}(\bm{0},{\bm{I}})$, and $(\ell,{\bm{o}}_{t},{\bm{a}}_{t})$ is sampled from our training dataset. Later in this paper, we will denote noisy action inputs by $\tilde{{\bm{a}}}_{t}:=\sqrt{\bar{\alpha}^{k}}{\bm{a}}_{t}+\sqrt{1-\bar{\alpha}
^{k}}\bm{\epsilon}$, in which the superscript of $k$ is dropped for simplicity. Besides, in practice, we prefer to predict a sequence of actions, i.e., an action chunk, in one shot to encourage temporal consistency (Chi et al., ) and to alleviate error accumulation over time by reducing number of decisions in a task (Zhao et al., ). Specifically, we model $p({\bm{a}}_{t:t+T_{a}}|\ell,{\bm{o}}_{t})$, where ${\bm{a}}_{t:t+T_{a}}:=({\bm{a}}_{t},\dots,{\bm{a}}_{t+T_{a}-1})$ is an action chunk and $T_{a}$ denotes the chunk size (Zhao et al., ). We provide a detailed discussion in App. A.

We now present the design of the architecture, including the encoding of multi-modal inputs and the network structure of $f_{\bm{\theta}}$, while details are deferred to App. B.

#### Encoding of Heterogeneous Multi-Modal Inputs.

The heterogeneity of multi-modal inputs is reflected in the structure; that is, the format and number of dimensions of each modality are significantly different. This has posed challenges for mult-modal training. To address this, we encode these diverse modalities into a unified latent space. Below are the encoding methods:

- •
Low-Dimensional Inputs are low-dimensional vectors that represent physical quantities of the robot, including the proprioception, the action chunk, and the control frequency. To encode them, we use MLPs (with Fourier features (Tancik et al., )), which can effectively capture the high-frequency changes in low-dimensional spaces.
- •
Image Inputs are high-dimensional and contain rich spatial and semantic information. To extract compact representations, we use an image-text-aligned pre-trained vision encoder, SigLIP (Zhai et al., ). We fix its weights during training to save GPU memory.
- •
Language Inputs are of varying length and highly abstract, posing integration challenges due to their complexity and ambiguity. To encode them, we use a pre-trained Transformer-based language model, T5-XXL (Raffel et al., ). We also fix its weights during training to save GPU memory.

In addition to structure, heterogeneity features the different amounts of information contained in different inputs. First, data in different modalities contain different amounts of information. For example, images usually contain more information than text, and after encoding, they produce more tokens. Second, different inputs of the same modality may hold very different amounts of information. For example, the exterior camera of a robot has a more global view and contains richer information than the wrist cameras, as shown in the upper right of Fig. 3. In this case, the model may learn a shortcut: only focusing on the exterior view and ignoring the wrist views, thereby losing the ability to perceive depth. To tackle this issue, we randomly and independently mask each multi-modal input with a certain probability during encoding to prevent the model from over-relying on a specific input.

Figure: (a) Loss w/o QKN & RMSN
Refer to caption: x5.png

#### Network Structure of f 𝜽 subscript 𝑓 𝜽 f_{\bm{\theta}} italic_f start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT .

We choose Transformer as the scalable backbone network (Bao et al., ; Peebles & Xie, ) and make the following three key modifications from Diffusion Transfomer (DiT) by considering the characteristics of our robotic problem:

- •
QKNorm & RMSNorm. The unstable numerical range of the inputting robotic physical quantities can lead to problems such as gradient instability and numerical overflow, especially when training large foundation models. To solve this problem, we add QKNorm (Henry et al., ) to avoid numerical instability when calculating attention. Besides, we also note that our problem can be considered as a time series forecasting task, and the centering operation in the original DiTs’ LayerNorm could cause *token shift* and *attention shift*, thus destroying the symmetry of the time series (Huang et al., ). Therefore, we replace LayerNorm with RMSNorm (Zhang & Sennrich, ) without a centering operation. Fig. 4 shows that large-scale pre-training tends to be very unstable or even explode without this modification.
- •
MLP Decoder. To improve the approximation capability for nonlinear robot actions, we replace the final linear decoder with a nonlinear MLP decoder as a projection from the latent space back to the physical space. As empirically shown in Fig. 4, without this design, RDT cannot effectively capture nonlinear dynamics and thus loses the ability to accomplish dexterous tasks that require delicate operations.
- •
Alternating Condition Injection (ACI). In our model, image and language inputs serve as conditions, which are high-dimensional and variable in length, contrasting with the class label conditions in traditional DiTs (Xie et al., ). These informative conditions are challenging to compress into a single token, making the original adaptive layer norm approach unsuitable. Therefore, we employ cross-attention to accommodate conditions of varying lengths avoiding the information loss in further compression. Besides, we further analyze that, given that image tokens are usually much more than text tokens, simultaneous injection of both modalities tends to overshadow text-related information, thus impairing the capability of the instruction following (see Fig. 4 for quantitative results). To mitigate this issue, we strategically alternate between injecting image and text tokens in successive layers’ cross-attention rather than injecting both in every layer.

### 4.2 Data

#### Training on Heterogeneous Multi-Robot Data.

To enable training on heterogeneous multi-robot data, we need a unified action space shared among various robots, which can provide a unified format for multi-robot actions. The mapping from the original action space of a robot to the unified action space should be physically interpretable, each of whose dimensions should have a clear physical meaning. This can encourage the model to learn shared physical laws from different robot data, thereby improving the efficiency of learning from data of different robots (Shah et al., ).

The design of the space consists of two steps. Firstly, for each robot, we can use a single space to accommodate both its proprioception ${\bm{z}}_{t}$ and action ${\bm{a}}_{t}$. This is because ${\bm{a}}_{t}$ is usually a subset of the desired ${\bm{z}}_{t+1}$ (de Wit et al., ; Kouvaritakis & Cannon, ), and thus the space of ${\bm{z}}_{t}$ naturally contains the space of ${\bm{a}}_{t}$. Secondly, we design a unified space that encompasses all the main physical quantities of most robots with gripper arms. As illustrated in the left side of Fig. 3, we embed the action space of a robot into this unified space by filling each element of the original action vector into the corresponding position of the unified action space vector according to its physical meaning, with the remaining positions being padded. The specific definition of the space is given in App. C.

Under this unified space, we are able to pre-train RDT on data from almost all modern robots with gripper arms and greatly expand the data scale toward the requirement for a foundation model. Specifically, our collection of pre-training datasets includes $46$ datasets of various robots, with a total size of $1$M+ trajectories and $21$TB. More details and preprocessing are deferred to App. D.

#### Collecting a Comprehensive Multi-Task Bimanual Dataset.

Though having been pre-trained on large-scale datasets, RDT could still need help to zero-shot generalize to the target dual-arm robot due to the embodiment gap. To bridge the gap, we need to collect a multi-task bimanual dataset on the target robot for fine-tuning. Recent advances in large language models (Ziegler et al., ; Brown et al., ; Touvron et al., ) have shown that high-quality fine-tuning datasets are crucial for model performance. We ensure the high quality of our dataset from three aspects: (1) Regarding quantity, we have collected $6$K+ trajectories, making our dataset one of the largest bimanual datasets nowadays; (2) Regarding comprehensiveness, we consider $300$+ challenging tasks, covering most manipulation task types, from pick-and-place to plugging cables, even including writing math equations; (3) Regarding diversity, we prepare $100$+ objects with rigid and non-rigid bodies of various sizes and textures and $15$+ different rooms with different lighting conditions. Besides, we further utilize GPT-4-Turbo (Achiam et al., ) to rewrite human-annotated instructions to increase text diversity. For more information, we refer to Fig. 6 and App. E.

## 5 Experiments

We aim to answer the following questions through real-robot experiments: $\mathcal{Q}$1: Can RDT zero-shot generalize to unseen objects and scenes? $\mathcal{Q}$2: How effective is RDT’s zero-shot instruction-following capability for unseen modalities? $\mathcal{Q}$3: Can RDT facilitate few-shot learning for previously unseen skills? $\mathcal{Q}$4: Is RDT capable of completing tasks that require delicate operations? and $\mathcal{Q}$5: Are large model sizes, extensive data, and diffusion modeling helpful for RDT’s performance?

### 5.1 Experiment Setups

Figure: Figure 5: Task definitions and visualizations. For $7$ challenging tasks, we describe their language instruction, randomization, and definitions of each sub-task. For Pour Water-L-1/3 and Pour Water-R-2/3, we show the resulting water levels in two images.
Refer to caption: x7.png

**Table 1: Dimensions when designing tasks. For Pour Water-L-1/3 and Pour Water-R-2/3, only the water levels of little, half (i.e., 1/2), and full are seen in training instructions. For Handover and Fold Shorts, the dataset only contains $5$ demos and $1$ demo of the skill, respectively. For Robot Dog, it requires delicate operations, as a slight angle when pushing joysticks can make the robot dog deviate.**
| TASK NAME | DIMENSION | EXPLANATION |
| --- | --- | --- |
| Wash Cup | Unseen Object ($\mathcal{Q}$1) | To wash one seen and two unseen cups with the faucet |
| Pour Water | Unseen Scene ($\mathcal{Q}$1) | To pour water into the cup in three unseen rooms |
| Pour Water-L-1/3 | Instruction Following ($\mathcal{Q}$2) | To pour water into the cup with the left hand until one-third full |
| Pour Water-R-2/3 | Instruction Following ($\mathcal{Q}$2) | To pour water into the cup with the right hand until two-thirds full |
| Handover | 5-Shot Learning ($\mathcal{Q}$3) | To move the marker to the box, where handover is needed due to far distance |
| Fold Shorts | 1-Shot Learning ($\mathcal{Q}$3) | To fold the shorts in half horizontally |
| Robot Dog | Dexterity ($\mathcal{Q}$4) | To push the joystick straight to control the robot dog to walk in a straight line |

#### Tasks.

We select $7$ challenging tasks to evaluate the generalizability and capabilities of RDT from different dimensions, including complex scenarios that the model may encounter in real-world tasks, such as various unseen elements and dexterous manipulation. An illustration of the dimension of each task is given in Table 1 while detailed definitions and visualizations are provided in Fig. 5.

#### Data.

We use the pre-training and fine-tuning datasets in Sec. 4.2. We now list the number of demos related to each task in our fine-tuning dataset. Wash Cup: $133$ demos for seen cups combined and 0 0 demos for unseen cups; Pour Water: $350$ demos for seen rooms combined and 0 0 demos for unseen rooms; Pour Water-L-1/3 & Pour Water-R-2/3: $18$ demos for the water level of little, $19$ demos for half, and $19$ demos for full; Handover: $5$ demos; Fold Shorts: $1$ demo; Robot Dog: $68$ demos.

Model Training and Inference.
We scale the size of RDT up to $1.2$B parameters, establishing it as the currently largest diffusion-based robotic foundation model. The model is pre-trained on $48$ H100 80GB GPUs for a month, giving a total of $1$M training iteration steps. It takes three days to fine-tune this model using the same GPUs for $130$K steps. We defer further details to App. F, including the running platform, design choices, and data augmentation techniques. For real-time inference, we adopt DPM-Solver++ (Lu et al., ), a recent sampling accelerator of diffusion models. It can reduce the diffusion steps required to sample an action chunk from $100$ steps to $5$ steps, achieving an action chunk inference frequency of $6$ Hz (action chunks per second) and an average action inference frequency of $381$ Hz (actions per second) on the target robot’s onboard RTX 4090 24GB GPU.

Baselines.
To comprehensively evaluate RDT, we consider the most advanced baselines in robotic foundation models and bimanual manipulation, including Action Chunking with Transformers (ACT) (Zhao et al., ), OpenVLA (Kim et al., ), and Octo (Ghosh et al., ). ACT is a state-of-the-art method in bimanual manipulation, which uses VAE to model the action distribution. OpenVLA is the largest open-source foundation model (7B), employing the discretization modeling. Octo is a diffusion-based foundation model, and its largest version has only $93$M parameters.

#### Metric and Hardware.

We employ the success rate as our main metric, which is calculated by dividing successful trials by total trials. Wash Cup is tested with $8$ trials for each cup (one seen cup, two unseen cups, $24$ trials in total). Pour Water is tested with $8$ trials for each room (three unseen rooms, $24$ trials in total). Pour Water-L-1/3 and Pour Water-R-2/3 are tested with $8$ trials each. Handover, Fold Shorts, and Robot Dog are tested with $25$ trials each. All the tests are performed on the ALOHA dual-arm robot (see App. G for hardware configurations). Experimental details, such as the implementation and hyper-parameters, are elaborated in App. H.

**Table 2: Ablation study results. Here are the success rates ($\%$) of the original RDT and its three variants in tasks of Wash Cup (unseen cup 2, total success rate), Pour Water (unseen room 3, total success rate), and Pour Water-L-1/3 (correct amount sub-task). All the models except RDT (scratch) are pre-trained before fine-tuning.**
| VARIANT<br>NAME | UNSEEN<br>OBJECT | UNSEEN<br>SCENE | INSTRUCTION<br>FOLLOWING |
| --- | --- | --- | --- |
| RDT (regress) | 12.5 | 50 | 12.5 |
| RDT (small) | 37.5 | 62.5 | 25 |
| RDT (scratch) | 0 | 25 | 62.5 |
| RDT (ours) | 50 | 62.5 | 100 |

#### Ablation Study.

Answering $\mathcal{Q}$5, we have conducted ablation studies on the model size, pre-training, and the modeling method to understand their importance. We consider the variants of: RDT (ours): the original RDT. RDT (regress): RDT without diffusion modeling. It models the deterministic mapping $(\ell,{\bm{o}}_{t})\mapsto{\bm{a}}_{t}$. RDT (small): RDT without large parameters. It has only $166$M parameters. RDT (scratch): RDT without pre-training. It is trained from scratch during fine-tuning. In Table 2, we evaluate these variants in terms of three dimensions of generalizability. Table 7 provides a comparison of different variants of RDT as well as baselines.

**Table 3: Quantitative results. We report success rates ($\%$) of ACT, OpenVLA, RDT (from scratch, no pre-trained), and RDT (ours, pre-trained) for $7$ tasks. Sub-columns in each sub-task cell represent different elements (objects, instructions, scenes). ACT is not language-conditioned and thus unavailable for instruction following. RDT (ours) consistently outperforms others.**
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Wash Cup: seen cup 1 $|$ unseen cup 1 $|$ unseen cup 2 (Unseen Object) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Pick Up Cup | Turn On Faucet | Get Water | Pour Out Water | Place Back Cup | Total |  |  |  |  |  |  |  |  |  |  |  |  |
| ACT | 50 | 12.5 | 37.5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 37.5 | 0 | 0 | 0 | 0 | 0 |
| OpenVLA | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Octo | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| RDT (scratch) | 37.5 | 12.5 | 0 | 0 | 12.5 | 12.5 | 0 | 0 | 0 | 37.5 | 12.5 | 0 | 25 | 0 | 0 | 0 | 0 | 0 |
| RDT (ours) | 87.5 | 87.5 | 50 | 62.5 | 75 | 50 | 50 | 75 | 50 | 87.5 | 75 | 50 | 87.5 | 62.5 | 50 | 50 | 75 | 50 |
|  | Pour Water-L-1/3 $|$ Pour Water-R-2/3 (Instruction Following) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Pick Up Bottle | Pour Water | Place Back Bottle | Total | Correct Hand | Correct Amount |  |  |  |  |  |  |  |  |  |  |  |  |
| OpenVLA | 50 |  | 0 | 0 |  | 0 | 0 |  | 0 | 0 |  | 0 | 50 |  | 0 | 0 |  | 0 |
| Octo | 0 |  | 0 | 0 |  | 0 | 0 |  | 0 | 0 |  | 0 | 0 |  | 0 | 0 |  | 0 |
| RDT (scratch) | 100 |  | 75 | 75 |  | 25 | 62.5 |  | 25 | 62.5 |  | 25 | 100 |  | 75 | 62.5 |  | 12.5 |
| RDT (ours) | 100 |  | 87.5 | 100 |  | 87.5 | 100 |  | 87.5 | 100 |  | 87.5 | 100 |  | 87.5 | 100 |  | 75 |
|  | Pour Water: unseen room 1 $|$ unseen room 2 $|$ unseen room 3 (Unseen Scene) |  |  |  | Fold Shorts (1-Shot) |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Pick Up Bottle | Pour Water | Place Back Bottle | Total | - | Total |  |  |  |  |  |  |  |  |  |  |  |  |
| ACT | 25 | 87.5 | 25 | 0 | 50 | 12.5 | 0 | 37.5 | 12.5 | 0 | 37.5 | 12.5 |  | - |  | 0 |  |  |
| OpenVLA | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |  | - |  | 0 |  |  |
| Octo | 50 | 0 | 12.5 | 12.5 | 0 | 0 | 12.5 | 0 | 0 | 12.5 | 0 | 0 |  | - |  | 4 |  |  |
| RDT (scratch) | 62.5 | 100 | 62.5 | 25 | 87.5 | 37.5 | 25 | 75 | 25 | 25 | 75 | 25 |  | - |  | 40 |  |  |
| RDT (ours) | 62.5 | 100 | 62.5 | 62.5 | 100 | 62.5 | 62.5 | 100 | 62.5 | 62.5 | 100 | 62.5 |  | - |  | 68 |  |  |
|  | Handover (5-Shot) | Robot Dog (Dexterity) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Pick Up<br>Pen | Switch<br>Hand | Drop<br>Pen | Fall into<br>Box | Total | Grab<br>Remote | Push<br>Joystick | Total | Walk<br>Straight |  |  |  |  |  |  |  |  |  |
| ACT | 44 | 0 | 0 | 0 | 0 | 88 | 32 | 32 | 32 |  |  |  |  |  |  |  |  |  |
| OpenVLA | 0 | 0 | 0 | 0 | 0 | 84 | 0 | 0 | 0 |  |  |  |  |  |  |  |  |  |
| Octo | 12 | 0 | 0 | 0 | 0 | 100 | 4 | 4 | 0 |  |  |  |  |  |  |  |  |  |
| RDT (scratch) | 88 | 32 | 24 | 16 | 16 | 100 | 64 | 64 | 32 |  |  |  |  |  |  |  |  |  |
| RDT (ours) | 100 | 56 | 56 | 40 | 40 | 100 | 76 | 76 | 48 |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

### 5.2 Results Analysis

From the results in Table 3, we can see that RDT consistently outperforms other baselines. This is because RDT employs diffusion with a powerful network architecture to model the distribution of multi-modal actions accurately, while discretization and VAE lack accuracy and expressiveness, respectively. Besides, the large number of parameters after large-scale pre-training provides a lot of prior knowledge, which significantly improves the generalizability. Here is a detailed analysis:

- •
$\mathcal{Q}$1 & $\mathcal{Q}$2: RDT can zero-shot generalize to unseen objects, scenes, and modalities. In Wash Cup and Pour Water, RDT can still achieve a high success rate on unseen scenarios, and its performance is not much different from that on seen ones. In contrast, the other baselines cannot even complete the entire task. In Pour Water-L-1/3 and Pour Water-R-2/3, from the third row of Fig. 5 or Fig. 10 (zoomed-in version), we can find that RDT understands precisely which hand to manipulate and how much water to pour and closely follows the instruction through its actions, even though it has never seen words like “one-third” or “two-thirds”. It is precisely because of large-scale pre-training that RDT has seen a large number of diverse objects, scenes, and instructions, leading to such strong zero-shot generalization.
- •
$\mathcal{Q}$3: RDT can learn new skills using only a few shots. In Handover and Fold Shorts, RDT has learned new and complex skills of handover and folding through few-shot learning, whose action patterns are very different from known skills, while the success rate of others is almost zero. Such improvement is also due to large-scale pre-training. Few-shot learning can help RDT quickly adapt to new working environments, which is of great significance for practical applications.
- •
$\mathcal{Q}$4: RDT can handle dexterous tasks. In Robot Dog, RDT accurately controls the angle when pushing the joystick, while others have caused the robot dog to deviate. This is because diffusion, with our powerful network architecture, can model the distribution of multi-modal and nonlinear actions so that the action precision can meet the requirements of dexterous tasks. We also note that the joystick and the remote control are both black, making the joystick not visually apparent. It probably makes ACT prone to failure. In contrast, large-scale pre-training has made RDT learn a better vision-language representation of the joystick concept, improving the recognition capability.
- •
$\mathcal{Q}$5: Large model size, extensive data, and diffusion are all essential factors for our excellence. In Table 2, there is a serious performance drop without any of these factors, demonstrating the necessity of our contributions. In particular, RDT (scratch) performs poorly on unseen objects and scenes, indicating that the knowledge from pre-training is critical for generalization.

## 6 Conclusion

In this paper, we tackled the challenges of data scarcity and increased manipulation complexity in generalizable bimanual manipulation by developing the Robotics Diffusion Transformer (RDT), a diffusion-based foundation model for language-conditioned visuomotor imitation learning. Our model was pre-trained on an extensive multi-robot dataset and fine-tuned on a self-collected bimanual dataset. We further introduce a Physically Interpretable Unified Action Space to unify action representations across different robots, enhancing robustness and transferability. Outperforming existing methods, RDT not only demonstrates significant improvements in dexterous bimanual capability and instruction following but also achieves remarkable performance in few-shot learning and zero-shot generalization to unseen objects and scenes.

### Ethics Statement

All the data used in this research comes from open-source and well-documented datasets, and we strictly follow all applicable licensing and usage guidelines. Our finetuning dataset is collected by the authors of this paper along with some volunteers.

While RDT is a model trained for scalable, language-conditioned visuomotor policy learning and tested on the ALOHA dual-arm robot, we emphasize that any harmful use of our model is neither intended nor encouraged, and we encourage responsible deployment on real-world robots.

### Reproducibility Statement

To reproduce our pre-training and fine-tuning processes, we have provided the code in the supplementary materials. We also include instructions for downloading the dataset, how to use the training code, and a guide for deploying on a real machine in the README file. Once the paper is accepted, we will fully open-source all our code, model weights, and fine-tuning datasets.

Please refer to App. D for pre-training dataset details, App. E for fine-tuning dataset details, App. F for RDT training details, App. G for hardware details, and App. H for experimental details and implementation of baselines.

## Appendix A Action Chunking Technique

In practice, we find that the errors in action prediction accumulate as the number of historical decisions increases due to the imperfection of the learned policy. This may cause the robot to drift out of the training distribution, reaching hard-to-recover states (Ross et al., ). To alleviate this, we prefer to predict multiple actions in one shot, thereby reducing the total number of decisions in a trajectory. In this way, we model $p({\bm{a}}_{t:t+T_{a}}|\ell,{\bm{o}}_{t})$, where ${\bm{a}}_{t:t+T_{a}}:=({\bm{a}}_{t},\dots,{\bm{a}}_{t+T_{a}-1})$ is an action chunk and $T_{a}$ denotes the chunk size (Zhao et al., ). To adapt Eq. 1 and Eq. 2 to this context, we could simply replace ${\bm{a}}_{t}$ by ${\bm{a}}_{t:t+T_{a}}$. Besides, according to Chi et al. (), action chunking is also helpful for improving temporal consistency. It can better consider the coherence of previous and subsequent actions when making decisions and may avoid sudden changes in actions that may cause damage to the robot.

## Appendix B Architecture Details

#### Encoding of Multi-Modal Inputs.

Encoding details are outlined below:

- •
Low-Dimensional Inputs. The proprioception ${\bm{z}}_{t}$ and the noisy action chunk $\tilde{{\bm{a}}}_{t:t+T_{a}}$ are first embedded into the unified action space. This space is used to unify the representation of ${\bm{z}}_{t}$ and $\tilde{{\bm{a}}}_{t:t+T_{a}}$ across various robots, which is elaborated in Sec. 4.2. Then, they are encoded into the token space by a shared MLP since they have similar physical meanings. Such continuous encoding can avoid precision loss in contrast to discretized encoding (Brohan et al., ; ; Kim et al., ). For frequency $c$ as well as the diffusion time step $k$, we encode them into the token space through two MLPs, respectively. Afterward, all of them are concatenated together in the length direction to achieve in-context conditioning (Peebles & Xie, ; Bao et al., ), resulting in an input token sequence of length $1+T_{a}+1+1$. Finally, position embeddings are added to distinguish different modalities and to inject temporal information in $\tilde{{\bm{a}}}_{t:t+T_{a}}$.
- •
Image Inputs. We encode the RGB images by a frozen SigLIP (Zhai et al., ) and utilize an additional MLP to project the output to the token space. To enhance the model’s ability to distinguish images based on viewpoint and time steps, we extend traditional sinusoidal positional embeddings to multi-dimensional grids, as shown on the right side of Fig. 3. This modification integrates spatial-temporal information, enabling the model to capture the relationships between input images. Specifically, we adopt the implementation by Liu et al. (), employing grid dimensions of $(T_{\text{img}},N_{\text{cam}},N_{\text{patch}},D)$. Here, $N_{\text{cam}}$ represents the number of cameras, set to three in our configuration, and $N_{\text{patch}}$ indicates the number of patches into which each image is divided by the ViT-based Image Encoder and $D$ denotes the embedding dimension.
- •
Language Inputs. Language instruction is encoded by a frozen T5-XXL (Raffel et al., ), and an MLP is used to project the output to the token space. When calculating attention for language tokens, we apply the language attention mask to mask out the pad tokens appended during batching.

During training, each input from various modalities is independently masked with a probability of $10\%$.

#### Network Structure of f 𝜽 subscript 𝑓 𝜽 f_{\bm{\theta}} italic_f start_POSTSUBSCRIPT bold_italic_θ end_POSTSUBSCRIPT .

After encoding, we feed the tokens of the low-dimensional inputs into the main network, which is adjusted from Diffusion Transformers (DiTs) with Cross-Attention (Peebles & Xie, ) due to their high scalability. For better training stability, we add QKNorm (Henry et al., ) into each attention layer and replace each LayerNorm with RMSNorm (Zhang & Sennrich, ). In each DiT block’s cross-attention layer, we alternately inject language and image tokens rather than simultaneously inject both, avoiding the issue of token imbalance between the two modalities. After $L$ DiT blocks, we normalize the output and project it back to the action space via an MLP decoder.

## Appendix C Physically Interpretable Unified Action Space

As mentioned in Sec. 4.2, we embed the actions of various robots into one unified space that includes all the main physical quantities of robots. This unified action space has a dimensionality of 128. Table 4 describes each element of the vector in this unified action space. For a specific robot, each element of the raw action vector is filled into the corresponding position of the unified action vector according to its physical meanings, with the remaining positions being padded.

**Table 4: Description of the unified action space vector. For single-arm robot cases, its arm is mapped to the “right” arm. For a robot arm with only 6 DoF, its joint positions will be filled in the first $6$ of the $10$ corresponding positions. The same is true for other physical quantities.**
| Index Range | Element Index | Mapped Physical Quantity |
| --- | --- | --- |
| [0, 10) | 0–9 | Right arm joint positions |
| [10, 15) | 10–14 | Right gripper joint positions |
| [15, 25) | 15–24 | Right arm joint velocities |
| [25, 30) | 25–29 | Right gripper joint velocities |
| [30, 33) | 30–32 | Right end effector positions |
| [33, 39) | 33–38 | Right end effector 6D pose |
| [39, 42) | 39–41 | Right end effector velocities |
| [42, 45) | 42–44 | Right end effector angular velocities |
| [45, 50) | 45–49 | Reserved |
| [50, 60) | 50–59 | Left arm joint positions |
| [60, 65) | 60–64 | Left gripper joint positions |
| [65, 75) | 65–74 | Left arm joint velocities |
| [75, 80) | 75–79 | Left gripper joint velocities |
| [80, 83) | 80–82 | Left end effector positions |
| [83, 89) | 83–88 | Left end effector 6D pose |
| [89, 92) | 89–91 | Left end effector velocities |
| [92, 95) | 92–94 | Left end effector angular velocities |
| [95, 100) | 95–99 | Reserved |
| [100, 102) | 100–101 | Base linear velocities |
| [102, 103) | 102 | Base angular velocities |
| [103, 128) | 103–127 | Reserved |

## Appendix D Pre-Training Datasets

Our pre-training dataset collection includes $46$ datasets, with a total scale of $1$M+ trajectories and $21$TB, making it the largest pre-training collection of robotics datasets to date. Table 5 presents the complete list of our pre-training datasets and their sampling weights. We assign an initial weight of $\sqrt{N_{j}}$ to each dataset with size $N_{j}$ and adjust it according to the diversity and quality of each dataset. Compared to linear weighting, this approach prevents excessive sampling of large datasets while ensuring smaller datasets are adequately sampled, thus enhancing the diversity of pre-training samples in each mini-batch. During the pre-training stage, we further observed and adjusted the weights of different datasets based on their intermediate loss results. We increased the weights of those slow-convergent datasets.

#### Main Datasets.

We list some main datasets as follows:

- •
RT-1 Dataset (Brohan et al., ) is a large diversve dataset including $130$K trajectories with multiple tasks, objects and environments. It is collected across $13$ different embodiments, each equipping a single exterior RGB camera. The action space includes the 6D end effector (EEF), gripper open, and base displacement with a control frequency of $3$Hz.
- •
DROID (Khazatsky et al., ) is a large-scale multi-task dataset with $76$K trajectories and $564$ scenes. It is collected via teleoperating a Franka Panda $7$-DoF Robot Arm, with both wrist and exterior RGB-D cameras. The action space includes $7$-DoF joint positions and a gripper width, while the proprioception additionally includes the 6D EEF with a control frequency of $15$Hz.
- •
RH20T (Fang et al., ) is a comprehensive dataset covering 110K trajectories and 140 tasks. It includes four different robotic embodiments and three different camera views, sampled at a frequency of 10Hz. It also includes both long and short tasks. Its state space is a mix of 6-DoF and 7-DoF joint positions, and it features a third-person perspective RGB-D camera.
- •
Mobile ALOHA Dataset (Fu et al., ) is a bimanual dataset containing 1K+ trajectories collected by the Mobile ALOHA robot. Its state space includes base movements and 14-dimensional joint positions of both hands, along with three or four first-person perspective cameras. Some of its data includes wide-ranging perspective changes and base movements, which were originally suitable for imitation learning.
- •
Other Datasets. The other data come from RH20T (Fang et al., ), RoboSet (Kumar et al., ), BridgeData V2 (Walke et al., ), and Open X-Embodiment (Collaboration et al., ). Most of them feature different robotic morphology and camera observation, enhancing both heterogeneity and variety of our pretraining datasets.

#### Data Cleaning.

Repetitive episodes and episodes of failure are excluded to ensure the quality of the pre-training datasets. We remove blank images, exclude erroneously recorded velocities, and filter out overly short trajectories. Overlength trajectories will be downsampled to avoid unfairness.

#### Preprocessing of Multi-Modal Observation/Action Inputs.

We describe the preprocessing details of each modality:

- •
Language Instruction $\ell$. We perform a simple cleaning on the raw text, such as removing illegal characters and extra spaces, capitalizing the beginning of sentences, and adding a period at the end of sentences. We leave the text variable-length.
- •
RGB Images ${\bm{X}}_{t-T_{\text{img}}+1:t+1}$. We employ a fixed-length image input strategy. We fix the image input order and format for all robots, with a total of three views: a static exterior view, a right-wrist view, and a left-wrist view, deemed sufficient for the requirements of most bimanual tasks. We treat a single-arm robot’s wrist camera as the right-wrist one and pad the unavailable views with the background color. When fed into the model, each image is padded into a square and resized to $384\times 384$, keeping its origin aspect ratio. Besides, we choose $T_{\text{img}}=2$ since a history length of two is adequate for most situations, striking a balance between efficiency and performance (Ghosh et al., ; Wu et al., ). Finally, we can write the image inputs as ${\bm{X}}_{t-1:t+1}:=(\{{\bm{X}}^{1}_{t-1},{\bm{X}}^{2}_{t-1},{\bm{X}}^{3}_{t-1
}\},\{{\bm{X}}^{1}_{t},{\bm{X}}^{2}_{t},{\bm{X}}^{3}_{t}\})$.
- •
Proprioception ${\bm{z}}_{t}$ and Action Chunk ${\bm{a}}_{t:t+T_{a}}$. We roughly align the scales of various datasets by unifying the units of physical quantities (m, rad, m/s, rad/s, etc) rather than strictly normalizing to $[-1,1]$ or $\mathcal{N}(0,1)$ as in prior work (Chi et al., ; Ghosh et al., ). For example, “$1$ (m)” in different datasets corresponds to the same real-world length. Rescaling the physical quantities will destroy such shared properties and thus impair the model’s ability to transfer across robots. We also employ the 6D representation (Zhou et al., ) for the EEF rotation to overcome the gimbal lock issue.
Before choosing $T_{a}=64$, we have referred to the previous ablation studies by Zhao et al. () and balanced between the performance and computational overhead. Besides, historical proprioceptions ${\bm{z}}_{i},i<t$ are excluded to prevent the model from learning shortcuts using the low-dimensional inputs only and thus sticking to fixed motion patterns. Instead, we encourage the model to learn generalizable decision-making structures from high-dimensional image features.
- •
Control Frequency $c$. In addressing the challenge posed by differing control frequencies across datasets, we feed the control frequency into the model, allowing the model to take this variation into account when making decisions.

**Table 5: The pre-training datasets and their corresponding weights.**
| Pre-Training Dataset | Sample Percentage (%) |
| --- | --- |
| RT-1 Dataset (Brohan et al., ) | 9.00 |
| TACO Dataset (Rosete-Beas et al., ) | 1.99 |
| JACO Play Dataset (Dass et al., ) | 1.10 |
| Cable Routing Dataset (Luo et al., ) | 0.27 |
| NYU Door Opening (Pari et al., ) | 0.33 |
| Viola (Zhu et al., ) | 0.40 |
| Berkeley UR5 () | 1.06 |
| TOTO (Zhou et al., ) | 1.06 |
| Kuka (Kalashnikov et al., ) | 1.66 |
| Language Table (Lynch et al., ) | 3.32 |
| Columbia Cairlab Pusht Real (Chi et al., ) | 0.40 |
| Stanford Kuka Multimodal Dataset (Lee et al., ) | 1.83 |
| Stanford Hydra Dataset  (Belkhale et al., ) | 0.80 |
| Austin Buds Dataset (Zhu et al., ) | 0.23 |
| Maniskill Dataset (Gu et al., ) | 5.78 |
| Furniture Bench Dataset (Heo et al., ) | 2.36 |
| UCSD Kitchen Dataset (Yan et al., ) | 0.40 |
| UCSD Pick And Place Dataset (Feng et al., ) | 1.23 |
| Austin Sailor Dataset (Nasiriany et al., ) | 0.50 |
| Austin Sirius Dataset (Liu et al., ) | 0.80 |
| BC Z (Jang et al., ) | 6.91 |
| UTokyo PR2 Opening Fridge (Oh et al., ) | 0.30 |
| UTokyo PR2 Tabletop Manipulation (Oh et al., ) | 0.50 |
| UTokyo Xarm Pick And Place (Matsushima et al., ) | 0.33 |
| UTokyo Xarm Bimanual (Matsushima et al., ) | 0.03 |
| Berkeley MVP (Radosavovic et al., ) | 0.73 |
| Berkeley RPT (Radosavovic et al., ) | 1.00 |
| KAIST Nonprehensile (Kim et al., ) | 0.46 |
| Tokyo U LSMO (Osa, ) | 0.23 |
| DLR Sara Grid Clamp (Padalkar et al., ) | 0.03 |
| Robocook (Shi et al., ) | 1.66 |
| Imperialcollege Sawyer Wrist Cam () | 0.43 |
| Iamlab CMU Pickup Insert (Saxena et al., ) | 0.83 |
| UTAustin Mutex (Shah et al., ) | 1.29 |
| Fanuc Manipulation (Zhu et al., ) | 0.66 |
| Play Fusion (Chen et al., ) | 0.80 |
| DROID (Khazatsky et al., ) | 10.06 |
| FMB (Luo et al., ) | 1.39 |
| Dobb·E (Shafiullah et al., ) | 1.20 |
| QUT Dexterous Manipulation (Federico Ceola, ) | 0.46 |
| Aloha Dataset (Zhao et al., ) | 4.98 |
| Mobile Aloha Dataset (Fu et al., ) | 4.98 |
| RoboSet (Kumar et al., ) | 4.48 |
| RH20T (Fang et al., ) | 10.99 |
| Calvin Dataset (Mees et al., ) | 3.32 |
| BridgeData V2 (Walke et al., ) | 7.44 |

## Appendix E Fine-Tuning Dataset

Our fine-tuning dataset is created using Mobile ALOHA robot (Fu et al., ), including $300$+ tasks, $6$K+ trajectories, and $3$M+ frames. It is also one of the largest open-source multi-task bimanual robot datasets to date. Fig. 6 gives a summary of this dataset. We have borrowed $3$ tasks ($140$ episodes in total) from the open-source [Songling dataset](https://openi.pcl.ac.cn/ARIO/Songling_datasets/datasets) (Wang et al., ).

- •
Multi-Modal Features. We collect the dataset with three RGB cameras positioned at the front and on the left and right grippers. We record dual-arm 6-DoF joint positions and velocities, along with the gripper angles. We manually annotated instructions for each task. To further augment our instructions and align them with the pre-training datasets, we utilize GPT-4-Turbo (Achiam et al., ) to generate $100$ expanded instructions and one simplified instruction for each task. This multi-modal information further enhances the richness and quality of our dataset.
- •
Diverse Objects and Scenes. Our dataset includes diverse tasks and scenes, encompassing more than 300 tasks, including skills such as picking up, inserting, writing, pushing, and pulling. It features 100+ objects with rigid and non-rigid bodies of various sizes and textures. We collect the dataset in 15+ scenes and introduce randomness during data collection for each task, such as varying the initial positions of objects and robots. To further increase diversity, we added random lighting conditions. For instance, pouring water was performed under both normal lighting and changing color conditions. These measures further enhance the diversity of our dataset.
- •
Challenging Tasks. Various challenging tasks are also considered, encompassing dexterous manipulations, such as unscrewing the cap from a plastic bottle, and comprehension tasks, such as spelling “love” with letter blocks. Furthermore, the dataset includes tasks that integrate both dexterity and comprehension, such as solving mathematical equations on the whiteboard. Additionally, our dataset incorporates bimanual tasks, such as inserting the charging cable into the phone. These complex, high-quality tasks further enhance the model’s downstream comprehensibility and generalizability.

Figure: Figure 6: Fine-Tuning dataset. Our dataset includes the following key features: (1) Diverse Objects and Scenes. Our dataset contains objects with different properties manipulated in different scenes and conditions. (2) Challenging Tasks. Our dataset incorporates dexterous manipulation, language and vision comprehension, and bimanual tasks. (3) Multi-Modal Features. Our dataset is annotated with rich multi-modal data, including 3-View RGB cameras, joint information, and augmented instructions.
Refer to caption: x8.png

## Appendix F RDT Training Details

#### Platform.

We use Pytorch (Paszke et al., ) and DeepSpeed (Rasley et al., ) to facilitate parallel training and employ a producer-consumer framework with TensorFlow Dataset () for fast data loading. Since most of the datasets in the Open X-Embodiment (Collaboration et al., ) are stored in the form of TFRecord, we convert all pre-training datasets into TFRecord for storage. In pre-training, we use the producer process to decompress the data from TFRecord and store it in a buffer on the hard disk. At the same time, we use the consumer process to read data from the buffer in a disorderly order and feed it to the model training. This not only decouples the TensorFlow (Abadi et al., ) and PyTorch environments but also alleviates the training performance loss caused by the small size of the shuffling buffer in the memory. In the fine-tuning stage, since the dataset is relatively small, we additionally implement a data reading pipeline using the HDF5 dataset for storage.

#### Padding Action and Proprioception.

To embed a specific robot action into the $128$-dimensional unified action space, we need to pad unavailable action elements. The usual practice is to pad with a 0 value or a specific value. But “0” actually has a physical meaning. For example, a speed of “0” generally represents stillness relative to the ground. This may confuse the model: Does “0” represent stillness or a filler value? To solve this problem, we concatenate the action and proprioception with a 0 0-$1$ vector indicating whether each dimension is padded before encoding them into the token space, resulting in a $256$-dimensional vector. This can supplement the missing availability information and eliminate confusion.

#### Inspecting Training Process.

During training, for every fixed period, we conduct a diffusion sampling and compare the sampled actions with the ground truth of the training dataset. Empirically, we discover a general positive correlation between the Mean Squared Error (MSE) of the two and the performance of deployment on the robot. This observation allows us to monitor the model’s training progress easily. When this MSE converges, we can generally stop training. We note that an overly low MSE may also mean overfitting.

#### Data Augmentation.

Overfitting is a common challenge in training large neural models, particularly in the fine-tuning phase. We utilize data augmentation techniques to resolve it. We perform image augmentation, including color jittering and image corruption, and add Gaussian noise to the input proprioception with a signal-to-noise ratio (SNR) of $40$dB. We also use GPT-4-Turbo to augment and expand the language instructions (Refer to Sec. E for more details on the instruction augmentation).

#### Some Fine-Tuning Details.

During fine-tuning, we removed a static part at the beginning of each episode, which might be caused by the operator not reacting after the recording started. Our language instructions are sampled from the original manually annotated instruction, the expanded instructions, and the simplified instruction with a probability of one-third. When the expanded instructions are drawn, we evenly sample one from the $100$ expanded instructions corresponding to the task. We did not apply Classifier-Free Guidance (CFG) because we found that this did not improve the performance of the model but instead brought the unstable robot arm behavior.

## Appendix G Hardware Details

Figure: Figure 7: Hardware features.
Refer to caption: x9.png

We provide a detailed overview of the hardware configuration of our target dual-arm robot. Our model is deployed and evaluated on the Cobot Mobile ALOHA, a robot using the Mobile ALOHA system design (Fu et al., ) and manufactured by [agilex.ai](https://global.agilex.ai). The key features of the robot are illustrated in Fig. 7
. It is equipped with two wrist cameras, a front camera, a laptop, and an onboard battery. The robot’s technical specifications are listed in Table 6. It is important to note we used the “mobile” ALOHA only to facilitate transportation and testing between various scenes and did not use its autonomous mobility feature during any training or inference stages. Our tasks are still static bimanual manipulation tasks.

**Table 7: Comparision of different baselines. We compare baselines as well as different variants of our model in terms of model size, data size, and modeling scheme.**
| METHOD NAME | LARGE MODEL | LARGE MULTI-ROBOT DATA | MODELING |
| --- | --- | --- | --- |
| ACT (Zhao et al., ) | ✗ | ✗ | VAE |
| OpenVLA (Kim et al., ) | ✓ | ✓ | Discretization |
| Octo (Ghosh et al., ) | ✗ | ✓ | Diffusion |
| RDT (scratch) | ✓ | ✗ | Diffusion |
| RDT (small) | ✗ | ✓ | Diffusion |
| RDT (regress) | ✓ | ✓ | Regression |
| RDT (ours) | ✓ | ✓ | Diffusion |

## Appendix H Experiment Details

#### Calculation of Total Performance.

The general performance in Fig. 1 of each method is calculated in three steps. Firstly, we calculate the success rate of a method in each task. We take an average of the total success rate and any additional requirement, i.e., the average of the values in the Total column and all columns to its right in Table 3. For example, in the Pouer Water-L-1/3, we take the average of Total, Correct Hand, and Correct Amount. Secondly, we calculate the success rate of each dimension of Unseen Object, Unseen Scene, Instruction Following, Few-Shot Learning, and Dexterity by averaging all the tasks in this dimension (see Table 1 for the correspondence). Lastly, we average the success rates of all the dimensions to obtain the overall result.

#### Implementation and Hyper-Parameters of RDT.

We list the details of the multi-modal encoders in Table 8 and the model parameter in Table 9. The image history size is $T_{\text{img}}=2$, the action chunk size is $T_{a}=64$, the language token space dimension is $4096$, the image token space dimension is $1152$, and the token space dimension of RDT is $2048$. We use adaptors to align each modality’s token dimension to $2048$. And all adaptors for multi-modal encoders are with GeLU activation (Hendrycks & Gimpel, ).

We use the AdamW optimizer (Adam et al., ) with a constant learning rate scheduler and hyper-parameters in Table 10 in the pre-training and fine-tuning stages. The model is pre-trained and finetuned on $48$ H100 80GB GPUs for $1$M steps and $130$K steps, respectively. Due to scheduling reasons, we did not start fine-tuning from the 1M pre-trained checkpoint but chose the 500K checkpoint. During the training stage, we use the DDPM scheduler with a glide cosine scheduler (i.e., squaredcos_cap_v2) and a step number of $1000$. During the sampling stage, we utilize the DPM-Solver++ (Lu et al., ) with a glide cosine scheduler and a sampling step number of $5$. During fine-tuning, we also filter out episodes with a length lower than $32$ and down-sample those with a length higher than $2048$ to $2048$.

**Table 8: Encoder configurations of RDT.**
| Modality | Encoder | Trainable | Adaptor |
| --- | --- | --- | --- |
| Language | T5-XXL (Raffel et al., ) | N | 2-layers MLP |
| Image | SigLIP (Zhai et al., ) | N | 2-layers MLP |
| Action | - | - | 3-layers MLP |

**Table 9: Model configurations for RDT.**
| Model | Layers | Hidden size | Heads | #Params |
| --- | --- | --- | --- | --- |
| RDT-1B | 28 | 2048 | 32 | 1.2B |

**Table 10: Hyper-parameters for both pre-training and fine-tuning RDT.**
| Hyper-Parameter | Value |
| --- | --- |
| Batch Size | 32$\times$48 |
| Learning Rate | $1\times 10^{-4}$ |
| Mixed Precision | bf16 |
| Warm-Up Steps | $500$ |
| $\beta_{1}$ | $0.9$ |
| $\beta_{2}$ | $0.999$ |
| Weight Decay | $1\times 10^{-2}$ |
| $\epsilon$ | $1\times 10^{-8}$ |

#### Implementation and Hyper-Parameters of ACT.

We directly employed the same architecture and hyper-parameters of ACT as that in the original paper (Fu et al., ), except for the hyper-parameters in Table 11. We trained ACT with $90\%$ of the $6$K fine-tuning episodes for $8000$ epochs (about $8$ days in total), while the remaining $10\%$ is treated as the validation set. We took the checkpoint at epoch $5413$ as the final outcome, according to the best performance in the validation set.

**Table 11: Adapetd hyper-parameters of ACT.**
| Hyper-Parameter | Value |
| --- | --- |
| Batch Size | 80$\times$4 |
| Learning Rate | $9\times 10^{-5}$ |
| Learning Rate for Backbone | $4\times 10^{-5}$ |

#### Implementation and Hyper-Parameters of OpenVLA.

We adopt the official implementation ([https://github.com/openvla/openvla](https://github.com/openvla/openvla)) and flagship pre-trained model and checkpoint at [https://huggingface.co/openvla/openvla-7b](https://huggingface.co/openvla/openvla-7b). For each task in evaluation, we further fine-tune the officially pre-trained OpenVLA with all the task-relevant demonstrations ($\sim 100$ episodes) from the fine-tuning dataset to facilitate convergence and train the model to around $95\%$ action token accuracy as suggested by Kim et al. () ([https://github.com/openvla/openvla/issues/12#issuecomment-2203772810](https://github.com/openvla/openvla/issues/12#issuecomment-2203772810)). Additionally, we experimented with both full-parameter tuning and LoRA methods using the entire dataset but did not achieve sufficient action token accuracy (approximately $60\%$) for deployment upon convergence (see Fig. 8). According to real-robot testing, such non-convergent checkpoints exhibit completely static or random
behaviors in the deployment.

Concretely, we adhere to the same hyper-parameters claimed in Kim et al. () for fine-tuning via LoRA (Hu et al., ) as detailed in Table 12.

Figure: Figure 8: The accuracy of action token prediction fluctuates rather than converges with the number of training steps when fine-tuning OpenVLA with the full fine-tuning dataset.
Refer to caption: extracted/5903256/openvla_train_acc.png

**Table 12: Hyper-parameters of fine-tuning OpenVLA for bimanual manipulations.**
| Hyper-Parameter | Value |
| --- | --- |
| Batch Size | 16$\times$8 |
| Learning Rate | $2\times 10^{-5}$ |
| Lora Rank | 32 |
| Image Augmentation | True |

#### Implementation and Hyper-Parameters of Octo.

We utilize the official implementation available at [https://github.com/octo-models/octo](https://github.com/octo-models/octo) and the most comprehensive pre-trained model, octo-base-1.5, hosted at [https://huggingface.co/rail-berkeley/octo-base-1.5](https://huggingface.co/rail-berkeley/octo-base-1.5). We follow the officially recommended practices for fine-tuning a bimanual robot, detailed in [https://github.com/octo-models/octo/blob/main/examples/02_finetune_new_observation_action.py](https://github.com/octo-models/octo/blob/main/examples/02_finetune_new_observation_action.py), employing a full-parameter approach. Additionally, we have incorporated an extra image tokenizer to process images from the right-wrist camera, enhancing the system’s manipulation capabilities. Furthermore, by integrating image augmentation during the fine-tuning process, we enhance the performance upon deployment in real-world robotics. We replicate the wrist image tokenizer from the pre-trained model to initialize the right-wrist image tokenizer. Similar to OpenVLA, we only fine-tune octo with the task-relevant demonstrations for each evaluation tasks, for we do not observe sufficient test MSE (approximately $10^{-1}$) for deployment upon convergence (Fig. 9). Concretely, we apply the default hyper-parameters with variations listed in Table 13:

Figure: Figure 9: The test MSE of action prediction fluctuates rather than converges with the number of training steps when fine-tuning Octo with the full fine-tuning dataset.
Refer to caption: extracted/5903256/octo_test_mse.png

oct

**Table 13: Hyper-parameters of fine-tuning Octo for bimanual manipulations.**
| Hyper-Parameter | Value |
| --- | --- |
| Action Head Type | DiffusionActionHead |
| Batch Size | 8$\times$8 |
| Action Chunk Size | 8 |
| Image Augmentation | RandomBrightness(0.1) |
| RandomContrast(0.9, 1.1) |  |
| RandomSaturation(0.9, 1.1) |  |
| RandomHue(0.05) |  |

## Appendix I More Results

We further provide a zoom-in view for water-level across $8$ trails in instruction-following evaluation in Fig. 10.

Figure: Figure 10: Visulization of the resulting water levels across $8$ trails in Pour Water-L-1/3 and Pour Water-R-2/3. Left: The water level completed by RDT in each trial is extremely close to the ground-truth 1/3 standard. Right: RDT made one mistake in pouring (empty cup) and one mistake in water level, but the other trials were in roughly good agreement with 2/3.
Refer to caption: x10.png

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: - The model uses frozen vision encoders. Does it mean existing pretrained visual representation is sufficient for robot manipulation? I wonder if the authors spot any cases where pretrained visual encoder is insufficient and lead to unsatisfactory performance.
- In pretraining on heterogeneous data with varied control frequency, the model takes control frequency as conditioning. I wonder how this strategy works in practice - does the model learns policy prior corresponding to different control frequency?
soundness: 3
strengths: The paper presents a complete and remarkable research work that pushes forward the boundary of large-scale robot learning. 
- The model is developed on top of the diffusion transformer with a unified action space, which allows large-scale pretraining on heterogeneous robot data to boost the performance
- The authors collect the largest robot dataset for bimanual manipulation with comprehensive task coverage for fine-tuning the model
- The experiments show that the advantage of the model from a foundation model aspect: generalization, few-shot learning, and scaling behavior
summary: This paper develops a 1.2B-parameter robotics foundation model that is trained and evaluated on real robot data for bimanual manipulation. The model is trained with imitation learning: (i) pretrained on 1M trajectories combining available datatsets collected for different robots, and (ii) fine-tuned on a self-collected dataset with 6k demonstrations for a Mobile Aloha robot. The model adopts a diffusion transformer (DiT) architecture that takes multi-modal inputs (images, language, etc.) and generate action chunks with multimodal distribution. 

The model is evaluated on 7 real robot tasks against mainstream baselines. The comparison shows that the model can: (i) generalize zero-shot to novel objects, scenes and language, (ii) learn new skills with few data, (iii) accomplish dexterous tasks. Ablation studies show that larger model and pretraining with large data significantly boost the performance.
weaknesses: - While the paper demonstrates that the foundation model is allows zero-shot and few-shot generalization, and can achieve dexterous manipulation, each of these characteristics is only validated on ~one task and may be insufficient. Evaluations on more tasks and existing benchmark tasks will complete the results.
- It seems that the baselines are not trained on the complete fine-tuning dataset. This doesn't form an apple-to-apple comparison.
- The writing of the paper has room for improvement. Some of the sentences are too long, which prevent reading of the paper smoothly.
- Mistakes in citing papers. For example, it seems that "Xie et al. 2020" is wrongly used as the reference of DiT in line 83 and 319.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: My questions are mainly also discussed in the weakness. Here is an summary:
1. Given that the diffusion transformer already contains multiple nonlinear layers, why does adding an additional layer improve performance? Have you conducted ablation studies to support this choice?
2. In the original UNet diffusion policy, the last layer is a Conv1dBlock. Have you compared the performance of your MLP block with this alternative?
3. What is the the MLP block performance gains on other tasks?
4. Can you provide more insight into how the model design specifically addresses high frequency change?
5. What are the performance gains from using all data through this unified action space compared to previous methods that only use robots with similar action spaces or retain subsets of inputs with a common structure?
6. How does the proposed padding strategy improve performance compared to zero padding? Could you provide comparative results?
7. How do these normalization layers integrate into the transformer architecture, and how do they address instability? Have you tested different configurations of these layers?
soundness: 3
strengths: This paper demonstrates strong performance in scaling up robotics models. It presents several interesting components that improve training stability and performance.

The unified action space, and especially the padding technique, is interesting.

The paper shows capabilities on several challenging real-world bimanual manipulation tasks.
summary: This paper presents an effort toward building a foundation model for bimanual manipulation. It proposes techniques to unify the action space, enabling training on a very large robot dataset. The authors also scale up the model to 1.2B parameters, making it the largest diffusion transformer model for robotics. In this process, the authors identify several key elements to improve training stability and performance. The resulting model achieves good zero-shot generalization performance on unseen and complex tasks.
weaknesses: Several claims are not very precise and not very clear. For example, the authors mention the nonlinearity and high frequency of robotic data. While it is true that the data is nonlinear, how does the proposed method tackle this challenge? The authors argue that changing the last linear layer to an MLP block solves this problem and brings significant performance improvements. While the performance is impressive, I think this requires more careful ablation experiments. Firstly, the entire diffusion transformer is already highly nonlinear due to stacking multiple layers, so why does adding more layers help in this case? Secondly, in the original UNet diffusion policy, the last layer is a Conv1dBlock; have the authors compared with that? Lastly, why is it only evaluated on the dexterity task?

It is also unclear how the model design choices address the “high frequency of robotic data.” Given that these two claims are highlighted in the abstract as the main challenges, I believe they require more careful analysis and discussion.

As the unified action representation is a major contribution of this paper, there should also be more analysis of this aspect. For example, what are the performance gains from using all the data because of the unified action space, compared to previous methods that use “robots with similar action spaces (Yang et al., 2023; Ghosh et al., 2023; Kim et al., 2024) or retain only a subset of inputs sharing the same structure (Collaboration et al., 2023; Yang et al., 2024)”? Additionally, what is the performance gain of the proposed padding strategy compared with padding with all zeros?

The authors argue the necessity of using RMSNorm and QKNorm, but they only show the loss without them (Figure 4(a)), which provides very little information on how effective the proposed approach is and whether it addresses the instability issue. It also does not mention how to integrate these normalization layers within the transformer.

Scaling up the models and data is certainly attractive, and the paper shows impressive results. However, most of the analysis are “binary”, which means the results are either with or without. Showing more datapoints (model with different size, using different percentage of the dataset) will present more insights to the reader.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: - Could the authors elaborate on why diffusion models were specifically chosen over other generative methods like VAEs or GANs for this task? While diffusion models show high expressiveness, a comparison or rationale would clarify their unique benefits in bimanual manipulation.

- The Physically Interpretable Unified Action Space is innovative, but how does it handle robots with vastly different kinematics or action constraints?
soundness: 4
strengths: - The paper introduces a novel application of diffusion models to bimanual manipulation, addressing the high-dimensional, multi-modal action space through a Physically Interpretable Unified Action Space. This approach is a creative extension of diffusion models in robotics, particularly for dual-arm coordination, a challenging domain with limited prior work.

- The model is rigorously tested, with comprehensive experiments demonstrating superior performance over existing baselines. The use of the largest multi-robot dataset and a specialized bimanual dataset for fine-tuning enhances the validity of results and supports the model's effectiveness.

- This work contributes substantially to the field by advancing foundation models for robotic manipulation. RDT’s capabilities for zero-shot generalization, few-shot learning, and instruction following mark a significant step towards adaptable and scalable robotic models, with promising implications for real-world applications.
summary: This paper presents the Robotics Diffusion Transformer (RDT), a large-scale diffusion-based model for bimanual robotics. RDT introduces a Physically Interpretable Unified Action Space to standardize actions across robots, enhancing generalization, and uses diffusion models to handle complex, continuous action distributions. Pre-trained on the largest multi-robot dataset and fine-tuned on a custom bimanual dataset, RDT demonstrates strong zero-shot and few-shot generalization, outperforming existing methods in dexterous, real-world tasks.
weaknesses: - The paper introduces a Physically Interpretable Unified Action Space for handling data heterogeneity, but additional details on potential limitations or failure cases during training with highly diverse data would be beneficial. This could include examples where action standardization might lead to loss of unique features across robots.

- Although the experiments show impressive results, expanding the evaluation to more varied and complex real-world tasks (beyond the 6,000-episode dataset) and more hardwares(beyond ALOHA) could further validate RDT's robustness.

- The paper proposes several innovative multi-modal encodings (e.g., masking, cross-attention) but lacks ablation studies on these design choices. Showing how each component contributes to performance could clarify their impact on handling visual and language-conditioned tasks effectively.

- There's a typo at L76, it should be "data" instead of "date" .

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. "Does the RDT-1B fine-tuning use the entire self-collected dataset, and is it not fine-tuned separately for each task in the evaluation?"
2. "Regarding the computation of success rate for the 'wash cup' task with 'seen cup1': the success rate (SR) for 'get water' is 50, for 'pour water' is 87.5, and for 'place back cup' is also 87.5, yet the overall SR is listed as 50. Since the 'get water' subtask has an SR of 50, and the following subtasks have SRs below 100, how is the total SR calculated as 50?"
soundness: 2
strengths: A foundational model for bimanual manipulation is absent in the current community, which is an important direction; aligning different robot embodiments is also a crucial question for pre-training on large-scale datasets. The proposed action representation is simple yet effective for pre-training on diverse robot datasets.
summary: This paper proposes a unified action representation to align different robots, facilitating pre-training on diverse robot datasets for bimanual manipulation. Additionally, it introduces a diffusion transformer-based architecture with several modifications for enhancing policy learning, and scaling up with large datasets.
weaknesses: Compared to the robot datasets used for pre-training the baseline, such as OpenVLA, this paper appears to use a more diverse set of datasets, including additional bimanual manipulation datasets like ALOHA and Mobile ALOHA, contributing nearly 10% of the total datasets. Fine-tuning a baseline pre-trained on single-arm datasets for a bimanual manipulation setting may result in poor performance on bimanual tasks, making it difficult to demonstrate that using the diffusion transformer architecture is superior to using a large language model as the pre-training backbone.

Meta-review: The paper introduces RDT (Robotics Diffusion Transformer), a diffusion-based foundation model for bimanual manipulation. The key scientific claims and findings include:
+ A novel vision-language-action (VLA) architecture based on diffusion transformers that effectively handles multi-modal inputs and complex action distributions for bimanual tasks
+ A Physically Interpretable Unified Action Space that enables training on heterogeneous robot datasets while preserving physical meaning
+  Successful scaling to 1.2B parameters through pre-training on the largest collection of multi-robot datasets to date
+ Strong empirical results demonstrating zero-shot generalization, few-shot learning, and instruction following capabilities on real robot hardware

Weaknesses:
- More analysis could be provided on limitations of the unified action representation and potential failure cases
- Additional details on scaling behavior and model variants would strengthen empirical validation
- Comparison with baselines could be more equitable in terms of training data and protocols
- Some implementation details of the architecture could be better clarified

The clear strengths in terms of novel technical approach, strong empirical results, and potential impact on the field outweigh the limitations. I recommend accepting this paper.

Decision: Accept (Poster)