Title: VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Imitation learning

Video prediction

Robot Manipulation
Abstract: Recent advancements utilizing large-scale video data for learning video generation models demonstrate significant potential in understanding complex physical dynamics. It suggests the feasibility of leveraging diverse robot trajectory data to develop a unified, dynamics-aware model to enhance robot manipulation. However, given the relatively small amount of available robot data, directly fitting data without considering the relationship between visual observations and actions could lead to suboptimal data utilization. To this end, we propose \textbf{VidMan} (\textbf{Vid}eo Diffusion for Robot \textbf{Man}ipulation), a novel framework that employs a two-stage training mechanism inspired by dual-process theory from neuroscience to enhance stability and improve data utilization efficiency. Specifically, in the first stage, VidMan is pre-trained on the Open X-Embodiment dataset (OXE) for predicting future visual trajectories in a video denoising diffusion manner, enabling the model to develop a long horizontal awareness of the environment's dynamics. In the second stage, a flexible yet effective layer-wise self-attention adapter is introduced to transform VidMan into an efficient inverse dynamics model that predicts action modulated by the implicit dynamics knowledge via parameter sharing. Our VidMan framework outperforms state-of-the-art baseline model GR-1 on the CALVIN benchmark, achieving a 11.7\% relative improvement, and demonstrates over 9\% precision gains on the OXE small-scale dataset. These results provide compelling evidence that world models can significantly enhance the precision of robot action prediction. Codes and models will be public.

VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

Abstract

Recent advancements utilizing large-scale video data for learning video generation models demonstrate significant potential in understanding complex physical dynamics. It suggests the feasibility of leveraging diverse robot trajectory data to develop a unified, dynamics-aware model to enhance robot manipulation. However, given the relatively small amount of available robot data, directly fitting data without considering the relationship between visual observations and actions could lead to suboptimal data utilization. To this end, we propose VidMan ( Vid eo Diffusion for Robot Man ipulation), a novel framework that employs a two-stage training mechanism inspired by dual-process theory from neuroscience to enhance stability and improve data utilization efficiency. Specifically, in the first stage, VidMan is pre-trained on the Open X-Embodiment dataset (OXE) for predicting future visual trajectories in a video denoising diffusion manner, enabling the model to develop a long horizontal awareness of the environment's dynamics. In the second stage, a flexible yet effective layer-wise self-attention adapter is introduced to transform VidMan into an efficient inverse dynamics model that predicts action modulated by the implicit dynamics knowledge via parameter sharing. Our VidMan framework outperforms state-of-the-art baseline model GR-1 on the CALVIN benchmark, achieving a 11.7% relative improvement, and demonstrates over 9% precision gains on the OXE small-scale dataset. These results provide compelling evidence that world models can significantly enhance the precision of robot action prediction. Codes and models will be public.

1 Introduction

In the rapidly advancing field of robotics, accurately predicting and executing precise actions based on sensory inputs is crucial. While traditional approaches [1-5] for robot manipulation often rely on labor-intensive hand-engineered features and models prone to errors, data-driven methods [6-8] offer promising solutions. However, the challenge lies in the difficulty and cost of acquiring high-quality robotic data. Recent advancements [9-12], particularly those utilizing large-scale online video data for learning a video generator, demonstrate significant potential in comprehending complex physical dynamics of the real world. These models, trained on diverse datasets [13, 8], possess a nuanced understanding of the world, suggesting the feasibility of amalgamating and leveraging varied robot visual trajectory data [14-16] to develop a unified dynamics-aware model for enhanced robot manipulation. Yet, achieving this unification poses challenges; merely fitting data without considering the relationship between visual observations and actions could lead to suboptimal utilization of the data. Hence, there is a pressing need to develop efficient training mechanism and model architecture

VidMan's Two-Stage Training

Slow

Instruction:"Place the

Dynamics-aware Visionary

System

2

Planning

Problem

1st Stage

bottleinthebowl"

Iterative

St1

St+1

Open-Sora

solving

Denoising

Layer-wise

Modulate

Instruction:"Place the

Adapter

Dynamics-modulated Action

System

1

Fast

2nd Stage

bottleinthebowl

St+1

Open-Sora

Automatic

One Pass

StM

O

Unconscious

that can effectively leverage existing cross-robot and cross-scene data to enhance action prediction accuracy.

As shown in Fig. 1, to optimize the utilization of diverse robot data [8], we draw upon insights from neuroscience's dual process theory [17-19], which unveils the complex mechanisms of information processing and decision-making in the human brain. This theory distinguishes between two cognitive processes: System 1, responsible for rapid, intuitive responses based on immediate sensory inputs, and System 2, which involves slower, long-horizon planning grounded in abstract concepts and understanding of world dynamics [20]. Inspired by these insights, we adopt a two-stage paradigm for robot learning, exemplified by our innovative framework, terms as VidMan ( Vid eo diffusion for robot Man ipulation). VidMan leverages the power of the video diffusion generation method Open-Sora [21] for robot imitation learning, tapping into the awareness of long-horizon dynamics inherent in video diffusion models to achieve more nuanced and dynamics-modulated robot action prediction. By learning different facets of data at distinct stages, VidMan acquires an inductive bias of inverse dynamics of robot control, wherein actions are the outcomes of state sequences, significantly enhancing the method's generalization performance, especially under scenarios with limited data.

Specifically, our VidMan employs a two-stage training mechanism, akin to the principles of dual process theory, to enhance stability and significantly improve data utilization: 1) In the first stage, namely the Dynamics-aware Visionary Stage , VidMan undergoes pre-training on the Open X-Embodiment [8] dataset (OXE) using a video diffusion model to predict future trajectories based on historical observations and language instructions. This stage involves the robot learning the dynamics of state transitions from data and accurately perceiving the current environmental state, enabling the model to develop a deep understanding of the environment's dynamics, forming a robust foundation for accurate action prediction; 2) In the second stage, dubbed the Dynamics-modulated Action Stage , VidMan incorporates a flexible yet powerful layer-wise self-attention adapter [22] to seamlessly integrate the pre-trained knowledge from the first stage into action prediction. Through shared neural architecture and parameters with the dynamics-aware visionary stage, this phase transforms VidMan into an implicit inverse dynamics model that infers dynamics-modulated actions without explicitly generating future visual trajectories, rendering it suitable for real-world robot control scenarios.

The performance of VidMan has been evaluated against SOTA baselines on the CALVIN [14] benchmark, where it achieves a 11.7% relative improvement. In addition, VidMan has shown notable effectiveness on the OXE small-scale dataset, achieving over 9% precision gains. This improvement is particularly evident when the data from the target robot is small, underscoring the effective data utilization of our method. Extensive ablation studies have been conducted to analyze the effects of various design decisions within our method. These experimental results suggest that VidMan represents a meaningful advancement in robotics, providing a valuable tool for developing more capable and responsive robotic systems.

2 Related Work

Language-guided Robot Manipulation. Language-guided robot manipulation has emerged as an elastic and straightforward approach to instructing robots to perform various tasks [14, 23]. Some existing methods [24, 25, 10, 26-30] leverage large language models (LLMs) to plan over the task domain and pass instructions to low-level action policies to generate robot actions. The hierarchical 2D policies [31-33] predict latent features or images of subgoals given a language instruction, which they feed into lower-level subgoal-conditioned policies. 3D policies [34, 35] combine 3D representations with diffusion objectives to learn manipulation from demonstrations using depth maps and camera extrinsics. Some methods [36, 37] also utilize 3D [38-40] or 2D [41-43] detection to identify objects and use constrained optimization methods to control robot operations Another line of work learns language-conditioned policies from unstructured play data, which contains demonstrations with and without language labels [44, 32]. These methods leverage sequence-to-sequence conditional variational auto-encoders to generate latent plans, which are then used to condition the action policy.

Pre-training for Robot Manipulation. The field of pre-training for robot learning has garnered significant attention in recent years [9, 26, 11, 45]. Some methods aim to learn useful visual representations through masked image modeling [9] and contrastive learning [11]. Previous research [46, 7, 47, 45, 25, 48] has focused on empowering robots and other agents with the capability to comprehend and execute language instructions, typically by learning policies conditioned on language. GR-1 [9] and RoboFlamingo [26] use a GPT-style framework to model action prediction as a token prediction task in CALVIN dataset [14], and achieve good results. Our approach is most similar to GR-1 in that it utilizes GPT-style framework to predict both video and action, while ours directly uses video's ability to capture future information to predict long sequences of actions.

3 Preliminaries

Robot Dynamics and Inverse Dynamics. The dynamics of a robot are typically characterized by the forward dynamics and the inverse dynamics. The forward dynamics describe how the current state and action determine the next state. Formally, given a state space S and an action space A , the forward dynamics can be represented as the conditional probability distribution: P ( s t +1 | s t , a t ) , where s t , s t +1 ∈ S are the robot state at timestep t and t +1 , respectively, and a t ∈ A is the action taken. Conversely, inverse dynamics describe the probability of an action given a transition from one state to another. The inverse dynamics are particularly important in scenarios where only passive observation data is available, such as data collected from the internet, which often lacks explicit action information. Formally, the inverse dynamics are given by:

Observations to States. In many real-world applications, direct access to the state space S is infeasible, and instead, sequences of image observations are provided. Let O denote the observation space, e.g. , image, proprio. To infer the underlying sequence of states S = { s 1 , s 2 , . . . , s T } from a sequence of observations O = { o 1 , o 2 , . . . , o T } . This process can be formally described by finding the sequence of states that maximizes the posterior probability given the observations:

Using Bayes' theorem, this can be further expressed as:

where P ( O | S ) is the likelihood of the observations given the states, and P ( S ) is the prior probability of the sequence of states. This formulation is essential in various applications, including hidden Markov models (HMMs) and other state estimation techniques. With more observations, the likelihood P ( O | S ) becomes more peaked around the true state sequence. This is because more data points allow for better discrimination between different state sequences.

4 Method

In this paper, we aim to enhance the precision of robot action prediction by exploiting the intricate dynamics encoded within robot visual trajectories. We propose VidMan, a novel framework leveraging

Robot Visual Trajectory

Uniform Sample

Video Tokenizer

𝑂ℎ

𝑂𝑓

𝑉ℎ

𝑉 𝑓

…

…

…

…

…

…

𝑉 𝑠

𝑉 𝑠

𝑂𝑠

𝑉 𝑓

→ 𝑉 0

𝜀 𝑉 𝑠

, 𝜖, 𝑘

→ 𝑉 𝑠

𝑘

𝑉 𝑐

𝑘

Open-Sora

𝜖𝜃 (𝑉

𝑐

𝑘 , 𝑦, 𝑘)

…

…

…

…

Language Instruction

𝒚

:

'Put the pot on the plate'

Text Tokenizer

𝑘

Open-Sora Block

×

12

ℒ𝑣

𝜖

𝑉ℎ

𝑉 𝑓

Action Head

𝑉action

ℒ𝑎

𝜖

'

1 st

Stage

2 nd

Stage

𝑄action

𝜙ad𝑎

(a)

(b)

Video diffusion for robot Manipulation. VidMan employs a dual-stage training strategy: in the first stage, the Dynamics-aware Visionary Stage , we enable the model to forecast and imagine potential future trajectories based on historical observations, leveraging the multi-frame prediction capability of the video diffusion model. Through this stage, the model is optimized to understand the dynamics of the environment. In the second stage, the Dynamics-modulated Action Stage , we introduce a lightweight layer-wise adapter to seamlessly integrate the visionary predictive stage with fast, adaptive action prediction. This approach decouples the knowledge of the world and embodiment into distinct processes while ensuring seamless integration through the training and utilization of shared parameters. Overview of our method is shown in Fig. 2. In the following sections, we will formulate each of these stages.

4.1 Dynamics-aware Visionary Stage

To endow the model with the knowledge of world dynamics, we formulate this knowledge acquisition stage as future image trajectory generation, which captures dynamic priors and predicts future state transitions more accurately according to Equ. (3). Specifically in our context, the goal is to predict future frames based on historical frames and language instructions. To achieve this, we leverage the capabilities of a multi-frame generation framework based on the video diffusion transformer model (VDT) Open-Sora [21], which has shown a remarkable ability to generate diverse and physically authentic successive frames aligned with language instructions. For simplicity, we use VDT to represent Open-Sora in the following content.

To prepare the training data, we utilize a pre-trained video tokenizer [49] to encode successive robot images in a trajectory O s = [ O h , O f ] into embeddings V s = [ V h , V f ] , where V h ∈ R m × W × H × C and V f ∈ R n × W × H × C stand for embeddings for m history images O h and n future images O f , respectively. During the forward diffusion process V k s ← ε ( V s , ϵ, k ) , the noise ϵ ∼ N (0; 1) is added to trajectory embeddings V s according to the diffusion step k ∈ [1 , K ] . The diffusion model is optimized to predict the added noise ϵ from V k s given the condition hints. As the condition hints, we pad V h with zero-valued embeddings V 0 to the same shape as V k s , which are concatenated with V k s along the channel dimension to form the condition visual embeddings V k c ∈ R ( m + n ) × W × H × 2 C . And the language instruction y is encoded by an text tokenizer [50] to obtain the language embedding. Mathematically, the denoised diffusion model is optimized according to:

In this training stage, we use only a third-person camera to predict the representation. This approach has two main advantages: a) most robot datasets include only third-person view data; b) training with a fixed third-person viewpoint can reduce the influence of view changes and help the model focus on predicting the transitions of the robotic arm itself. Additionally, our method can easily be extended to multiple cameras by simply inputting multiple cameras of viewpoint.

4.2 Dynamics-modulated Action Stage

According to Equ. (1), actions can be accurately predicted given the states with the inverse dynamics model. One straightforward approach to combine an inverse dynamics model with the VDT learned in the first stage is to separately construct an inverse dynamics model that maps from image observations to actions. During deployment, this model could predict actions from the generated observations of the dynamics-aware visionary stage. However, by this means, actions are only predictable after the VDT conducts a time-consuming iterative denoising diffusion process, which is not ideal for high-frequency robot control. Moreover, the accuracy of the actions is heavily dependent on the accuracy of the predicted observations. Since not all pixels are important for predicting actions, this method could introduce unnecessary bias and time costs. Additionally, learning a separate inverse dynamics model from scratch does not leverage the pre-trained parameters of the VDT.

To address these issues, we propose directly adapting the VDT into an inverse dynamics model. In this way, the dynamics knowledge and implicit states learned in VDT can be seamlessly leveraged to facilitate the prediction of actions. Below, we introduce a lightweight adapter module that effectively transforms the VDT into an implicit inverse dynamics model. The output of this adapted model is then decoded into a sequence of actions using a diffusion-based action head.

Implicit Inverse Dynamics Adapter. To transform the VDT into an inverse dynamics model, we incorporated a layer-wise adapter which is inspired by [22] after each layer in VDT. Each adapter includes a multi-head self-attention and a feed-forward network (FFN) with a gated mechanism. We use h learnable action tokens Q action, concatenating them with the features ouput from each layer of the VDT, and input them into the layer-wise adapter. This fuses the knowledge of each layer of the VDT to produce h final action embeddings V action. Since we only reuse VDT parameters without its observation generation function, we disable the iterative denoising process by using a fixed diffusion step k ← K , which turns V k s into pure Gaussian noise. Formally, the action embeddings are fused with the introduce layer-wise adapter parameterized by ϕ :

where ϵ ( θ,ϕ ada ) represents the VDT parameters incorporated with the layer-wise self-attention adapter with parameters ϕ ada. The flexibility of the layer-wise adapter allows for the integration of domainspecific knowledge into the action embedding. For example, if the downstream robotic manipulation task includes proprioceptive information, we can use a proprioception embedder to convert it into tokens, which are then concatenated with the output of each layer of the VDT to serve as the additional keys and values for the layer-wise adapter. Please refer to Appendix A.2 for more details.

Diffusion-based Action Head. The fused action embeddings, V action, are subsequently translated into low-level control signals. To achieve this, we utilize a diffusion-based action head [51], π ϕ dec , responsible for decoding action embeddings into executable action signals, such as determining the 7 degrees of freedom (DoF) for the end-effector pose and gripper status. Similar to the dynamics-aware visionary stage, the objective of action prediction is:

where ϵ ′ ∼ N (0 , 1) , l ∈ [1 , L ] stands for the diffusion step and ε denotes the forward diffusion process. Note that this diffusion-based action head is relatively small compared to the VDT, making its computational cost and time consumption almost negligible.

5 Experiment

In this section, we conduct comprehensive experiments to validate the effectiveness of our methods, along with in-depth ablation studies.

5.1 Experiment Settings

5.1.1 2-stage Training Setting

Settings of dynamics-aware visionary stage. We initialized our model with the weights from Open-Sora's [21] 16x256x256 text-to-video model, which was trained on internet data [52-54]. Since we concatenated historical frames along the channel dimension, we append a zero matrix to the parameters of the tokens embedder layer in Open-Sora accordingly. We use Open X-Embodiment Dataset [8] to train our model, which was constructed by pooling 60 existing robot datasets from 34 robotic research labs around the world. We referred to the Octo [7] codebase to selected 25 datasets from it, which are heterogeneous not just in terms of the robot type, but also in the sensors (e.g., including or not including wrist cameras) and labels (e.g., including or not including language instrctions). Unless otherwise specified, we sampled video sequences from trajectories at intervals of 3, resulting in 4-frame video sequences, with 2 historical frames and 2 future frames. At this stage, we only used images from the third-person camera with 256 × 256 resolution. We use a maximum of K = 1000 diffusion step.

Settings of dynamics-modulated action stage. In this training stage, we used data from both the Open X-Embodiment (OXE) and CALVIN [14] datasets for training. We additionally incorporated images from a wrist camera as extra observations. For the data in OXE, we predicted 12 action steps, whereas for the CALVIN data, we predicted 10 steps. For the diffusion policy head, we set the noise addition steps to 100. We used a 224 × 224 third-person camera and a 224 × 224 wrist camera. In this stage, we set VDT's diffusion step to the maximum, i.e., k = K , and used pure noise as V s K . The VDT does not conduct the iterative denoising process. As for the diffusion-based action head, we set the maximum diffusion step as L = 100 . More details can be found in Appendix A.2

5.1.2 Benchmark and Baselines

Simulation Evaluation: The CALVIN benchmark utilizes the PyBullet[55] simulator and involves a Franka Panda Robot arm interacting with various environments labeled A, B, C, and D. Each environment includes a desk, a sliding door, a drawer, a button controlling an LED, a switch for a lightbulb, and three colored blocks (red, blue, and pink). These environments differ in desk textures and object positions. CALVIN provides 24 hours of unstructured tele-operated play data, with 1% annotated with language descriptions. Each instruction chain consists of five sequential language instructions for execution. Evaluation follows a zero-shot generalization setup, training models on environments A, B, and C and testing on D. Performance metrics include success rates and average completion of sequential tasks, as per prior studies[26, 9]. Notably, CALVIN lacks a motion planner, requiring all models to predict robot pose trajectories.

Offline Evaluation: We also report offline metrics, including the average of xyz accuracy and euler angle accuracy (Avg xyz ang) and MSE for end-to-end action prediction on Bridge [56], Taco Play [57], Cable Routing [58] and Autolab UR5 [59], which are presented in OXE [8]. Following Octo [7], we use continuous action space. XYZ accuracy measures the precision of the robot's predicted 3D position (X, Y, Z coordinates) compared to the ground truth values during evaluation. Euler angle accuracy measures the precision of the robot's predicted orientation angles (rotations around X, Y, and Z axes) compared to the ground truth values during offline evaluation. Specifically, XYZ accuracy refers to whether we predicted the XYZ delta within 0.5 radians and 50% of the norm while in motion. Euler angle accuracy indicates whether we predicted the rotation delta within 0.5 radians during movement. Additionally, we reported the mean squared error (MSE) which reflects how well each model predicts the actions.

Baselines: On the CALVIN benchmark, we compare our approach to the hierarchical 2D policies of MCIL [31], HULC [32], and SuSIE [33], which predict latent features or subgoal images based on language instructions and feed them into lower-level subgoal-conditioned policies. These methods can train the low-level policy using the full CALVIN dataset, rather than being restricted to the language-annotated subset. We also compare against large-scale 2D transformer-based policies like RT-1 [47], RoboFlamingo [26], and GR-1 [9], which pretrain on extensive interaction or observational (video-only) data. Additionally, to better highlight the performance of our method, we compare with the 3D methods, 3D Diffusion Policy [34] and 3D Diffuser Actor [35]. Both share a similar goal of combining 3D representations with diffusion objectives to learn manipulation from demonstrations using depth maps and camera extrinsics. As for OXE benchmark, we compared our method with

RT-1-X [8] and Octo [7]. RT-1-X is an openly available generalist robot policy which are trained on OXE using RT-1 [47] framework. Octo is a transformer-based diffusion policy that supports flexible task and observation definitions, pretrained on OXE. These baselines are used to validate the benefit of our two-stage training strategy since both RT-1-x and Octo are optimized with all data in a train-from-scratch manner. To demonstrate the effectiveness of using diffusion models in learning dynamics priors and modeling states in the first stage, we implemented a GPT-style (VidMan-GPT) policy, as a baseline. This baseline autoregressively predicts the next images and actions in the first stage. This allows us to compare, in a fair manner, the benefits of simultaneously predicting multiple frames in a diffusion-like manner versus predicting only the next frame in a GPT-like fashion for modeling dynamics priors and state representations. We only predict a single frame and the corresponding action. We discarded the diffusion scheduler and used random masking [60] to reconstruct images to calculate the video loss.

5.2 Comparison Results

Multi-Task Performance. We first conducted the initial stage of training on OXE, then utilized CALVIN's static camera and wrist camera as the two input sources for the second stage of training on CALVIN. Following the setup in GR-1, we used the portion of the CALVIN dataset that includes language instruction labels, while also incorporating CALVIN's proprioceptive state data as additional input into the layer-wise adapter. Tab. 1 compares the performance of mainstream methods such as SuSIE, RT-1, RoboFlamingo, GR-1, and 3D Diffuser Actor against VidMan. Compared to hierarchical 2D policies like SuSIE, which struggle to effectively leverage language information at the low level despite being able to train on the entire dataset ("All"), these methods generally underperform in terms of generalization compared to end-to-end policies like GR-1 and RoboFlamingo. Our method adopts an end-to-end training approach, outperforming the best hierarchical architecture, SuSIE, by 0.73 in Avg. Len., while using less data. When compared to large-scale 2D transformer-based policies such as GR-1 and RoboFlamingo, which also use an end-to-end output approach, our method is highly comparable. Thanks to pretraining on the large-scale robotics dataset OXE and utilizing the Open-Sora architecture, which is more vision-friendly, our method outperforms GR-1 by a clear margin on average length (11.7% relative improvement). In comparison to 3D Diffuser Actor, which leverages depth information to predict actions, our model remains competitive due to the ability to pretrain on large-scale datasets without depth information, which are easier to collect.

Offline Performance. To evaluate offline performance, in this part, the second stage of our model is optimized with OXE dataset. We evaluate performance on in-distribution tasks. Specifically, we conduct two parts of evaluation: 1) the evaluation on domains that have small-scale sub-datasets in OXE (Taco Play, Cable Routing and AUTOLab UR5), where we would expect pretraining with larger datasets can significantly improve performance; 2) the evaluation on domains with large-scale dataset (Bridge), where we expect further improvement to be more challenging. We compare our results with RT-1-X, Octo-small, Octo-base, and VidMan-GPT on the four sub-datasets. We compared the average

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

0

10

20

30

40

50

60

Bridge

0

1

2

3

4

5

6

0

10

20

30

40

50

60

70

Taco Play

0

2

4

6

8

10

12

14

16

0

5

10

15

20

25

30

Cable Routing

0

2

4

6

8

10

12

14

0

5

10

15

20

25

30

35

MSE

Autolab UR5

RT-1-X

Octo-small

Octo-base

VidMan-GPT

VidMan

MSE

values of xyz accuracy and angular accuracy (Avg xyz ang), the higher the better, as well as the MSE (the lower the better). As shown in Fig. 3, it can be observed that our method outperforms the state-of-the-art open-source method, Octo, in both evaluation settings. Particularly on the small-scale sub-datasets CableRouting and Autolab UR5, our method improves the offline average xyz angle accuracy by 9.9% and 9.0% over Octo, respectively. The key difference between Octo and our method is in the optimization approach. Octo employs a co-training strategy, utilizing all data at once. However, our two-stage training strategy demonstrates a significant improvement, especially in domains with limited data. This improvement is attributed to our approach's ability to utilize the training data more efficiently by implicitly leveraging the inductive bias of inverse dynamics. Regarding Taco Play, the improvement we achieved is less notable. This is likely because the Taco Play scenes are relatively simple, and various methods tend to achieve similar performance levels in such environments. Additionally, our method shows improvements of 5.6% on the large-scale dataset Bridge which also demonstrates the effectiveness of our method. We have also achieved significant improvements over our own baseline, VidMan-GPT, across all four datasets, demonstrating the superiority of our model architecture. Details can be found in Tab. 7

5.3 Extra Ablation Studies

In this section, we conduct ablation studies for VidMan to answer the following questions: 1) Can co-training obtain a similar performance to our two-stage training? 2) Whether pretrained with general video data on the internet help improve the performance? 3) How much performance is gained by using the layer-wise self-attention block? 4) Does increasing the frame sampling interval positively impact action prediction? In the following, we answer each of these questions.

The importance of two-stage training. In our second stage, we only use action as a supervision signal to learn an implicit inverse dynamics model P ( a t | s t , s t +1 ) in Equ. (1). In other words, it suggests that we should not co-train with both actions and frames generation at the same time, otherwise a joint distribution P ( a t , s t +1 | s t ) is learning. To validate this viewpoint, we conducted experiments where the frames generation loss L v is also used in the second stage along with L a . The result of this variant is presented as "co-train" in Tab. 2a. In comparison to using only L a in the second stage, namely "action-only" in the table, the average score of "co-train" degrades severely and approaches the performance of baseline methods in Tab. 2a. This further supports the effectiveness of our two-stage training strategy.

Pretrained with general video data. To evaluate whether pretraining with general video data (not specific to robotics) from the internet helps improve performance, we conducted two control experiments. 'w/o Pre-train' refers to our model without video pretraining, 'w/ Ego4d' represents pretraining during the first stage using the general dataset Ego4d [13] for video prediction, and 'w/ OXE' refers to pretraining using the robot-specific dataset OXE. For the Ego4d dataset, we followed the same processing pipeline as GR-1. As shown in Tab. 2b, pretraining with robot-specific video data resulted in a 0.53 increase in the average task length on CALVIN compared to 'w/o Pre-train', while pretraining with general data ('w/ Ego4d') slightly decreased performance by 0.13. This suggests that training on web-based general data can yield marginal improvements, and pretraining in robotics domains significantly boosts performance.

Effect of layer-wise adapter. To measure the effect of the layer-wise adapter, we removed it (w/o adapter) and directly extracted the observed information into the action policy head by concatenating learnable action token to observation tokens before Open-Sora blocks. We found that this significantly reduced performance, comparing row 1 and row 3 in Tab. 2c. Additionally, we found that training only the layer-wise adapter while keeping the Open-Sora blocks fixed leads to faster convergence, as shown in Fig. 4. The training loss of the 'freeze" curve converges faster than the 'not freeze" curve. If both the adapter and Open-Sora blocks are trained, better accuracy can be achieved, by comparing row 2 and row 3 in Tab. 2c. In this sense, we can quickly adapt to a specific robotic scenario with the adapter being trained. In brief, layer-

wise adapter can provide good performance. If Open-Sora blocks' parameters are not fixed, better performance can be achieved; if the parameters are fixed, faster convergence can be obtained.

Effect of frame sampling interval. We compare the effectiveness of different frame sampling intervals ( i.e. 1, 2, 3, and 4) on Bridge Dataset and CALVIN. With a larger frame interval, a longer historical information our model can perceive and a longer future information needed to be predicted. The total length of the four experimental frames is 4, including 2 historical frames and 2 future frames. Results are shown in Tab. 3. It is observed that increasing the intervals from 1 to 3 improves performance on Bridge and CALVIN. The potential reason is that consecutive frames are very similar, and predicting frames that are farther away from the current step helps the robot gain a better understanding of the future. However, when the interval increases from 3 to 4, this improvement saturates quickly. We speculate that predicting frames too far into the future may not offer effective guidance for immediate local action prediction.

6 Conclusion

In this paper, we propose VidMan, a novel framework utilizing video diffusion models for robot imitation learning, which addresses the limitations of current GPT-style paradigms in real-time applications. By combining a Dynamics-aware Visionary Stage, which develops a deep understanding of environment dynamics through pre-training on the Open X-Embodiment dataset, with a Dynamicsmodulated Action Stage that efficiently integrates this knowledge into action prediction, VidMan achieves both high precision and computational efficiency. This two-stage approach, ensures robust and rapid action generation, significantly improving performance on benchmarks like CALVIN and the OXE dataset. In the future, we will expand VidMan to be able to perceive more dimensions of information.

Acknowledgements This work was supported in part by National Science and Technology Major Project (2020AAA0109704), National Science Foundation of China Grant No. 62476293, National Key Research and Development Program (Grant 2022YFE0112500), National Natural Science Foundation of China (Grants 62101607), Guangdong Outstanding Youth Fund (Grant No. 2021B1515020061), Shenzhen Science and Technology Program (Grant No. GJHZ20220913142600001), Nansha Key RD Program under Grant No.2022ZD014, The Major Key Project of PCL (No. PCL2024A04, No. PCL2023AS203).

References

Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, and Jianxiong Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 1386-1383. IEEE, 2017.

Menglong Zhu, Konstantinos G Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang, Cody Phillips, Matthieu Lecce, and Kostas Daniilidis. Single image 3d object detection and pose estimation for grasping. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 3936-3943. IEEE, 2014.

Ester Martinez-Martin and Angel P Del Pobil. Vision for robust robot manipulation. Sensors , 19(7):1648, 2019.

Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. The International Journal of Robotics Research , 41(7):690-705, 2022.

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning , pages 651-673. PMLR, 2018.

Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manipulation dataset. 2024.

Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. https://octo-models.github.io , 2023.

Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864 , 2023.

Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139 , 2023.

Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. Large-scale actionless video pre-training via discrete diffusion for efficient policy learning. arXiv preprint arXiv:2402.14407 , 2024.

Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601 , 2022.

Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080 , 2023.

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18995-19012, 2022.

Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for languageconditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters , 7(3):7327-7334, 2022.

Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956 , 2019.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning , pages 1094-1100. PMLR, 2020.

Philip M Groves and Richard F Thompson. Habituation: a dual-process theory. Psychological review , 77 (5):419, 1970.

Jonathan St BT Evans. Dual process theory: Perspectives and problems. Dual process theory 2.0 , pages 137-155, 2017.

Bertram Gawronski and Laura A Creighton. Dual process theories 14. The Oxford handbook of social cognition , page 282, 2013.

Keith E Stanovich. Who is rational?: Studies of individual differences in reasoning . Psychology Press, 1999.

Zangwei Zheng, Xiangyu Peng, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. URL https://github.com/hpcaitech/Open-Sora .

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716-23736, 2022.

Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters , 5(2):3019-3026, 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378 , 2023.

Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems , 36, 2024.

Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning , pages 991-1002. PMLR, 2022.

Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth International Conference on Machine Learning , 2023.

Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Zequn Jie, Lin Ma, Guangrun Wang, and Xiaodan Liang. Making large language models better planners with reasoning-decision alignment. arXiv preprint arXiv:2408.13890 , 2024.

Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648 , 2020.

Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters , 7(4):11205-11212, 2022.

Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.10639 , 2023.

Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation , 2024.

Tsung-Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402.10885 , 2024.

Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652 , 2024.

Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973 , 2023.

Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet-1billion: A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11444-11453, 2020.

Yang Cao, Yihan Zeng, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS , 2023.

Yang Cao, Yihan Zeng, Hang Xu, and Dan Xu. Collaborative novel object discovery and box-guided cross-modal alignment for open-vocabulary 3d object detection. arXiv preprint arXiv:2406.00830 , 2024.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 , 2023.

Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems , 35:9125-9138, 2022.

Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters , 2023.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 , 2023.

Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robocat: A self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306.11706 , 2023.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022.

Vikash Kumar, Rutav Shah, Gaoyue Zhou, Vincent Moens, Vittorio Caggiano, Abhishek Gupta, and Aravind Rajeswaran. Robohive: A unified framework for robot learning. Advances in Neural Information Processing Systems , 36, 2024.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/ 20-074.html .

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 , 2023.

Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402.19479 , 2024.

Alexandros Stergiou and Ronald Poppe. Adapool: Exponential adaptive pooling for information-retaining downsampling. 2021.

Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXiv:2305.10874 , 2023.

Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016.

Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL) , 2023.

Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task agnostic offline reinforcement learning. 2022.

Jianlan Luo, Charles Xu, Xinyang Geng, Gilbert Feng, Kuan Fang, Liam Tan, Stefan Schaal, and Sergey Levine. Multi-stage cable routing through hierarchical imitation learning. arXiv pre-print , 2023. URL https://arxiv.org/abs/2307.08927 .

Lawrence Yunliang Chen, Simeon Adebola, and Ken Goldberg. Berkeley UR5 demonstration dataset. https://sites.google.com/view/berkeley-ur5/home.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12104-12113, 2022.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning , pages 785-799. PMLR, 2023.

Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning , pages 694-710. PMLR, 2023.

Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. In 7th Annual Conference on Robot Learning , 2023.

A Appendix

The outline of the Appendix is as follows:

Negative impacts and limitations.

Details of the network and training process.

Description and usage guide for OXE.

Additional experimental results.

Visualizations of video prediction and action prediction.

A.1 Negative Impacts and Limitations

Potential Negative Social Impact. Our method has no ethical risk on dataset usage and privacy violation since all the benchmarks are publicly available and transparent.

Limitations and Future Works. Our current model operates solely on 2D vision and does not possess 3D perception capabilities. This limitation affects its performance on tasks that demand accurate 3D spatial understanding. Our future work will prioritize the integration of 3D perception into the model. Our model's capacity to comprehend complex human instructions is limited. It currently lacks the sophistication of a CLIP's language encoder. We are considering state-of-the-art Large Language Models (LLMs), such as LLama [61], as a promising avenue for future exploration to enhance these capabilities. The model's perception is not fine-grained. It processes the entire image as a single input for action prediction. We believe that incorporating fine-grained auxiliary inputs, such as object-bounding boxes or masks, could significantly enhance the model's performance. We are exploring ways to integrate such detailed inputs to refine the model's perceptual abilities.

A.2 Network and Training Details

Model details. For the video autoencoder, we utilized Stabilityai's VideoAutoencoderKL [49] to encode the video into embeddings. For the text encoder, we employed CLIP's text encoder [62]. For the diffusion model, we used STDiT-XL/2 [21].

Training details. The video diffusion transformer of VidMan contains 12 layers (we reduced the original number of layers in Open-Sora to 12) and 16 heads with a hidden size of 1152. While training on OXE for video prediction in stage 1, we used 16 Nvidia V100 32GB GPUs, with a batch size of 24 per GPU resulting in a total batch size of 384. This stage will cost 42 hours. While training on OXE in stage 2, we used 16 Nvidia V100 32GB GPUs, with a batch size of 18 per GPU. We employed gradient accumulation, updating the parameters every 5 steps, resulting in a total batch size of 1440. This process took 32 hours. While training on CALVIN in stage 2, we used 8 Nvidia V100 32GB GPUs with a total batch size of 400. We use the AdamW [63] optimizer with an inverse square root decay learning rate schedule [64], with weight decay of 0.1. Training hyperparameters are shown in Tab. 4

Open-Sora

block

outputs

Self attention

FFW

𝑲

𝑸

+

+

Learned

action

queries

State

tokens

𝑽

Layer-wise adapter. As shown in Fig. 5, following [22], our model employs a layer-wise self attention block, which comprises a self-attention layer and a feed-forward network (FFN). This block incorporates a gating mechanism to refine the information extracted by the VDT block into the action query.

A.3 More Details of Open X-Embodiment Datasets

We trained VidMan using a curated selection from the Open X-Embodiment Dataset [8], which is a rich repository of robot learning datasets. Our training data encompasses a variety of robot forms, environments, and tasks, reflecting a broad spectrum of robot types, sensor configurations, and labeling styles, including the presence or absence of language instructions.

Inspired by the approach taken in Octo [7], we began by filtering out datasets that did not include image streams or those that did not utilize delta end-effector control. From the remaining datasets, we carefully selected those with the highest diversity and task relevance, omitting any that were too repetitive, had low-resolution images, or were too specialized.

We then divided the chosen datasets into two groups: 'more diverse" and 'less diverse", based on the variety of tasks and environments they represented. To ensure a balanced training, we assigned a higher weight to the 'more diverse" datasets. Additionally, we adjusted the weight of some large datasets that had an abundance of data points to maintain a balanced mix.

To streamline the training process, we filled in any gaps in the camera channels with zero-padding and standardized the gripper action spaces across all datasets. This standardization meant that a gripper command of +1 consistently signified 'the gripper is open", and 0 signified 'the gripper is closed".

A.4 Additional Experimental Results

Why use pure noise in stage 2. In the second stage, a key issue is how to utilize the video prediction model pre-trained in the first stage for action prediction. We replaced the noisy images from the first stage with zero embeddings, pure noise, and images without noise, and then concatenated them with historical frames along the channel dimension as input to the VDT. All experiments were trained and tested for offline validation on the Bridge dataset. These experiments used a small batch size of 192 and were trained for 100k iterations with VDT block frozen. The results are shown in Tab. 5. We found that using pure noise during training allows the model to achieve a certain level of performance during evaluation. One possible reason is that using pure noise aligns with the denoising process, effectively distilling the pre-trained model's future perception capabilities into action prediction.

Effect of historical frames v.s. future frames. To explore whether the length of historical frames or future frames has a greater impact on action prediction performance, we conducted analytical experiments with varying lengths of historical and future frames. We experimented with different settings: 1) history frame n=1, future frame m=1; 2) n=2, m=1; n=2, m=2. We kept the length of historical and future frames consistent across both the first and second training stages. The frame interval was set to 3. Results are shown in Tab. 6. Comparing row 1 and row 2, if the length of historical frames is shortened, there will be a significant decrease in performance on both Bridge and CALVIN. Comparing row 2 and row 3, shortening the length of future frames also leads to a decrease in performance, but not as significant as with the historical frames.

Detailed numerical metrics on OXE data. The numerical metrics in Fig. 3 are displayed in Tab. 7.

Results on RLBench. We also conducted experiments on RLBench [23]. We used 18 tasks from RLBench for multi-task robot learning evaluation. Each task includes 2-60 variations. For example, in the stack blocks task, 'stack 2 red blocks" and 'stack 4 purple blocks" are considered as two variants. During testing, our model has to handle novel object poses, randomly sampled goals, and randomly sampled scenes with different semantic instantiations of object colors, shapes, sizes, and categories. Each multi-task agent is evaluated independently on all of the tasks. Evaluations are scored either 0 for failures or 100 for complete successes. We report average success rates on 25 evaluation episodes per task for agents trained with 100 demonstrations. On the RLBench benchmark,

we compared our method with 3 methods: PerAct [65], RVT [66], Act3D [67]. To align with methods like RVT, we also reported the variance of 5 seeds in the evaluation on RLBench.

A.5 Visualization

Visualization of our video prediction. We investigate the video prediction performance of VidMan trained on OXE in stage 1. Qualitative results are shown in Fig. 6. It can be observed that VidMan correctly predicts future frames, although some details (such as occluded objects) are missing. This predictive capability in the second stage can strongly guide action prediction as potential future forecasts.

Visualization of our offline action prediction. As shown in Fig. 7, we visualized VidMan's results on OXE offline evaluation. It can be observed that our model effectively fits the ground truth actions across various scenarios, potentially demonstrating its applicability to real robot tasks.

50

100

150

200

0.004

0.002

0.006

0.008

0.000

0.008

0.006

0.004

0.002

0

5

Time in one episode

10

15

0.0025

S(00'0-

0.0050

S0000

00000

0.0050

0.0025

250

5

Time in cne episode

10

200

15

0.005

-0.010

0.010

0.005

0.000

0.015

0

400

5

Time in one episode

10

000

15

0.05

ot'o-

ot'0

0.05

0.00

800

Time in one episode (subsampled)

5

Time in one episode

10

15

1000

0.025

0.050

0.125

0.100

0.050

0.025

0.000

0.075

0

1200

5

Time in one episode

pitch

10

15

1400

0.02

0.04

0.06

0.06

0.04

0.02

0.00

0

5

Time in ene episode

1600

roll

10

15

0.8

0.6

0.4

0.2

0.0

1.0

0

onug pune6

5

Time in eme egisode

grasp

10

15

50

100

150

200

250

D001

DOS1

2000

2500

0000

3500

Time in one episode (subsampled)

-0.010

0.005

0.010

0.005

0.000

10

Time in one episode

20

30

40

0.02

0.04

0.04

0.02

0.00

10

Time in ene episode

30

40

9.01

0.02

0.01

0.03

0.02

0.00

10

Time In one episode

20

30

40

0.05

0.15

0.10

0.00

0.05

10

Time in one episode

20

30

40

0.02

0.04

0.06

0.08

0.06

0.04

0.02

0.00

0

10

Time in one episode

20

pitch

30

40

-0.04

0.06

0.06

0.02

0.10

0.06

0.04

0.02

0.00

10

Time in one:

20

roll

episode

30

40

0.8

0.6

0.4

0.2

0.0

0

predicted action

grundtrue

10

Time In one episode

20

grasp

30

40

50

100

150

200

0.01

0.02

0.03

200

0.01

0.00

250

Q

5

Time in ane episode

19

15

20

25

0.01

0.04

0.03

0.02

0.01

0.00

Time in ose episode:

10

M00

15

20

25

0.01

0.02

0.03

0.02

0.01

0.00

5

Time in one episode

10

15

20

1000

25

0.025

0.050

0.100

0.075

0.050

0.025

0.000

0.125

Time in one episode (subsampled)

Tme in one episode

10

15

20

25

5200-

0.050

0.100

0.050

0.025

0.000

1500

Time in one episode

10

pitch

15

20

25

0.10

0.15

0.20

0.25

0.05

0.10

0.00

50'0

5

Time in ane episode

10

2000

15

20

25

9.2

0.6

0.4

0.0

1.0

8'0

3

Tirte in ose episode

10

grasp

15

predicted action

ground truth

20

2500

25

50

100

150

200

0.005

0.010

0.015

0.020

0.010

0.005

0.000

0

250

Time in ose episede

10

15

20

25

0.01

0.04

0.02

0.01

0.03

0.00

0

5

Time in one episede

10

15

500

20

25

0.02

0.03

0.01

0.04

0.03

0.02

0.01

0.00

0

Time in one episode

10

15

20

25

0.02

0.04

0.06

0.06

0.04

0.02

0.00

0.08

1000

Time in one egisode (subsampled)

ot

15

20

25

0.025

-0.050

5200-

0.100

0.050

0.025

0.000

5

Time in ene episode

1500

10

pitch

15

20

25

0.05

0.10

0.10

0.00

0.05

5

Time in one episede

10

roll

15

20

2000

25

1.0

0.8

0.6

0.4

0.2

0.0

5

Time in one episode

10

grasp

15

predicted action

ground trth

20

25

150

200

250

1000

Time in one episode (subsampled)

1500

2000

2500

0.025

0.03

0.03

0.04

0.04

pitch

0.20

TOlI

1.0

grasp

0.005

0.020

0.015

0.010

0.005

0.000

Q

5

Time in ere episode

10

20

25

30

0.01

0.02

0.01

000

Time in ene episode

10

15

25

30

0.01

0.02

0.01

0.00

Time in one rpisode

10

15

20

25

30

0.02

0.01

0.03

0.04

0.03

0.01

0.00

200

Time in ene rpitooe

10

15

25

0.02

-0.04

0.06

0.02

0.00

Time in are episode

10

15

20

25

30

0.05

st'o

0.10

0.05

0.00

Q

Time in one episode

10

15

25

30

0.6

0.4

0.2

0.0

0.8

0

5

Time in ane episode

10

15

20

predicted action

groutd truth

25

30

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We believe that the main claims reflect the areas we are targeting: robot manipulation and imitation learning.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not have theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 5 and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We already include the data and instructions in Section 5 and Appendix and the code will be released upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 5 for more details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our experiments are quite stable with multiple runs. All results are observed via a fixed seed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix for more details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research is conducted with the NeurIPS code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the creators of all existing assets. All the used existing assets are open-source.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission explores video pre-training for manipulation. Specifically, it trains a language-conditioned video prediction model to imagine future frames for the given task and an inverse dynamics diffusion model to predict actions, conditioned on the imagined future. The paper’s idea is that the first model can act as a slow process that thinks what will be done in the scene, while the second model is a fast action model, borrowing terminology from the dual process theory.\n\nThe experiments show multi-task manipulation results on RLBench and offline results on a subset of OXE datasets, while the paper also discusses interesting ablations.\n\nSTRENGTHS:\n1. The paper is clearly written and easy to follow in general.\n\n2. The ablations show some interesting results, surprisingly negative but still valuable. Most notably, it seems that training the video model on web data doesn’t really add anything to the performance. This is good to know and it contrasts findings from recent papers such as [GR-1], which claim that video pre-training on non-robotic datasets can be helpful. Another interesting finding is that the second model better predicts actions only, rather than both actions and video frames. This also contrasts [GR-1].\n\n3. As a technical contribution, using adapters to learn the inverse dynamics model on top of the weights of the video prediction model is an interesting idea.\n\nReferences:\n\n[GR-1] Unleashing large-scale video generative pre-training for visual robot manipulation\n\nWEAKNESSES:\n1. The major problem with this submission is the evaluation. Specifically, the results are very weak and unsafe for conclusions.\n\na. On RLBench, the paper seems to modify slightly the standard multi-task setup introduced in [PerAct] by changing the number of cameras used. Another possible difference is the number of seeds used. [PerAct] and other competitors [3DDA, Act3D, RVT, etc] use 5 seeds. I could not find this detail in the submission but I assume one seed was used, since there is no reported variance. Changing the setup doesn’t allow for direct comparison with prior works. As a result, the authors probably had to retrain the baselines.\nWhile modifying the setup and retraining the baselines is not an issue per se, the selection of the baselines is highly disputable. [PerAct] is an old baseline published two years ago. Since then, [Act3D, RVT] achieved big leaps in 2023 and the current SOTA on RLBench is [3DDA]. Claiming SOTA on RLBench without comparing against actual SOTA models is wrong. In fact, with one camera only [3DDA] seems to achieve better results than the proposed approach.\n\nb. Another controversy is the use of web data and OXE as pre-training. I mentioned in the strengths that revealing that web data don’t help is actually useful. The submission says “training on web data can help improve marginal performance” (line 313), which, while moderate, is actually an understatement; it’s better to say that it doesn’t really help. It would also be interesting to evaluate the usefulness of OXE pretraining; is training on RLBench enough for the model’s RLBench performance?\nThe above are important questions also since the baselines do not use any type of pre-training (save for the ablation VidMan-GPT). If the proposed approach uses many more data and still cannot compare against the baselines [3DDA, RVT, Act3D], then maybe this two-stage video pre-training is not the right direction. This would also be valuable to know.\n\nc. The main competitor of this submission is [GR-1]: they also propose a two-stage video pre-training scheme, albeit with autoregressive models (instead of diffusion), trained on Ego4D (instead of OXE + web data) and training for both frame and action prediction at the second stage (instead of only actions). The submission acknowledges that [GR-1] is the most similar work (line 89). On that aspect, a fair and direct comparison against [GR-1] is required so that we know which design choices are indeed the best among the two, i.e., diffusion vs autoregressive, Ego4D vs OXE++, joint frame-action prediction vs action only. I appreciate the effort of putting together VidMan-GPT as a proxy for [GR-1] but it is not the same as the original model. Thus, it is unsafe to conclude among the two models. Instead, I would recommend that the submitted model is evaluated and ablated on CALVIN.\n\n2. Another issue of the submission is its motivation. Most of the paper (for example the whole intro section) focuses on motivating the co-training with videos. But this motivation is somewhat generic, meaning that [GR-1] could also have the same motivation. Given that the submission acknowledges [GR-1] as closely related, what is the motivation beyond what supports [GR-1]? Borrowing from my previous paragraph, why do we need diffusion models for world modeling or any new video co-training model in general? What’s lacking from previous related approaches? Such targeted motivation would lead to stronger arguments for the proposed approach – if of course those can be validated through the experiments.\n\n3. A last concern regarding motivation/presentation is the connection to dual process theory. From my knowledge about fast and slow thinking, the submission’s description of system 2 as “slow planning” is indeed fair. I can see why the manuscript argues that language-conditioned future prediction is a type of slow processing, since the agent judges about the changes it wants/expects to achieve and then executes. However, the description of system 2 in the related literature also refers to reasoning. I’m not exactly sure that planning through a black-box video diffusion model can really count as system 2 reasoning, since the model predicts pixels end-to-end and does not model the causal factors of variation a robot can affect, e.g. motion, pose difference. I would be reluctant to characterize the proposed approach as inspired by the dual process theory. I think it’s more inspired by representation learning and goal-conditioned policy learning [UniPi, SuSIE].\n\n\n\nReferences:\n\n[PerAct] Perceiver-actor: A multi-task transformer for robotic manipulation\n\n[Act3D] Act3d: Infinite resolution action detection transformer for robotic manipulation\n\n[RVT] Rvt: Robotic view transformer for 3d object manipulation\n\n[3DDA] 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations\n\n[GR-1] Unleashing large-scale video generative pre-training for visual robot manipulation\n\n[SuSIE] Zero-shot robotic manipulation with pretrained image-editing diffusion models\n\n[UniPi] Learning Universal Policies via Text-Guided Video Generation\n\nQUESTIONS:\nOverall, the submission explores an interesting topic and I’d like to endorse the honesty of testing on RLBench, where 2D polices (like the proposed one) struggle, as well as not wasting time with real-world experiments. However, the experimental design is severely lacking, if not flawed. The baselines used are obsolete and the proposed method uses a lot of external data (which is good) but still fails to compete against strong baselines that are trained in-domain. In addition, a lot of discussion on and a direct comparison with [GR-1] is needed, as well as re-motivation based on these findings. The current version may lead to confusing or misleading conclusions due to lacking evidence. I encourage the addressing of all the issues.\n\nTo increase my score I'd like to see:\n\n1. Direct comparison with GR-1 on the same dataset.\n\n2. Discussion on the relation of this work to GR-1 and \"Learning Universal Policies via Text-Guided Video Generation\". The outcome of this should better motivate the approach, not as general two-step video-guided manipulation but on the importance of the specific contributions of the paper.\n\n2. Inclusion of modern baselines on RLBench.\n\n\nMinor: In total, the manuscript is easy to follow but there are grammatical or syntactical mistakes here and there, which could be easily fixed using online tools, if not careful proofreading.\n\n\n--------------------------------------------\nAfter the rebuttal, I'm increasing my score to weak accept.\n\nLIMITATIONS:\nAddressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework that utilizes a pre-trained video diffusion model to learn dynamic information and then adapt it to output actions for manipulation tasks by incorporating an action-aware adapter. In the second stage, this framework can transfer the learned dynamic knowledge for action learning in a single denoising step.\n\nSTRENGTHS:\nThe idea presented in this paper, which aims to enhance decision-making through video pre-training, is not new in the fields of RL and robotics. However, I believe this topic is compelling and this paper could inspire further research in this direction.\n- This paper is well-written, and the presentation is clear.\n- This paper incorporates a layer-wise adapter to utilize the pre-trained 1-stage parameters for action learning, which is common in VLM but novel in the robotics area.\n- Success rates on 15 tasks in RLBench and prediction accuracies on OXE datasets demonstrate the efficacy of the proposed method\n\nWEAKNESSES:\n- Although the proposed adapter is validated to be effective in the experiments, this paper lacks more evidence to demonstrate its superior performance compared with other approaches. For example, can we use a projector after the open-sora outputs instead of layer-wise adapters? Can we utilize the hidden representations of predicted videos like [1]?\n- While I appreciate the authors' efforts in proposing a new video-based decision-making system and achieving satisfactory performance, I find the design of Stage 2 to be lacking in interpretability (If I am mistaken, please correct me). Stage 2 does not appear to sufficiently utilize the knowledge learned in Stage 1. The paper employs an adapter to cross-attend with the encoded features of historical images in open-sora, which can be viewed as extracting a latent representation from historical data, especially since the open-sora layers are also trained. Learning a latent representation can be accomplished in various ways, not limited to using an adapter. As a result, comparison with some representation learning baselines (e.g., R3M) is encouraged. Therefore, I suggest the authors explain what useful information is contained in these features and why cross-attention with these features is sufficient to distill the knowledge learned in Stage 1.\n\n[1] Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning, 2024\n\nQUESTIONS:\n- The authors claim to reduce the open-sora model to 12 layers with 80M parameters. How is the 12-layer model initialized with the pretrained 700M weights? Do the authors simply select 12 layers and load the original weights?\n\nLIMITATIONS:\nThis paper has discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for video diffusion in robot manipulation (VidMan), which contains two stages. In the first stage, VidMan adopts large-scale Open-X datasets for pertaining an Open-Sora-like architecture for video prediction. In the second stage, an adapter is introduced to learn the inverse dynamics for policy learning. The whole framework can be used for robot manipulation tasks such as RLBench.\n\nSTRENGTHS:\n1. The whole framework adopts advanced techniques like video diffusion, open-sora, and adapter, which are popular for LLM/VLM-style research. Such a framework can motivate related research on introducing LLM/VLM architecture in robot manipulations.\n\n2. The framework is motivated by dual process theory, which is reasonable.\n\nWEAKNESSES:\n1. The whole framework is very similar to small-scale diffusion planners like decision-diffuser, i.e., trajectory diffusion for planning and inverse dynamics for action prediction. Indeed, VidMan has unique contributions for scaling, but the basic concepts are similar to previous methods. \n\n2. The existing framework requires high-quality expert trajectories in pertaining. If the non-optimal data is contained in the dataset, the video diffusion model may generate sub-optimal trajectories, and the resulting actions may also be non-expert. Can you discuss more about how to extend such a framework to sub-optimal datasets. \n\n3. Some empirical results are not convincing. Please refer to the questions.\n\nQUESTIONS:\n1. Is the video tokenizer directly adopted from a pre-trained model? It may be better to fine-tune it with Open-X robot data.\n\n2. In Table 1, I find that PerAct's results are lower than those of the original paper; can you explain this?\n\n3. RVT is a multi-view benchmark that often requires multi-view images as input. Can VidMan generate multi-view images? Are the results in Table 1 learned by single-view images?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method called Video Diffusion for Robot Manipulation (VidMan) that uses a two-stage training mechanism. In the first stage, the VidMan model is trained to perform future state prediction (i.e. dynamics learning) on a large diverse dataset (Open X-Embodiment). In the second stage, the model is finetuned using a cross-attention adaptation layer to instead learn an inverse dynamics model. The method is evaluated on RLBench tasks, outperforming baselines, and on offline OXE data. The authors further provide ablation experiments to illustrate the effect of each of the components of the method.\n\nSTRENGTHS:\n- The work addresses an important and timely problem, which is how to leverage large-scale data to learn dynamics, and then propagate that information to downstream policy learning.\n- The empirical results of the method seem strong, in particular on the RLBench tasks. \n- Ablations demonstrate that training the second stage on an action-only objective significantly improves performance compared to co-training.\n\nWEAKNESSES:\n- I think the work could better highlight the validation of the central hypothesis of the work that the two-stage training process is actually more effective than a single stage training process (just learning the inverse dynamics model). Although the authors provide a baseline that compares the performance when using a co-training objective in stage 2 of training, if I understand correctly, the ablation that does not perform stage 1 of training at all and directly learns an inverse model is the one found in Table 2(b) “w/o domain train”. I think as this is one of the main points that the work is trying to argue for, this should be highlighted to a greater degree. Directly comparing the entire VidMan system to models like Octo or PerAct can’t really validate this hypothesis because those methods are so different. Additional experiments to test this on the offline evaluation setting would also help.\n- The baselines selected for the RLBench task seem somewhat arbitrary – For instance, robotic view transformer (RVT) by Goyal et al. in 2023. significantly improves upon PerAct in the RLBench task suites, and RVTv2 improves upon that further. \n- The manuscript has several grammatical errors that do not interfere with the reader’s understanding of the work (e.g. L292 “Whether pretrained with general video data on the internet help improve the performance”), but I suggest the authors carefully revise the manuscript. \n- The connection to the two-process model for cognitive systems seems rather tangential, and I would recommend the authors to omit it in favor of more grounded discussion about the concrete implementations of the training stages.\n\nQUESTIONS:\n- Can the authors clarify what the term “intrinsic dynamics” means? \n- In L284, “This improvement is attributed to our approach’s ability to utilize the training data more efficiently by implicitly leveraging the inductive bias of inverse dynamics”, should this be “forward dynamics” instead of “inverse dynamics”?\n\nLIMITATIONS:\nThe authors give a brief description of the limitations, which is rather terse. I would appreciate it if the authors could address additional limitations of their work from various aspects.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VidMan, a two-stage framework for robot manipulation using a pre-trained video diffusion model. The first stage involves training on an actionless video dataset to predict future visual trajectories using denoising diffusion. The second stage adapts this model into an inverse dynamics model, predicting actions from the learned dynamics using a smaller and faster diffusion-based action head to learn an action distribution implicitly conditioned on state transitions through the video prediction model. VidMan outperforms multiple strong baselines in a variety of environments, showing impressive generalization capabilities and good overall performance.\n\nSTRENGTHS:\n**Novel Approach:** The two-stage training approach is a deep insight that may have a large impact on the robot learning community. Separating the modeling of environment dynamics from action prediction may lead to further advancements in behavior cloning.\n\n**State-of-the-Art Performance:** VidMan achieves substantial improvements over strong baselines, particularly in scenarios with limited data.\n\n**Thorough Evaluation:** Comprehensive experiments on multiple benchmarks (RLBench, OXE) validate the performance of the method, and the approach is also supported by extensive ablation studies.\n\nWEAKNESSES:\n**Lack of Comparison to SOTA Baselines:** The paper lacks comparisons to state-of-the-art baselines like Robotic View Transformer (RVT), a direct follow-up to PerAct that presented competitive results on the same set of RLBench tasks chosen for this work. I recommend that this work be cited and included in the baselines. The work can be found here: https://arxiv.org/abs/2306.14896\n\n**Qualitative Results:** While the quantitative results are thorough, the paper could benefit from more qualitative results (e.g. videos or a website) to illustrate the model's performance and capabilities in dynamic environments. The videos provided in the supplementary were of high quality, but it is difficult to tell whether the few provided videos are representative of the policy's performance. Additionally, the provided video prediction results in Figure 7 within the appendix primarily show static scenes. I would prefer results with more complex dynamics to be convinced that the video prediction model is performant.\n\n**Scalability Concerns:** The performance improvement over baselines seems to diminish as the number of demonstrations increases. This raises the concern that the method may not continue to show increased performance at larger scales. Additionally, the authors show only marginal improvement from pre-training on internet-scale video data, suggesting that the method may not sufficiently utilize non-robot data.\n\n**Lack of Real-World Experiments** The authors provide no experiments with their policy running on a physical robot. Additionally, it is not shown whether the policy can run in real-time. One suggestion could be to evaluate policies trained on the OXE dataset in the SIMPLER environments (https://arxiv.org/abs/2405.05941). I understand that this is not a reasonable request for the purposes of your rebuttal, but it may be a good choice for the camera-ready version of this work.\n\nQUESTIONS:\n**Inference speed:** How quickly does the model run in online scenarios, and what are the computational resource requirements during deployment?\n\n**Missing tasks:** Why were specific tasks like \"insert peg,\" \"stack cups,\" and \"place cups\" omitted from the RLBench evaluation, when they are included in PerAct and RVT?\n\n**Missing baseline:** Why were RVT results not included, given that they have results for the same set of tasks and outperform PerAct? I will consider updating my rating if this baseline is included.\n\n**Intuition for Web Data Ablation:** Given that video pre-training with web-scale data only marginally improves performance (+0.8% in success rate), is this due to the domain gap between web data and simulation data, because the in-domain data closely aligns with the evaluation data, or a different reason?\n\n**Accuracy metric:** Can you precisely explain what the accuracy metric is measuring? Are the actions discretized and classified, or is there some error threshold that warrants a successful prediction? This should be explained in Section 5.1.2.\n\nLIMITATIONS:\nThe paper addresses some limitations, particularly in terms of scalability to 3D perception tasks, but further discussion is needed regarding the model's real-world deployment and computational efficiency", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}