Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training
Authors: Haoran He
Chenjia Bai
Ling Pan
Weinan Zhang
Bin Zhao
Xuelong Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Imitation Learning
Actionless Video Pre-Training
Video-Based Policy Learning
Discrete Diffusion
Primary area: reinforcement_learning
TL;DR: We develop a video-based multi-task policy using discrete diffusion, facilitating efficient policy learning by incorporating foresight from predicted videos.

Abstract: Learning a generalist embodied agent capable of completing multiple tasks poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. In contrast, a vast amount of human videos exist, capturing intricate tasks and interactions with the physical world. Promising prospects arise for utilizing actionless human videos for pre-training and transferring the knowledge to facilitate robot policy learning through limited robot demonstrations. However, it remains a challenge due to the domain gap between humans and robots. Moreover, it is difficult to extract useful information representing the dynamic world from human videos, because of its noisy and multimodal data structure. In this paper, we introduce a novel framework to tackle these challenges, which leverages a unified discrete diffusion to combine generative pre-training on human videos and policy fine-tuning on a small number of action-labeled robot videos. We start by compressing both human and robot videos into unified video tokens. In the pre-training stage, we employ a discrete diffusion model with a mask-and-replace diffusion strategy to predict future video tokens in the latent space. In the fine-tuning stage, we harness the imagined future videos to guide low-level action learning with a limited set of robot data. Experiments demonstrate that our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared to previous state-of-the-art approaches with superior performance.

Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training
Haoran He"
Chenjia Bai
Ling Pan
Weinan Zhang
Bin Zhao
Xuelong L
Hong Kong University of Science and Technology
Institute of Artificial Intelligence (TeleAI), China Telecom
Shanghai Jiao Tong University
*Shanghai Artificial Intelligence Laboratory
Abstract
Learning a generalist embodied agent capable of completing multiple tasks poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. contrast, vast amount human videos exist, capturing intricate tasks and inter- actions with the physical world. Promising prospects arise for utilizing actionless human videos for pre-training and transferring the knowledge to facilitate robot policy learning through limited robot demonstrations. However, it remains chal- lenge due to the domain gap between humans and robots. Morcover, it is difficult extract useful information representing the dynamic world from human videos, because of its noisy and multimodal data structure. In this paper, we introduce a novel framework to tackle these challenges, which leverages unified discrete dif- fusion to combine generative pre-training on human videos and policy fine-tuning small number of action-labeled robot videos. We start by compressing both human and robot videos into unified video tokens. In the pre-training stage, we employ discrete diffusion model with a mask-and-replace diffusion strategy predict future video tokens in the latent space. In the fine-tuning stage, we harness the imagined future videos guide low-level action learning with limited of robot data. Experiments demonstrate that our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared previous state-of-thee approaches with superior performance. Our project webpage is available /tttps://vidoo- difo. .gi titthu..00
Introduction
How do we derive general-purpose robot agent that can complete wide variety of tasks? We believe that recent advances vision and language give us a clue, which delves into pre-training foundation models on extremely large and diverse datasets, followed by fine-tuning on specific domains. For instance, through pre-training on internet-scale datasets (55 large language models
various downstream tasks such as question answering, coding, and image generation. However, unlike general visual language tasks that can exploit copious amounts of data available on the Internet, embodied tasks necessitate high-quality egocentric data roboties domains for precise control. Collecting such data can be expensive time-consuming due to the reliance robot interactions through teleoperation or kinematic solvers [31], and significant gaps in embodiments and dynamics persist when applying them to different robots.
81 82, 63
and vision models [9 69 72) showcase impressive performance
In contrast to the limited availability of robot data, there is a wealth of human interaction videos capturing intricate tasks and varied interactions with the physical world 27 These videos inherently
'Correspondence to: Chenjia Bai (baicc www.@echccanttccccm.c
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

encapsulate rich semantic information regarding objects, environmental backgrounds, and hand- object interactions across diverse scenarios, making them potentially valuable for acquiring shareable knowledge relevant to embodied tasks.
Motivated by this, many works have emerged learn various objectives pre-trained on human actionless videos, aiming capture useful knowledge that can be beneficial for embodied tasks. These approaches involve learning pre-trained image representations
Video
LurpoSeaki: learcction
Discrete Diffusion
Purtar Videos
Self Superised Pre
Videos
Vouai Eoeard Stalk
61 57 68 9I] trajectory representations 4
85 74
reward functions [13] [56] and world models [59
Rober Videes
However, they are still limited to comprehending the dynamic rules of the world or reasoning based on long-term behavior rather than relying solely on step-by-step transitions. We summarize three main
Asliow for Rebec Videos
Fine-tuning Segg?
(st Ray thy Mt)
Figure 1: Overall framework of VPDD.
challenges that bottleneck their performance: (i) The domain gap between humans and robots which hinders knowledge transfer; (ii) Complex, diverse and noisy behavior patterns hidden in human videos which are difficult to learn; (iii) Large-scale data from different modalities (e.g.. videos, actions, texts) which requires a scalable and high-expressive model architecture to process.
To address these challenges, we propose Video-based Policy learning framework via Discrete Diffusion (VPDD). VPDD bridges the visual gap between the human and robot domains by represent- ing these two data types as unified latent representations. Then, VPDD performs video prediction as a pre-training stage with actionless videos, which acquires the commonsense knowledge shared be- tween human and robot interactions, including dynamic rules and behavior patterns (e.g., pick, place, push) to understand and complete tasks. Then, VPDD performs policy learning via a fine-tuning stage with action-labele robot videos, where VPDD learns to predict actions with foresight from future video predictions. The pre-training stage learns extensive knowledge from human video prediction, and the fine-tuning stage concentrates on training parameters specifically associated with actions. To tackle the challenge of modeling the noisy and complex distribution of large-scale videos while enabling the mulli-moda generation of both videos and actions. we leverage the generative capability and flexible architecture offered by discrete diffusion models 29 We provide an overview of our method in Fig. [
In summary, we highlight our contributions as follows. (i) We propose VPDD, a novel pretraining- finetuning paradigm for learning an actionable policy with limited robot data accessible. This paradigm demonstrates superior ability transferring valuable knowledge from large-scale actionless human videos to downstream embodied tasks. (ii) We formulate both video prediction and action learning processes as unified discrete denoising problems, showing the supreme effectiveness in handling high- digh-dimensional, multi-modis data. (iii We conduct thorough experiments using human videos from Ego4D [27], as well as embodied datasets from Meta- World (98] and RLBench [43), showcasing its ability to predict dynamic- consistent future videos. Our actionable discrete diffusion policy also exhibits superior performance compared to previous state-of-ther approaches 32 61
dynamic-consistent
24
75 15
encompassing both seen and unseen scenes for multi-task robotic problems.
2 Preliminaries
2.1 Multi-Task POMDP
In this work, we consider a generalist vision-based agent that is capable of addressing multi-task predicaments, where the landscape characterized by the inherent challenge of acquiring different skills across tasks and partial observability when dealing with image inputs. Given a specific task T ~ p(T), we further approach the problem as a task- specifiee Partially Observable Markov Decision Process (POMDP). defined as (ST O, PT RT /u, 7)) Here, O is a shared observation space as we use image observations for all tasks. We also assume all tasks share the same action space with the same embodiment.

---

2.2 Vector Quantized Model
In order to unify the feature space of both human videos and robot videos, we leverage the Vector Quantized Variational Auto Encoder (VQ-VAE) [83] compress sigh-dimensional data points into information-rice discretized latent codes. Given a high-dimensional video segment æ E RTTxHxWx the encoder F first converts it the temporal-spatial features E( (zm,id) Rixhxxxd where x x w represents the encoded sequence length and is much smaller than 7 H x W. Then transfer the continuous features into discrete space by performing nearest neighbors lookup in codebook of embeddings Z
{er}i== E R to obtain the tokens
2q = Quantize( (,m,) == arg min, ||=mji ej e,ll
where the video tokens 2n E IR xxxxxxx can be faithfully reconstructed via decoder, i.e., = G(z,). The encoder E, decoder G. and codebook 2 be trained end-to-end via the following loss function 1111 + |sg]| F?(=)) 2y112 + B||sg] F(x)) where sg denotes stop gradient.
= c
2.3 Discrete Diffusion Model
The discrete diffusion model was first proposed to deal with discrete state space with transitions converging to a binomial distribution (76). and then extended to multinomial diffusion with more options for transition matrices [36 2 In this work, utilize discrete diffusion with the absorbing state for sequence prediction of discrete tokens. Besides J tokens from a codebook, an additional [MASK] token is introduced. We denote 2; as one-hot vector identifying the token index. The forward process from Øk-1 to æy follows a Categorical distribution of QA as
q œ 2 x = Cat(<xip Q&TE-1) xx Q
(2)
where [Qk]ma g( m æk-1 n) E ((-+I)x(++1) is the Markov transition matrix from k I to k; which is formulated as:
a
9R
81
Ba
Q Rk---
8
9p.
+ 82
where ar [0, 1] the probability of retaining the token, and each ordinary token has a probability of 7t to be replaced by [MASK] token, leaving a chance of 3x (1 a 7x) be diffused.
Importantly, due to the property of the Markov chain, we can derive the probability of æy at arbitrary timestep directly from Øy
g( (20) = xx. Q, with Qs = = -.....Q -1.
(4)
Besides, the posterior of this diffusion process is tractable g( [m 1
( mo)
( 1== 1e In the reverse process, rather than explic-
( (26)
itly predicting the posterior through denoising neural network, the wo parameterissaionn enhances stability and allows for fast inference (by skipping Ák steps per iteration). The reverse transition with reparameterisation is formulated as
po (2x)) E' g( To) (2ol
#o
where the neural network predicts the logits of the target data g(xo).
3 Methodology
We commence with the pre-training of our model through future video prediction, enabling the learning of a general dynamic pattern across diverse domains. Subsequently, we fine-tune the model using a limited dataset of robot data for policy learning, leveraging foresight from predicted videos. Our framework is illustrated in Figure 2
3

---

Stage 1: Video Tokenizing
Stage 2 Pre- Pre-Trainingwitth MixedDaaa
Large Scale
MM.
C,MC
Human V Vidoos
Robot Videos
XK
X
X
9(x, (22-1)
Instructions
X
Prndicted Funure Videos
"Handle and spread cards"
close drawer"
CLIP
Conv3D Encoder
Historical
y
Po.
Vitloo TalroTokem Tokens
Pa
Action Tokens
Codebook
Quantize(
Percelver ID
GPT2 Transformer
Transformer
G 44 5
video tokens C
Slage Fine Fine-Tumi Robot Data
M M.M. M
M M.
M M.C..... MMM...
C,
C. C.Gper
X
X
Pa Iru
-
X
Conv3D Decoder
Profittd ctions
Instructions
CLIP
Po.
Dennise Tokens Tokens
ts
Denoise: PertinnTakmmn
(02 018.5)
Historical Frames
Perceiver
execute
Transformer
Transformer
Figure 2: Overall pipeline of VPDD. video-based VQ- (Q-V is leveraged to encode both human and robot videos into discrete latent codes. Subsequently, unified discrete diffusion is firstly pre pre-trained on these video latent codes via self-supervised objective, predicting future videos conditioning on language instructions and historical videos. The pre-trained video prediction model pó, can capture temporal dynamics and task-specific representations. Lastly, we fine-tune our diffusion model on limited number of robot data. In each diffusion step of the fine-tuning stage, leverage po, to provide hidden representations z to benefit downstream action learning with video foresight. This integration video prediction and action learning achieved through our unified discrete diffusion.
3.1 Data Preparing and Tokenizing
Robot Data Collection. We use the rule- based script policy to rollout 20 expert demonstrations for each task in Merl-World 198). We also run VPDD on 16 tasks from RLBench [43]. more challenging benchmark involving 6-Dof manipulation that necessitates multi-view images from a 3D scene select actions. Following the multi-view manipulation frameworks [24 75). we utilize the script motion-planner to collect 10 demonstrations for each task. Each demonstration from robot-data
formulated as т; (v1, 01;'' Ut, az,... PT, ar) with i (ot- 1 1.... 0t-1, where set / = 4 throughout the paper and denotes the actions. In the context of Meta- World, represents single-view RGB image at timestep t. For RLBench, o fofrone oleft. o,
right owristt comprises RGB multi-view images (i.e., front, left shoulder, right shoulder, and wrist). Consequently. in RLBench, 0r is formulated as Ve = foffont 2 vi right oprrist..
Human Data Collection. As for human data collection, we obtain untrimmed videos from the open-sourced Ego4D dataset (27. which contains massive-scale human-objes interactions of various durations ranging from 5 seconds to 7 hours. We filter out videos without human- -ojject interaction and segment each video into short clips with 8-frame intervals (60] Thus each video represented as {U1, 12, ... Va where denotes clip containing 4 frames. This approach yields total of 196,177 clips human videos, comprising approximately 4M frames. More details on data collection and processing are given in $C.1
VQ- VAE Encoding. To extract usefuul features from raw videos in both human and robot domains, conventional approach is to directly encode them into an embedding space using pre-trained vision models like VTT. However, these models are usually specifically trained on image dataset (7T). posing significant challenge due to the domain gap with our interaction videos. Thus, we leverage VQ-VAE compress the diverse and noisy videos into discrete latent codes, which provide a unified codebook for mixed videos and alleviate the domain gaps between human and robot videos. Formally, we adopt VQ-VAE architecture introduced by VideoGPT (94] for encoding videos into discrete latent space. The codebook comprises 2048 codes, each represented by 256-dimensional embeddings. The encoder architecture consists series of 3D convolutions that downsample by a factor of 4 over space-time (resulting in 64 total reduction), followed by 6 attention residual blocks. Consequently,

---

each video clip is E { is embedded into latent codes eg. The architecture for the decoder is the reverse of the encoder, featuring attention residual blocks followed by an equivalent number of 3D transposed convolutions for upsampling over space-time. The VQ-VAE is pre-trained on large-scale videos and remains fixed in the subsequent processes, providing flexibility for various downstream utilization methods.
Action Discretizing. For subsequent pre-training and fine-tuning, we process the collected continuous actions via uniform action discretization [46 M.. In the case of Korl-, the action space is 2-tuple consisting of the change in the 3D space of the end-effector followed by normalized torque that the gripper fingers should apply. Here all the continuous dimensions are discretized into 48 bins uniformly. Thus, the robot action can be represented using ordinals of the discrete bins as a 4 integer number. For RLBench, an action consists of the gripper open state and 6-DoF pose including position and rotation. The position discretized into 360 bins, and rotation is discretized into Euler angles as 1 1-degree bins for each of the 3 rotation axes 77 Gripper open state is a binary value.
3.2 Video Prediction via Unified Díscrete Diffusion
Extracting general patterns useful for downstream decision-making from large-scale in thelwild human videos is challenging, primarily because of the absence of labeled actions and the complexity of the underlying structure of human interactions. Different from previous ways of learning a visual representation, we propose novel objective to further unleash the representation and temporal modeling ability of diffusion models. Specifically, after obtaining discrete tokens from VQ-VAE encoding, train unified discrete diffusion model the latent space via self-supervised objective. This objective involves predicting future videos based on observed historical videos for both humans and robots, while masking action tokens. Benefiting from the proposed objective and the 2n paraaettteriitio of discrete diffusion, the diffusion model is incentivized to capture both the high-level temporal dynamics and the levelevel visual commonalities between historical and future videos at each diffusion step. Then the acquired knowledge can be leveraged to guide action denoising at each step.
Unified Transition Matrix. The presence of a transition matrix determines the nature of the discrete diffusion model (2). While the original discrete diffusion is limited to modality, drawing inspiration from UniD3 (37). which enhances the transition matrix to encompass both images and text, we construct unified transition matrix to capture global connections between the two modalities-vvieed and actions. The matrix [Q*]m, below illustrates the unified transition process:
96
36
8
D
0
0
0
:
9
A
Q+
0
Ouy + 82
G
0
7 7kk
73 74
where Br and B2 are the probabilities of a token to be replaced by any other accessible tokens in different modalities. The dimension of Qs is (J J 1) (J J +1,, where J and J are the number of tokens in different modalities, i.e., J is the size of codebook in VQ-VAE and J" is the number of action classes in discretization. The sum of each column in this transition matrix is one to preserve probability mass. Mathematically, we have 33g = (1 ag 7///J and 3 (1 ag - '²)/J*. All the mass of the stationary distribution falls on the [MASK] token, which satisfies the prerequisite for discrete diffusion model transition matrix 2. The details of the diffusion process are provided in IA.I
Unified Objective. We cast both video prediction and action learning as a conditional generative prob- lem, and the goal is to maximize E,wuir [log (=( (]) Here [x", I,",, where x" (e++h+1, , M++h+M] represents future video segments with M clips, and x² [a................ H-1 denotes action sequences of I steps. y = e, serves as the condition containing historical video tokens. 1 is the language instructions describing current tasks. In practice, we train two separate de- noising networks, namely pe, (x) [z.. y. l) and po, (af-1 Øk: 2 2), to learn videos and actions, respectively. Here, 2ag represents the hidden representation of predicted future videos given by pe, at each diffusion step, which is utilized to guide action learning. Formally, क = Softmax Pettmax(MMP((( (MLP(zag)
5

---

As the actions are absent during the pre- vre-trainng stage, we mask x² and freeze p8,, as illustrated in The network is trained to minimize the following variational lower bound (VLB) 76 29
Figure 2
Libb Lo +
- k= CK-1 + LK,
(6)
where
Co
Roy (ee) [logpo, (arg) 21 y: 1) log po, ( (rr (r zny. 1]]
(7)
Ck-1- Bylarlan) (m)) (DKL ((<x (22, xo)
[pe] - y, pe, (क 20k Ray. ()))
(8)
CK = Eg(=o) [DLL (9(2* (z)) p(zx))]
(9)
Lx is constant number that can be ignored in the training, the prior distribution p(æk) is fixed:
p(æx) = (BRKBKK.. ... 3k. 8K 7k]]
(10)
Model Architecture. As in Eq. (8. the neural network ps, receives TR. history y, language I and the diffusion timestep k inputs. These inputs are individually embedded into embeddings h of size d via separate MLPs f, depicted as:
h1 = ((CLIP()) hT; = fT: hz = Jas (xx), hy= fz()),
where language instructions is encoded with CLIP's language encoder (66]. Afterwards, the embeddings formulated as input tokens Ptokens IN( x [h,, hT Nan hy + F)0e), where g is the positional embedding, and LN denotes layer normalization [3] for stabilizing training. The input sequence that represents video can be extremely long so a standard Transformer with O(n²) complexity is hard to fit. We adopt Perceiver Transformer [41] to tackle this problem, as it has Perceiver latent-space Transformer, where instead of attending to the entire input, it computes cross-attention between the input and a much smaller set of latent vectors (which are randomly initialized and trained). These latents are encoded with self-attention layers, and are cross-attended with the input to match the size for the final outputs. More details about the Perceiver Transformer are referred $A.2
been widely utilized for modeling long sequences 75 24
3.3 Learning to Act via Few-Shot Fine-Tuning
During the fine-tuning stage, we leverage a limited dataset of robot data, including both videos and actions, for rapid adaptation. Both x" and x" attend to training the diffusion model. Given that pe, has been trained sufficiently to capture fruitful information predict future videos from history, we freeze Pe, and solely tune parameters of pe, to minimize Cvlb- As expressed in Eq. (8). the input of po, consists of Øg. language I, hidden representation Zaz. and diffusion timestep k. In this case, we are tasked with predicting a sequence of action tokens zg considerably shorter than video-token sequence 2%) we employ GPT2 (67) Transformer to process tokens embedded with MLPs. GPT2 has demonstrated an impressive ability to solve multi-task problems and model multimodal distributions. The model architecture of po, closely resembles that of MTDIFF-P [32]
More details of our method can be found in Appendix A.3
4 Related Work
18 27 for policy learning Robot Learning from Human Videos. Leveraging human videos (26]
23 23
is promising extract commonsense knowledge from human activities, which can be shared to embodied scenarios that suffer from scarce robot data [17 22 Since the human data is actionless and the domain gap between humans and robots exists, a main branch of research employs human video to learn shareable visual representations [12 [77 via contcconre [57], video-language alignment
[6 48 value function [8), and perceptual skills 40
[22 Visual affordance like human-object
interaction hotspots [54] 25 and the post-grasp trajectory S are also helpful for embodied agents in goal-conditioned imitation. Alternative methods involve extracting hand trajectories 4 [55 or keypoints [93 to transfer plans to robots. Different from the above methods, we eliminate the domain gaps by learning video tokens and representations for video prediction, which implicitly captures visual features, affordances, and long-term plans. Other works attempt to infer actions from videos via inverse kinematics G. (50) whereas we learn action prediction through policy fine-tuning without external models.

---

Pretraining for Generalized Policy Learning, Early works of policy adaptation emerged in meta-Ru RL 33 (9). while the pre-training and fine-tuning environments are assumed be similar. Leveraging the transformer architecture, works perform pre-training in multi-task datasets by optimizing the
multi-task policy [51 70 79 XD] or self-supervised objectives 78 73 59]. tasks involving visual
observations, methods adopt visual tokens for transformer-B bbasd multi-task policy learning [10] [1]
and adaptation [53]. Additionally, some studies pre-train a reward function through video-language
correspondence 113 56 or diffusion models 121 38] for downstream RL training. Different from the previous Transformer and continuous diffusion frameworks, our work first integrates visual tokens with discrete diffusion to predict consistent videos and actions simultaneously. Concurrently, GR-1 [8 utilizes human data to pre-train a GPT-style architecture for predicting future observations. In contrast, we perform video prediction instead of sie-bb-sttep image prediction using unified discrete diffusion architecture.
Diffusion Models for RI. Diffusion models are a powerful family of generative models 72 69 that can be categorized into continuous Gaussian diffusion models and discrete diffusion models that handle discrete visual tokens symbols [29] Continuous diffusion models have found extensive
applications as multi-modal policies [87 64. [6. environmental dynamics (95), and planners to
generate action 44 [0] or state sequences [4],, guided by desired properties. Several methods
also extend the diffusion models for multi-task learning 32 62 [9] with low- www-dimensional states, while we specifically address the more challenging image-based setting. UniPi [20] and its following work [100] are related to our method by performing video generation via continuous diffusion, while adopt more flexible architecture with discrete diffusion to seamlessly connect the pre- training and fine-tuning stages, without relying on a task-specific inverse-dynamic model for acting. Additionally, train the model video tokens that maintain temporal consistency, while UniPi relies on super-resolution to improve the time consistency of generated frames.
5 Experiments
5.1 Bnechmarks and Baselines
After the video pertaining in Ego4D 27 we use the following robotic benchmarks to evaluate our method.
Meta-Worid. The WorldWorld benchmark [98] contains 50 distinct manipulation tasks that require a Sawyer robot to interact with various objects with different shapes, joints, and connectivity. The action is the 3D position movements of the robot's end effector and the gripper openness. We follow recent works (7) to extend the original environment to a more challenging setting with random goals, and refer to it as MT50-rand. We train the policy with 20 demonstrations per task, and report the average success rates on 50 evaluation episodes per task.
RLBench. RLBench 43 is a more challenging 3D manipulation benchmark with diverse tasks concerning interactions with a wide range of objects. We select 16 tasks from RLBench to evaluate our method, where each task has several possible variations, such as the shapes, colors, sizes and positions of objects. The input observations are captured from four RGB cameras positioned at the front, left shoulder, right shoulder, and wrist. The action is an 8-dimensional vector including 3-dimensional transitions, 4-dimensional quaternion, and a binary value about gripper openness. We follow the convention by using macro steps 422, which are key turning points in the action trajectory where the gripper changes its state (open/close) or the joint velocities approach to zero. We train the policy with 10 demonstrations per task and report average success rates on 25 evaluation episodes per task.
Baselines for Meta-World. We compare the proposed method VPDD with the following baselines: (i) R3M-Diffusion is a discrete diffusion model sharing identical architecture with pa_. leveraging the R3M (61) ResNet50 encoder to encode images as input. R3M is also trained on Ego4D videos via a contrastive learning objective and stands as the state-of-the-an (SOTA) visual representation
specifically designed for manipulation tasks; (ii) VC-1-Diffusion utilizes VC-1 [88 encoder (ViT-L) to extract image representations, which is also trained on large-scale egocentric videos (27] and ImageNet [71] using Masked Auto Eico-Ennoiing [33]. (iii) MTDIFF-P B2] is the SOTA method for multi-task RL, which employs a continuous diffusion model with a Transformer architecture. Since is designed to handle baste input, we employ the R3M encoder to extract hidden embeddings from images, which are then fed into MTDIFF-P; (iii) Video-MTDT is an extension of Decision Transformer (MT) [15), lcarning from multi-task data with video tokens as input; (v) VPDD-w/o-
7

---

Netttd
block
seill
atick
blocks
RVT (10 denoni
54.67 1.69
76.0
6.89
90.0
91.38 0.60
97:33
1.50
VPDD (0ard)
70.07 + 1.39
5.27
79.38 4.99
58.07 $ 3.77
24.0 8 3.72
ST.33 8.22
00.07 1,89
DEL0 0.00
Nettod
balb
put
vace
place
Mase
put is captourd
bacions
push
insert
p4k
stack
cupa
place
cupa
PRRACT desue]]
70.07
30.07
1.80
1.50
50, 30.67
Table 1: Success rates (mean and std %) across 3 random seeds of various multi-task agents trained with 10 demonstrations and evaluated on 25 episodes per task. VPDD (Ours) outperforms SOTA methods of RLBench, i.e., PERACT and RVT, with an average improvement of 1.24x. Note that VPDD only takes RGB images as input while both RVT and PERACT utilize additional depth images as inputs.
human excludes human videos during pre-training, remaining other parts unchanged. This bascline helps to ablate the effect of pre-train on large-scale human videos; (vi) SODA [3 is a recently proposed diffusion-based representation method that employs an encoder generate representations z from input images to support the denoising process. We pre-train the encoder by employing the video prediction objective on the same dataset and subsequently feed the learned into pa, during fine-tuning. See more details in SB
Baselines for RLBench. Learning policies for RI.Bench is more challenging as it requires under- standing the 3D scene structure for predicting the 6D poses of end-effectors. The baselines used in Meta-World all fail in the benchmark since they are disadvantaged with single-view observations. In contrast, VPDD can predict multi-view images, implicitly recovering the 3D geometry in manipu- lation. Thus, we use the following SOTA imitation architectures designed for 3D manipulation: (i) RVT 24 stands as the SOTA method, initially re-rendering visual observations into orthographic projections of cube views and subsequently predicting the next move based on these multi-view projections.; (ii) PERACT [75] encodes RGB-D images into voxel grid patches for 3D representation and predicts the action using the Perceiver Transformer.
5.2 Results Analysis
S00A
56.37
Success Rate (%)
44.72
45.9920.07
20127 S
80-
Meta-World
Figure 3: Single-view and multi-view images from Meta-World outton-press and RLBench drug-stic stick tasks, sampled from videos predicted by pe,.
Figure 4: Average success rate across 3 seeds on MT550-and. Each task is evalu- ated for 50 episodes.
Question 1. Does our pre-trained diffusion model (i.e., po, capable of generating dynamic-consistent future videos?
Although our primary contribution is not about video generation, it remains cru- cial to predict consistent future videos to aid in policy fine-tuning. The predicted
raw videos are visually depicted in Fig. 3 with more video samples including real
robots (trained by RoboSet data 77) accessible at https://video-da diff gi thub.
These videos are reconstructed from predicted video tokens by using the decoder of VQ-VAE. After pre-training, the video-predie prediction model Pe, demonstrates the capability to generate dynamically con- sistent single-view view videos for Meta- World and multi-view videos for RLBench. Furthermore, we computed the Frechet Video Distance (FVD) score 34 86 averaged across frames on 32 generated video samples from the Meta- World dataset. From the results in Table score.
Method
FVD (L)
UniPi (20
264.66
Meta-World
VPDD (Ours)
235.81
Table 2: Comparison of FVD
we observe that the FVD score of VPDD considered acceptable and even lower than the hierarchi- cal video synthesizing method UniPi 200 (score reported from its original paper). We attribute the
8

---

capability of generating high-quality videos to the well-trained video codebook and the proposed discrete diffusion model learned from large-scale human data.
Question 2. How does VPDD compare to other offline baselines for vision-based multi-task decision- making?
To evaluate the learned policy after fine-tuning, we take the first action generated by ps, to interact with the environment. The results
on Meta- World and RLBench are referred to Fig. 4 and Table T respectively, yielding the following key findings: (i) VPDD outper-
VI50 RIM Citulinnnonn
forms other SOTA methods in success rate by a large margin. For
Succoss (late
VPDD performs the best across 50 tasks with random
goals. For RI Bench, VPDD even outperforms the SOTA imitation architectures based on voxel and multi-view representations that are carefully designed for 3D manipulation, which usually require point clouds 3D world rendering for scene understanding. Notably, VPDD achieves a remarkable success rate on put in cupboard, insert peg, stack cups and place cups, while both RVT and PERACT strug- gle on these challenging tasks. This verifies the efficacy of video pre-training for few-shot policy fine-tuning: (ii) According to Fig. VPDD obtains a 6.9% relative improvement through pre-traine pre-training on both human and robot videos compared with HODD-w/o-hhumat R3M and VC-1 with notable 16.4% and 26.5% higher success rate, demonstrating the potential of our diffusion representation via video prediction; (iii) R3M-Diffusion outperforms MTDIFF-P which employ R3M encoder with continuous diffusion architecture by 26.0%, and Video-MTDT with Transformer architecture by 60.6%. This highlights the superior capacity of discrete diffusion models compared to other model architectures.
Talle Texture
Camera Pos
Figure 5: Average success rate across seeds on shifted button seitoon-reess-2 and handle- press-v2 tasks.
Furthermore, VPDD surpasses
Question 3. How does VPDD generalize unseen scenes?
As suggested in recent works 192 (97), we evaluate the generaliza- tion ability of our model in the two most challenging settings, i.c., camera view and visual background. Specifically, we alter the cam- position and table texture in the visual scene of Meta-World to assess the generalization ability of our method. According to Fig. []] VPDD exhibits superior generalizability. attributed the training of the diffusion representation on large-scale diverse human videos. Regarding the shift in camera position, VPDD outperforms the VPDD-w/o. -hhmmmn by 63% and R3M by 252%. Moreover, we show that VPDD can generalize on different tasks with competitive performance, demonstrating its potential to serve as a foundation model. Detailed results and settings can be found in $C.3
15
Number of Demonstratios
Figure 6: Average success rate across 3 seeds on MT50-rand, where VPDD is trained on a different number of demon- strations.
Question 4. Can VPDD maintain satisfactory perfor- mance when provided with fewer robotic demonstrations?
Success Rate: 196)
30
Leveraging the lcrge-scale video pre-training, VPDD can learn the policy using only small number of demonstra- tions. To validate the sample efficiency of our method, we
conduct ablation study the number of demonstra- tions used in the fine-tuning stage. The results, depicted in Fig. (6 reveal that the performance of VPDD exhibits linear growth after training on 5 or more demonstrations, indicating the potential for VPDD to achieve better per- formance with increased demonstration data. Moreover, VPDD maintains comparable success rate even when only 1 demonstration is used in the fine-tuning process.
vriginal MT50 Shited Camera Position
Number of Human
Ablation on the number of human videos during the pre-training stage, where the red curve is evaluated using the same experimental setting as in Fig. 4 and the blue curve corresponds to the setting in Fig. 5
Figure 7:
Question 5. How does VPDD perform when trained on different amounts of human data?
The superior generalizability of VPDD, which is validated in Fig. (5) stems from large-scale human-date pertaining. which effectively extracts commonsense

---

knowledge and representations that can be shared between human and unseen robot scenarios. To further investigate the effects of human-date pre-training, we ablate the number of human videos used during pre-training and present the performance changes in Fig. 7 Our findings indicate that an increased number of human videos enhances success rates, particularly in improving generalizability by a large margin.
Question 6. Does VPDD outperforms other diffusion- based representation learning method?
To verify the representation capability inherent in our unified discrete diffusion framework, we reproduce SODA (39, which serves strong diffusion representation method. As shown in Fig. 4 VPDD outperforms SODA in the context of policy fine-tuning. We hypothesize that VPDD provides ceare-ttoffine representations (i.e., 2gg) throughout the action-denoising steps from K 1, exactly encapsulating useful information the denoising network focuses on at each step 35 In contrast, SODA produces representation that remains constant across all steps during action denoising.
6 Conclusion
We propose VPDD, video-based policy learning framework via discrete diffusion. With VQ- VAE tokenizer, we bridge the gap between human and robot videos by discrete latent codebook. We leverage unified discrete diffusion for pre-training on large-scale actionless mixture videos and subsequent fine-tuning the policy on limited number of robot demonstrations. Experiments demonstrate that VPDI achieves superior performance on variety of challenging manipulation tasks and showcases impressive generalization ability benefited from human video prediction. More Discussions of our work are given in D
Acknowledgments
This work is supported by Shanghai Municipal Science and Technology Major Project (2011SHZDZX0102) and National Natural Science Foundation of China (62322603 & 62306242). We also thank the anonymous reviewers for their valuable suggestions.
References
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In International Conference on Learning Representations, 2023.
[2] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, 2023.
[3] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/16007.06 (0440, 2016.
[4] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. 2022.
[5] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13778-13790, 2023.
[6] Homanga Bharadhwaj. Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint arXiv:2302.02011 2023.
[7] Homanga Bharadhwaj, Jay Vakil, Mohit Sharma. Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In First Workshop on Distrrb-istiintiion Generalization Robotics at CoRL. 2023, 2023.
[8] Chethan Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh, Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic offline RL from internet videos via falue-function pre-training. In NewrIPS 2023 Foundation Models for Decision Making Workshop, 2023.
10

---

[9] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English. Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv: 1,,1277, 2023.
[10] Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robocat: A self- improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306. 1706, 2023.
[11] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, and et al. Rt-1: Robotics transformer for real-world control scale. In Robotics: Science and Systems, 2023.
[12] Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation? arXiv preprint arXiv:2312.12444/ 2023.
[13] Annie Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" mn-the-wild" human videos. arXiv preprint arXiv:2103. 2021.
[14] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference Learning Representations, 2023.
[15] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 14:14584-55097, 2021.
[16] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[17] Open X.-Embodiment Collaboration. Open x-embodiment: Robotic learning datasets and RT-X models. CoRR, abs/2310.08864 2023.
[18] Dima Damen, Hazel Doughty. Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The mitches dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPPMMI), 43(11): 4141, 2021.
[19] Zibin Dong. Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu, Aligndiff: Aligning diverse human preferences via behavior- customisable diffusson model. In International Conference on Learning Representations, 2024.
[20] Yilun Du, Sherry Yang, Bo Dai. Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Neural Information Processing Systems, 2023.
[21] Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng. Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. In Neural Information Processing Systems, 2023.
[22] Hao-Shu Fang, Hongjie Fang, Zhenyu Tang. Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition @ CoRL2023, 3:5, 2023.
[23] Kuan Fang, Te-Lin Wu, Daniel Yang, Silvio Savarese, and Joseph J Lim. Demo2vec: Rea- soning object affordances from online videos. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2139-2147, 2018.

---

[24] Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Wei Chao, and Dieter Fox. RVT: Robotic view transformer for 3d object manipulation. In 7th Annual Conference on Robot Learning. 2023.
[25] Mohit Goyal, Sahil Modi, Rishabh Goyal, and Saurabh Gupta. Human hands as probes for interactive object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3293-3303, 2022.
[26] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilo,, Moritz Mueller-Freitag al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842- 5850, 2017.
[27] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[28] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Tri- antafyllos Afouras. Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exoAd: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:231 1. 2023.
[29] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706, 2022.
[30] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. Advances in neural information processing systems, 31, 2018
[31] Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, pages 8387-8406. PMLR, 2022.
[32] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang. Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is effective planner and data synthesizer for multi-task reinforcement learning. In Neural Information Processing Systems, 2023.
[33] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[34] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[35] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840 6851, 2020.
[36] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems, 2021.
[37] Minghui Hu, Chuanxia Zheng, Zuopeng Yang, Tat-Jen Cham, Heliang Zheng, Chaoyue Wang. Dacheng Tao, and Ponnuthurai N. Suganthan. Unified discrete diffusion for simultaneous lanionelanguage generation. In International Conference on Learning Representations, 2023.
[38] Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. arXiv preprint 2rXiv22312.14134, 2023.
12

---

[39] Drew A Hudson. Daniel Zoran, Mateusz Malinowski, Andrew K Lampinen, Andrew Jaegle, James L McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottleneck diffusion models for representation learning. arXiv preprint arXiv:2311.199000 2023.
[40] Mingxiao Huo, Mingyu Ding, Chenfeng Xu, Thomas Tian, Xinghao Zhu, Yao Mu, Lingfeng Sun, Masayoshi Tomizuka, and Wei Zhan. Human-oriented representation learning for robotic manipulation, 2023.
[41] Andrew Jaegle, Sebastian Borgeaud, Jean- Saan-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general architecture for structured inputs & outputs. International Conference on Learning Representationss 2022.
[42] Stephen James and Andrew J Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 7(2): 1121-161,,,,,, 2022.
[43] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[44] Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew Davison. Coarse-to-fine q- attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13739-13748,
2022.
[45] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
[46] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[47] Ya Jing, Xuelin Zhu, Xingbin Liu, Qie Sima, Taozheng Yang, Yunhai Feng, and Tao Kong. Exploring visual pre-training for robot manipulation: Datasets, models and methods. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11390-11395. IEEE, 2023.
[48] Siddharth Karamcheti, Suraj Nair, Annie Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-drives representation learning for robotics. In Robotics: Science and Systems (RSS), 2023.
[49] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
[50] Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao- Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In International Conference on Learning Representations, 2024.
[51] Kuang-Huel Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35:27921-27936, 2022.
[52] Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion-based task execution. arXiv preprint arXiv:2312.11594 2023.
[53] Xingyu Lin, John So, Sashwat Mahalingam, Fangchen Liu, and Pieter Abbeel. Spawnnet: Leamingg generalizable visuomotor skills from pre-trained networks, 2023.
[54] Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3282-3292 2022.
13

---

[55] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-lame ann-lagg tasks. Advances in neural information processing systems, 32, 2019.
[56] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. LIV: Language-image representations and rewards for robotic control. In International Conference on Machine Learning, volume 202, pages 23301-23320, 2023.
[57] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre-training. In International Conference on Learning Representations, 2023.
[58] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, fincent-Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems, 36, 2024.
[59] Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. 2023.
[60] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. EmbodiedGPT: Vision-language pre-training via embodied chain of thought. Neural Information Processing Systems, 2023.
[61] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning, 2022.
[62] Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang, Metadif- fuser: Diffusion model as conditional planner for offline meta-rl. 2023.
[63]
OpenAl et al. Gpt-4 technical report, 2023.
[64]
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejud, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In International Conference on Learning Representattinss 2023.
[65] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cap- pelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 / 16, 2023.
[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[67] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. 2019.
[68] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training CoRL, 2022.
[69] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 204.0120 1(2):3, 2022.
[70] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky. Jackic Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.061 206..2655..6. 2022.
[71] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211- 252, 2014.
14

---

[72] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding. 2022.
[73] Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning, pages 19561-19579. PMLR, 2022.
[74] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. In 6th Annual Conference on Robot Learning, 2022.
[75] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In 6th Annual Conference on Robot Learning, 2022.
[76] Jascha Sick-Dii,,,,,,, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015.
[77] Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parameter- compositional multi-task reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[78] Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor. SMART: Self-supervised multi-task pretraining with control transformers. In Interna- tional Conference on Learning Representations, 2023.
[79] Adrien Ali Taiga, Rishabh Agarwal, Jesse Farebrother, Aaron Courville, and Marc G Belle- mare. Investigating multi-tast task pretraining and generalization in reinforcement learning. In International Conference on Learning Representationss 2023.
[80] Garrett Thomas, Ching-An Cheng, Ricky Loynd, Felipe Vieira Frujeri, Vibhav Vineet, Mihai Jalobeanu, and Andrey Kolobov. Plex: Making the most of the available data for robotic manipulation pretraining. In Conference on Robot Learning, pages 2624-2641. PMLR, 2023.
[81] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[82] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[83] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representa- tion learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[84] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, Advances neural information processing systems, 30, 2017.
[85] Chen Wang, Linxi Fan. Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint 2rXiv:2302.11242. 2023.
[86]
Ting-Chun Wang, Ming-Yu Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. arXiv preprint arXiv: 06601, 2018.
[87] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. In International Conference on Learning Representations, 2023.
[88] Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations, 2024.
15

---

[89] Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with in m-lhh-wild videos for reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[90] Zhou Xian, Nikolaos Gkanatsios, Theophile Gervet, Tsung-Wer wei Ke, and Katerina Fragkiadaki. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference Robot Learning, 2023.
[91] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv:2203.06175. 2022.
[92] Annie Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. arXiv preprint arXiv:2307.03653 2023.
[93] Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj, Samarth Sinha, and Animesh Garg. Learning by watching: Physical imitation of manipulation skills from human videos. 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7827-7834. IEEE, 2021.
[94] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Vidcogpt: Video generation using vq-vae and transformers, 2021.
[95] Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In International Conference on Learning Representations, 2024.
[96] Ruihan Yang. Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777 2020.
[97] Sizhe Yang, Yanjie Ze, and Huazhe Xu. Movie: Visual model-based policy adaptation for view generalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[98] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100, PMLR, 2020.
[99] Haoqi Yuan and Zongqing Lu. Robust task representations for offline meta-reinforcement learning via contrastive learning. In International Conference on Machine Learning, pages 25747-25755 PMLR, 2022.
[100] Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit- Yan Yeung, and Chuang Gan. Ro- bodreamer: Learning compositional world models for robot imagination. arXiv preprint 12377, 2024.
16

---

A The Details of VPDD
A.1 Discrete Diffusion
Discrete diffusion models with a mask-and-replace strategy gemerate sequences from [MASK] tokens. In this paper, they are trained by sampling a sequence æ = |<" æ"). masking tokens according to a linear schedule [29] that corresponds to increasing the probability of being in the absorbing state linearly over time, and learning to predict the masked tokens given language and historical videos.
Specifically, for the proposed unified transition matrix Qs in Eq. (6), the computation of diffusion process (xx//20)) in Eq. (4) can also be obtained in the following closed-form 37:
TD Cg.To +
[Tx 12x0)3 1-'--''0 13(0)) 2 + (20) + (20))
if argmax zo E [0, J),
(10)))) = {1 if argmax zo E [J,J+ otherwise.
where (xr)
0 otherwise.
[N] = argmaxz J' and aks Bkg 3 k. 7x are the corresponding cumulative product.
(11)
The reverse process predicts 2 and σ by training denoising network po; (#t [. v, and Pa, (0 Øk z 1). respectively. Then the forward process is used to compute Pa, (x) [k 1) as expressed in Eq. (12 and po. (o 1/0k 2ag. 1) as expressed in Eq. (I3)
po; (ar 1k y.
( ak Po (3rg) mk. y, L),
(12)
Pog (2 [k Pay, Σ
( _ |a 26) (28)) zay. 1)
(13)
A.2 Perceiver Transformer
We the Perceiver Transformer to encode extremely long input sequences, which improves the computation efficiency. We maintain a set of latent vectors zq of dimensions 2008 x266 which randomly initialized. Then compute cross attention between the input sequence and =q. The process is illustrated in Fig. g zQ are processed with 6 self-attention layers, and then decoded into output with the same dimension space of input via cross attention.
Imput
Output
K
V
Encode
Process
Decdde
K
K
Cross atten
Cross atten
Latents
Self atten
Figure 8: Illustration of Perceiver Transformer architecture. Perceiver latent-space transformer. Q K, V represent queries, keys, and values, respectively. We use L = 6 self attention layers in our implementation.
A.3
Implementation Details
In this section, we give the pseudocodes of pre- pre-training and fine-tuning in Alg [ and Alg rrespee- tively. Then we describe the details of the training process, architecture and hyperparameters:
Following previous works 36
we sample diffusion timestep l: ~ q(k) with importance sampling
strategy, where (k)
E C.
17

---

We set batch size as 20 for pre-training and 40 for fine-tuning We train our model using Adam with 2e learning rate for 2e training steps.
optimizer 49
For pre-training, we represent denoising network pe, Perceiver Transformer described in A.2 MLP ft. which processes the language embeddings given by the CLIP encoder, MLP fy- which processes the historical video tokens, and MI.P Sæs 2-layered ML.Ps (prepended by layer norm 3 and with Mish activation). MLP fTs, which processes diffusion timestep k, is 2-layered MLP (prepended by Sinusoidal embedding and with Mish activation). Finally, use linear layer to project the encodings given by the Perceiver Transformer into the original dimension space of the imput.
For fine-tuning, represent denoising network po, GPT2 Transformer. Similar to the architecture of po,, we first use MLPs with Mish activation to project different inputs into the same hidden dimension space, and then recover the encodings given by the GPT2 Transformer into the original dimension space via linear layer.
We choose the sequence length = 4 and M for future actions and videos prediction.
We set 20 which means that we predict future videos after 20 steps.
We set diffusion timesteps K 100.
We run all the experiments on single RTX 3090 machine.
B The Details of Baselines
We describe the implementation details of baselines used in our experiments as follows:
R3M-Diffusion. We borrow the pre-trained R3M encoder from https://githht com/ facabookressercch/rzi and leverage the encoder to encode sing-view RGB images in Meta- World and multi-view RGB images in RLBench. Thus, we skip the pre-training process and directly train our discrete diffusion model on the robot data. The model architecture and hyper-parameters R3M-Diffusion are almost the same as p8.- The encoded hidden representation encoded by R3M is denoted as 2gaM then the denoising network can be written as po (æ) [x 2RSM 1).
MTDIFF-P. We borrow the official codes of MTDIFF-P from /tttps://itthbb com/ tinnerhrhe/MTUS
In order to process sige-diimesional images instead of low-dimensional states, we leverage the R3M encoder in R3M-Diffusion to obtain the visual representations.
VIDEO-MTDT. We use the language to indicate tasks, which are encoded as a vector with size 2048 by the same CLIP encoder used in VPDD. We take the video tokens used in VPDD as the states. Then we follow the implementation from https://githii wwa.ooo--trttssoooerreeoomoomoee to train Video-MTDT on the limited robot data.
com/kz//
VPDD-w/o-human. During the pre- tr-irrinnig stage of VPDD, we remove the human videos and only use the robot videos for training.
SODA. Following the SODA paper [39] and open-sourced codes from https://github. com/ FutureXiang/soon we first maintain an encoder ESODA equipped with the same Perceiver Transformer in pe: Then during the pre-training stage, representation 2 is first encoded by
ESODA conditioning on e, and I, i.c., = ESODA (es, l). Then -;-1 is obtained via the following
process:
Ø; Attn (Kk, z)
(14)
Fk Attn (k-11
where Attn is an attention operation 88 55 where the queries are formed from x, the keys and
values from y. The encoder is trained end-to-end and to minimize the same loss term Lvlh- During the fine-tuning stage, Σay is replaced of output by ESODA. The model architecture and other hyper-parameters during fine-tuning remain the same.
18

---

Algorithm Pre- Pre-Training Stage of VPDD
Initialize: given unified transition matrix (Qk), well-trained VQVAE, training iterations N, initial network parameters 81 and 02, learning rate
1: for video clips 1 in (T,) do
2: E; f VQVAE-EEOooo (
3: end for
4: for to N do
5: Sample batch B = (xx", x²" .y. 1) from human and robot videos, where x <-
6: Sample diffusion timestep from [1, K] with importance sampling strategy
7: zx (( where Wo [x" x=1 Eq. (II) and
Lo if k
8: Lub
Eq.
-
otherwise.
9: 01 01 Vo.
C
D Adam optimizer
10: end for
Algorithm 2 Tuning Stage of VPDD and Evaluation
# Fine- Timing Process
Initialize: given unified transition matrix {Qu}-
well-trained VQVAE, training itera-
tions N. pre-trained network parameters 01 and initial parameters 02. learning rate 7)
1: for video clips i and actions a, in {Ti} do DOVAAEEncoder((y) ae Discretize(a:)
2:
3: end for
4: for to N do
5: Sample a batch B =
L) x x,,
from videos and discretized actions dataset.
6: Sample diffusion timestep from [1, K] with importance sampling strategy
7: æy + (( (2)), where 20 = [x=", xx" >> Eq. and
Lo
k ===, 1.
8:
Lvb Ck-1
t> Eq.
otherwise.
9: O2 + 02 - Vo,
D> Adam optimizer
end for
Evaluation Process
Given task, reset the environment
2: Obtain the initial video clips vo. language instructions I
3: for = 0 to max do
4: Initialize [2) k p(œk) > Eq, (TO)
5: Construct <- e, VQVAB- EacAE-EEcdde(r
6: for k K to do
7: Sample xi_1 ~ Pe, (2)
Dk: y. 1) and obtain Zay from pe, encoding t Eq. pky Zæy.
12
8: Sample _ Poz (mf.
Eq.
9:
26-1 [x -
10:
end for
Obtain predicted videos via (QVAE-Decoder(aa
12: Reconstruct executable action sequence (ar, - from a
Execute the first action a, as the current action to interact with the environment
14: Obtain the next observed image(s), and update U.
15: end for
C Datasets and Experiments
C.1 Dataset
Meta-World. We use the official codes from https://githhbb http://githuu.ccooaaaaaaaaaaaaae hwmm/Parama-Fuundattinn
Metaworld to collect 20 expert demonstrations for each task in Keta-Worrd. The image size
19

---

is 260 x 260. The dimension of action is 4, representing the 3D position movements of the end effector and the variation of gripper openness. Every demonstration collected has 150 time steps.
RLBench. We use the official codes from https: thub. cwm/ con/ppracttppracct to generate a dataset for RLBench via motion planner. Each demonstration consists of multi-view RGB images (i.e., front, left shoulder, right shoulder and wrist) and 8- 8-dimensional actions including 3-dimensional transitions, 4-dimensional quaternion and a binary value about gripper openness. Following prior works 42 75.. we extract macro actions (keyframe actions) from collected demonstrations and lever- age networks to predict the keyframe actions instead of consistent continuous actions. Specifically, set of keyframe actions {k1, .2. km J C A is captured with simple heuristic: action is keyframe if (1) the joint velocities are near zero and (2) the gripper open state has not changed. Each data point in the demonstration can then be cast as a "predict the next (best) keyframe action" task 44 In this way, the sequence length of actions that need to be predicted is significantly reduced from hundreds of small steps to typically less than 10 macro steps.
C.2 Task Details
Task
Variation Type
# of Variations
Language Template
slide block
color
4
"slide the block
target"
sweep to dustpan
size
2
"sweep dirt to the dustpan"
meat off grill
category placement placement
2
"take the the--0 off the grill"
turn tap put in drawer
2
"turn tap"
3
"put the item the drawer"
close jar drag stick
color
20
"close the jar"
color
20
"use the stick to drag the cube onto the target"
stack blocks
color, count
60
"stack -boccks_blckss blocks" "screw in the light bulb" "put the money away in the safe on the shelf"
bulb put in safe place wine
color
20
placement
3
placement
3
"stack the wine bottle to the of the rack"
put in cupboard
category
9
"put the in the cupboard"
push buttons insert peg
color
50
"push the button, [then the button]"
color
20
"put the ring on the spoke"
stack cups
color
20
"stack the other cups on top of the cup"
place cups
count
3
"place cups on the cup holder"
Table 3: Language-Conditionn Tasks in RLBench
43 with various variations.
We take Meta-World as a main benehmark to evaluate our method and baselines, which consists of 50 diverse manipulation tasks. The poses and positions of goals are randomly generated during evaluation. These tasks require an agent to identify the observed sing-view RGB images and reach the goals with the correct behavior. See Fig. 9 for sample visualization of the tasks.
We select 16 tasks out of 100 tasks from RLBench 4% that involve at least two or more variations to evaluate the multi-task and generalization capabilities of agents. Task variations include randomly sampled colors, sizes, counts, placements, and categories of objects. The of colors include 20 instances:
colors (red, maroon, lime, green, blue, navy, yellow, cyan, magenta, silver, gray, orange, olive, purple, teal, azure, violet, rose, black, white). The set of sizes include 2 instances: sizes (short, tall]. The set of counts include 3 instances: counts = (1, 2, 3). The placements and object categories are specific to each task. For instance, put in cupboard includes YCB objects. In addition to these semantic variations, objects are placed on the tabletop at random poses. The tasks in RLBench require an agent to process multi-view RGB images properly and generalize different variations that could be unseen in training. The details of variations for each task are referred to Table
C.3 Experimental Setup and Additional Results
Generalize to unseen scenes. To evaluate the generalization ability of our method, we change the camera position and table texture the Meta- benchmark generate out-of-distribbtion observations. Borrowing the official codes from /ttpp:://gii http:/gtttth......g.geeee http:/ghhww....mmllll/////aatttt
20

---

eary
slock
(a) window-open
(b) dial-turn
(c) close jar
(d) meat-off-gri -gill
place
(e) Faucet-open
(f
sweep-into
(g)
stack blocks
(h) place cups
Figure 9: Visualization of several tasks in Meta-World and RLBench.
Unseen Tasks
VPDD
VPDD (oracle)
hand-insert-v2
32%
36%
bin-picking-v2
78%
84%
door-unlock-v2
100%
100%
Table 4: Average success rate of VPDD unseen tasks.
we set the camera at another corner position and generate unseen table texture randomly while rendering. See Fig. 10 for visualization.
Generalize to unseen tasks. In the following, we show that VPDD can generalize on different tasks from the same domain. experiments, during stage 2, we train the VPDD on 47 tasks on MetaWorld and leave 3 unseen tasks to test the generalizability. In stage 3, we fine-tune the pretrained model on these 3 tasks. We report the success rate of the final model trained over 0.2M gradient steps. The following experimental results in Table CK3 demonstrate the promising generalization ability of our model, as the performance gap compared with the oracle very small.
21

---

(a) button press
(b) shifted camera position
(c) shifted table texture
(d) handle-press
(e) shifted camera position
(f) shifted table texture
Figure 10: visualized example shifted camera position and table texture on button press--v2 and handle handle-press-v2 p2ess-v2 tasks. Table texture is generated randomly during evaluation.
D Limitations and Future Work
We present illustrative samples of robot videos generated using discrete diffusion model Ps, in Fig. and Fig. I1 More samples available at hhtttp://videooddifff .aff.gittubb.. io is noteworthy that VPDD is able to generate dynamic consistent future videos, incorporating information beneficial for low-level control while maintaining coherence across distinct perspectives. However, is imperative to acknowledge that our primary contribution is not generating high- videos with exceptional resolution and meticulous semantic details. Consequently, some blurriness may be observed in our predicted videos, exemplified by deviations in the gripper's 's pose. See Fig. 12 for failure example.
For future work, we could consider enhancing the coherence of videos across diverse views by lever- aging the recently released Ego-exo4d data [28]. This extension encompasses considerations beyond solely temporal dynamics. Moreover, for the augmentation of video quality and the optimization of decision-making performance, it is worth exploring to scale up both the training dataset and model capacity.
E Broader Impact
This paper presents work whose goal is to advance the field of Machine Learning. Specifically, we propose a novel pretraining- finetuning framework make better use of copious amount human actionless videos. Since this method is easy to reproduce (as we will release our code soon) and exhibits the SOTA performance, encourages future research to further advance this field. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
peerrainingfiietuuiing
22

---

(a) front
(b) front
(c) front
(d) front
left shoulder
(f) left shoulder
(g) left shoulder
(h) left shoulder
right shoulder
right shoulder
(k) right shoulder
(I) right shoulder
T
(m) wrist
(n) wrist
(0) wrist
(p) wrist
Figure 11: Predicted video given by pe, for task put the crackers in the cupboard.
23

---

(a) front
(b) front
(c) front
(d) front
(c) left shoulder
(f) left shoulder
(g) left shoulder
(h) left shoulder
ii) right shoulder
right shoulder
(k) right shoulder
(I) right shoulder
(m) wrist
wrist
(0) wrist
(p) wrist
Figure 12: Predicted video given by po, for task put the mustard in the cupboard. The pose of the robotic arm in the video is somewhat blurry, with deficiencies in correspondence across different views.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions and scope are introduced in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: has been discussed Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the implementation details in Appendix A.3 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: We provide the code and sufficient instructions of our approach in the supple- mentary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cmm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/pil wwwppbblccgguiicuceccccc
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide them in Appendix B and Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the error bars for evaluation under different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these details in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our method in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite them throughout our paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We already provide sufficient implementation details our approach in Appendix AA.3 We also provide detailed instructions run method in supplementary
material.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "F1g4gq3NEj", "reviewer": "Reviewer_v4Bj", "summary": "The paper targets at transferring the knowledge from human videos to robotic manipulation policy. The paper proposes VPDD, a method that first pre-trains on video generation on both human and robot videos and then fine-tunes on robotic manipulation data with action labels. The intuition is, by learning to generate human and robot videos, the model obtains a \"prior\" (e.g., task planning, trajectory planning, etc.) of robotic manipulation which makes it easier to learn diverse manipulation tasks with even few-shot data. The authors compare VPDD to several robotic pre-training baselines and show non-trivial improvements over several benchmarks.", "questions": "1. In Figure 4, VPDD w/o human videos is still better than other baselines. What's the reason here? Assuming VPDD-w/o-human has seen the same robot data as other baselines, what's the main factor that causes such a difference?\n\n2. The authors tried VPDD on unseen scenes such as varying background and camera poses. What about unseen tasks? Assuming there are some tasks presented in human videos but not in robot videos, will the model learn to do these tasks, at least to some extent?\n\n3. As a bonus, it would be good to evaluate some off-the-shelf pre-trained video generation models such as Stable Video Diffusion for future exploration, but I don't think it affects the main conclusions of this paper.", "limitations": "See above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear motivation that robot manipulation data is scarce and utilizing the large amount of human data will be helpful. The design of the model and the training pipeline is sensible and the results look promising.", "weaknesses": "1. The pre-training stage is trained jointly on human and robot data. The amount of human data is pretty abundant here but how many robot videos are we using here? If the robot videos are too few, will it affect the resulting pre-trained model from this stage? If the pre-training stage still needs a lot of robot videos, then it still faces the problem of lacking enough robot data. On the other hand, if the pre-trained model can still generalize pretty well without robot data in the pre-trianing stage, i.e., pre-training on human data only and fine-tuning on a small amount of robot data can work pretty well, it would be very helpful.\n\n2. Since the main message of the paper is training on human data can help learning robotic manipulation policies, it would be good to see how the performance changes when trained on different amounts of human data, for example, a scaling curve with x-axis being the number of human videos and y-axis being the performance. It would be interesting to see if the performance can keep going up by feeding more and more human data, or is there somewhere it saturates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and a positive assessment of our work! We carefully address your concerns as follows.\n\n>1. ... how many robot videos are we using here? If the robot videos are too few, will it affect the resulting pre-trained model from this stage?...\n\nAs stated in Section 3.1 of the paper, we use only a few robot demonstrations compared to large-scale human videos to demonstrate the sample efficiency of our method. For Metaworld, we have approximately $2e^5$ frames (50 tasks$\\times$ 20 episodes$\\times$ 200 steps), which are only 5% of the human videos; For RLBench, we have around $1e^5$ frames, accounting for 2.5% of the human videos. Although the action-labeled data is very limited, our method achieves the best performance compared with the extensive baselines.\n\nWe would like to clarify that the main motivation of our paper is to make the best use of the actionless video data for downstream decision making, rather than assuming the robot videos are inaccessible. However, it is reasonable to expect that using less robot data in the pre-training stage will result in worse model performance. This is because the representation $z_{\\tilde{{x}}_{0}^{\\rm v}}$ provided for finetuning would be less accurate and contain noise that affects the performance. To validate the performance of VPDD without seeing robot data during pre-training, we add an experiment where no robot data is used in the pre-training stage, and evaluate its performance on five tasks in Metaworld. **See the results in Table 1 of the global pdf**. We observe that while the performance of VPDD-w/o.-robot decreases, it remains competitive compared to the baselines.\n\n\n\n>2. ... it would be good to see how the performance changes when trained on different amounts of human data ...\n\nThanks for your constructive suggestions! We add additional experiments following your advice. Given limited time and resources, we re-train our model using 0%, 1/3 and 2/3 of the human data. We find that while the multi-task decision-making performances are not significantly different, the generalization abilities distinctly improve with more human data. **The added experimental results are refered to in Figure 1 of the global pdf**. \n\n>3. In Figure 4, VPDD w/o human videos is still better than other baselines. What's the reason here? Assuming VPDD-w/o-human has seen the same robot data as other baselines, what's the main factor that causes such a difference?\n\nThanks for the question. We would like to clarify that, without human videos, VPDD still includes three-stage training (i.e., video tokenization, video prediction, and policy fine-tuning) but is trained solely on robot data. As a result, VPDD-w/o.-human also learns to predict future videos in the robot domain and thus capture significant features for subsequent finetuning. Indeed, the video prediction becomes less generalizable due to the limited video data, while it still provides some foresight through future predictions. Meanwhile, VPDD adopts advanced discrete diffusion and carefully designed transformer architectures with action tokenization for action sequence prediction, which has proved to be more efficient than continuous diffusion in high-dimensional trajectory modeling [1-3].\n\n**Reference**\n\n[1] Vector quantized diffusion model for text-to-image synthesis. CVPR. 2022\n\n[2] Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023\n\n[3] Diffusion models: A comprehensive survey of methods and applications[J]. ACM Computing Surveys, 2023\n\n>4. The authors tried VPDD on unseen scenes such as varying background and camera poses. What about unseen tasks? Assuming there are some tasks presented in human videos but not in robot videos, will the model learn to do these tasks, at least to some extent?\n\nThank you for the question. There are two cases to analyze the generalization to unseen tasks: (a) Tasks that are unseen in pre-training but seen in fine-tuning. The experiments in Q1 sufficiently demonstrate that VPDD can generalize to such unseen tasks since the model has learned similar behaviors from human videos. To further validate the generalizability of VPDD in this case, we consider a realistic setting where only incomplete robot tasks are seen during pre-training. During the pre-training stage, we train the VPDD on 47 tasks on MetaWorld and leave 3 unseen tasks to test the generalizability. In finetuning stage, we fine-tune the pretrained model on these 3 tasks. We report the success rate of the final model trained over 0.2M gradient steps. We can see the performance gap compared with the oracle (tasks seen in pre-training) is small.\n\n| Unseen Tasks   | VPDD | VPDD (oracle) |\n| -------------- | ---- | --- |\n| hand-insert-v2 |   32%   |  36%   |\n| bin-picking-v2 |  78%    |  84%   |\n| door-unlock-v2 |  100%    | 100%    |\n\n(b) When tasks are unseen in both pre-training and fine-tuning, the model struggles to choose the correct actions. It is challenging because it does not know the action space and has no information about the meaning of specific actions. As a result, VPDD struggles to achieve high success rates in such scenarios.\n>5. ... it would be good to evaluate some off-the-shelf pre-trained video generation models such as Stable Video Diffusion for future exploration...\n\nThank you for your insightful suggestions! While it is a promising idea, it requires extensive revisions that are impractical within the rebuttal period. One potential approach is to add an action-prediction head to Stable Video Diffusion (SVD) and then apply our pre-training and fine-tuning methods. However, multi-modal training with both actions and videos presents significant challenges for SVD and other models, so we will leave this as future work.\n\nWe hope to have addressed all the raised concerns. Thank you again for your time and valuable feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wLfkn586tP", "reviewer": "Reviewer_kd9D", "summary": "The authors propose utilizing large scale internet videos along with robot data to train a diffusion model which predicts video future frames conditioned on past video frames and language description of videos. This diffusion model operated on the vector quantized embeddings of the video frames. Subsequently, this diffusion model is adapted for action generation which is trained using a smaller amount of robot video data with action labels. The analysis is performed on the Metaworld suite and RLBench.", "questions": "1. What are the computational costs of training the backbone?\n2. Are the results on RLBench and Metaworld finetuned for every task or are in multi-task setting, i.e., do you have one model that yields the aggregate result on Metaworld or do you have one model per task?", "limitations": "1. The method is currently limited to simulation and it remains to be seen if the results will transfer to real robots. \n2. The method still requires 20 demonstrations per task during pretraining. It’s not clear if this is a kosher assumption. What if behavior cloning was used on these 20 demonstrations? Would this yield higher results? If so, its unclear why one would use VPDD?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written, clear and easy to understand\n2. The idea is well motivated. The question of utilizing internet scale knowledge for embodied intelligence is an open area of research.\n3. The experiment results are statistically significant with multiple runs reported accompanied by error bars. \n4. The method performs well in comparison to baselines.", "weaknesses": "1. **Missing Baselines:** How does the method compare against other generalist imitation learning methods like Octo, RT-1 etc? It's unclear how such methods compare in the finetuning setting. \n2. **Missing Ablations on amount of robot data used during pretraining:** While no action labels have been used during pretraining, the model still has access to robot trajectories. How does the method perform when no robot data is present during pretraining?\n3. **Lack of robot experiments:** Does the model extend to any real robot setups?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and for a positive assessment of our work! We carefully address your concerns as follows.\n>1. How does the method compare against other generalist imitation learning methods like Octo, RT-1 etc?\n\nThanks for your suggestions. We would like to clarify that Octo and RT-1 are directly trained on large-scale robot data with action labels (e.g., Octo is trained on 800K episodes from the Open X-Embodiment dataset). In contrast, our method contains a human pretraining stage that leverages actionless video pretraining, significantly reducing the requirement of action-labeled robot data (i.e., only 1000 episodes for 50 tasks in Meta-World and 160 episodes for 16 tasks in RLBench). As a result, we believe the contributions of VPDD and Octo/RT-1 are orthogonal, where VPDD focuses on leveraging actionless pretraining and few-shot finetuning, while Octo/RT-1 focuses on directly learning generalized actionable policy from large-scale robot data. Additionally, we note that Octo is a nearly concurrent work, and neither Octo nor RT-1 presents evaluation results on Meta-World and RLBench.\n\nTo further validate the superior performance of our method, we train our model using 400 episodes from Language Table [1] blocktoblock dataset (400 out of 200,000 episodes), and finetune the pretrained Octo-Base model using the same dataset for $1e5$ steps. The Language Table is part of the Open-X dataset and is an extremely challenging benchmark due to the random instructions and block locations in each episode. We evaluate their performance in the Language Table environment over 100 episodes.  We find that Octo-Base gets a **0%** success rate while our model can achieve a **10%** success rate. Our findings are consistent with a recent work [2].\n\n[1] Corey Lynch et al., Interactive Language: Talking to Robots in Real Time, Arxiv 2022.\n\n[2] W. Zhao et al., VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation, RSS 2024.\n\n\n>2. Missing Ablations on amount of robot data used during pretraining: While no action labels have been used during pretraining, the model still has access to robot trajectories. How does the method perform when no robot data is present during pretraining?\n\nThanks for your suggestion. We would like to clarify that the main motivation of our paper is to make the best use of the actionless video data for downstream decision making, rather than assuming the robot videos are inaccessible. However, it is reasonable to expect that using less robot data in the pre-training stage will result in worse model performance. This is because the representation $z_{\\tilde{{x}}_{0}^{\\rm v}}$ provided for finetuning would be less accurate and contain noise that affects the performance. To evaluate the performance of VPDD without seeing robot data during pre-training, we add an experiment where no robot data is used in the pre-training stage, and test its performance on five tasks in Metaworld. **See the results in Table 1 of the global pdf**. We observe that while the performance of VPDD-w/o.-robot decreases, it remains competitive compared to the baselines.\n\n>3. Lack of robot experiments: Does the model extend to any real robot setups?\n\nDue to the resource limitation, we do not evaluate VPDD in real-world manipulation tasks. Nevertheless, the chosen simulation benchmarks in our paper are realistic and challenging, especially the 3D manipulation tasks with a Franka arm from RLBench. These tasks are also widely used in previous robot learning works and make it easy to compare VPDD to other baseline methods. To verify that our method can handle real-world scenarios, we fine-tune VPDD on real robot data obtained from RoboAgent data [1]. We observe that our model can quickly adapt to this domain and generate high-fidelity future videos in this context. We provide the predicted videos on the website video-diff.github.io. Additionally, we'd like to clarify that while real-world experiments can indeed be valuable, they are not a mandatory requirement for acceptance at NeurIPS. \n\n[1] Bharadhwaj H, Vakil J, Sharma M, et al. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. Conference on Robot Learning (CoRL). 2023.\n\n>4. What are the computational costs of training the backbone?\n\nThank you for the question. In our experiments, we use 8 RTX 3090 GPUs to train the VQ-VAE for video tokenization over a period of 3 days. The pre-training process takes approximately 2.5 days per RTX 3090, while fine-tuning requires about 1.5 days per RTX 3090. The training time could be reduced by utilizing more advanced GPUs and employing parallel computation.\n\n>5. Are the results on RLBench and Metaworld finetuned for every task or are in multi-task setting, i.e., do you have one model that yields the aggregate result on Metaworld or do you have one model per task?\n\nThanks for the question. They are finetuned in a multi-task setting, which leverage a single model to generate aggregated results in all downstream tasks. \n\n>6. The method is currently limited to simulation and it remains to be seen if the results will transfer to real robots.\n\nPlease see the answer to Q3.\n\n>7. The method still requires 20 demonstrations per task during pretraining. It’s not clear if this is a kosher assumption. What if behavior cloning was used on these 20 demonstrations? Would this yield higher results? If so, its unclear why one would use VPDD?\n\nThanks for the question. We have included the behavior cloning results without video prediction. As shown in Figure 4, MTDiff-p is a diffusion model that performs behavior cloning on 20 demonstrations pre-task. Our method outperforms MTDiff-p by 17.98\\%.\n\nWe hope to have addressed all the raised concerns and would be happy to respond to further questions and suggestions. Thank you again for your time and efforts!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ImCLsGlY3T", "reviewer": "Reviewer_ew7s", "summary": "In this paper, the authors focus on the problem of policy learning via leveraging large video data without action labels. To this end, they employ a discrete diffusion framework which is first employed to predict future quantized video frames. Afterwards, the model is fine-tuned on robot data to learn the final policy. The authors evaluate their approach on the Meta-World and RLBench and demonstrate promising quantitative results over other approaches.", "questions": "1. While I believe the current results in the paper are convincing, it still may strengthen the paper to demonstrate results on even more datasets.", "limitations": "Yes, the authors have properly addressed limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a very timely problem: Most available video data has no action labels. A major question is how to properly leverage this data to learn general policies for control tasks.\n\n2. For an approach which utilizes a pre-training and a fine-tuning stage with frozen networks, the approach isn't overly complex. It's simple relative to other approaches. It therefore should be easy for researchers to implement and reproduce results.\n\n3. Quantitative results for both Meta-World and RLBench are strong over baselines. The results in are quite promising.", "weaknesses": "1. Technical novelty is limited. The approach is similar to a lot of other approaches: quantize videos using VQVAE, predict future tokens using transformer, somehow account for missing action tokens.\n\n2. The scope of domains for evaluation domains is somewhat limited. It would be interesting to see if the results hold on other robotic datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and for a positive assessment of our work. Here, we carefully address your concerns as follows:\n\n>1. Technical novelty is limited. The approach is similar to a lot of other approaches: quantize videos using VQVAE, predict future tokens using transformer, somehow account for missing action tokens.\n\nThanks for the comment. While some modules in our method are adapted from previous works, we have innovatively applied and carefully designed these architectures within the novel context of video-based robot decision-making, thus yielding significant contributions to this field. Additionally, We would like to highlight that the main contribution of our paper lies in the proposed pretraining-finetuning paradigm for efficient policy learning with a limited number of robot demonstrations. We achieve this through pre-training on large-scale actionless human videos and finetuning the action-prediction parameters on robot data. Through this paradigm, we can develop a generalist agent capable of processing multimodal information as inputs and performing multiple tasks using limited robot data, which holds significant importance for the community. \n\nWe **are the first** to formulate both video prediction and action learning processes as unified discrete denoising problems, enabling the learning of different modalities (videos in pre-training and actions in fine-tuning) across different stages. Notably, our paper also provides insights to the community, including methods for bridging the gap between human and robot domains by representing videos as discrete latent codes and enhancing policy learning with foresight from predicted videos.\n\n>2. The scope of domains for evaluation domains is somewhat limited. It would be interesting to see if the results hold on other robotic datasets.\n\nThanks for your suggestion! To further validate the superior performance of our method, we train our model using 400 episodes from Language Table [1] blocktoblock dataset (400 out of 200,000 episodes), and finetune the pretrained Octo-Base model using the same dataset for $1e5$ steps. The Language Table is part of the Open-X dataset and is an extremely challenging benchmark due to the random instructions and block locations in each episode. We evaluate their performance on the Language Table environment over 100 episodes. We find that Octo-Base gets a **0%** success rate while our model can achieve a **10%** success rate. Our findings are consistent with a recent work [2].\n\n[1] Corey Lynch et al., Interactive Language: Talking to Robots in Real Time, Arxiv 2022.\n\n[2] W. Zhao et al., VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation, RSS 2024.\n\nWe hope to have addressed all the raised concerns and would be happy to respond to further questions and suggestions. Thank you again for your time and efforts!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NpNZgvRTKT", "reviewer": "Reviewer_wSvy", "summary": "The authors propose an approach that takes advantage of human-object interaction videos to improve policies. Their proposal involves three steps: 1) learn a video tokenizer common to both human and robot videos. 2) Learn a discrete-diffusion model that performs video token denoising on both human and robot videos. 3) Learn a discrete-diffusion model that performs action denoising on only robot data. They show this technique results in better performance as compared to baselines.", "questions": "See above weakness section.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Proposed idea is very interesting. It brings together ideas from self-supervised pre-training and diffusion policies.\n2. They achieve strong performance on a number of manipulation benchmarks outperforming many baselines.\n3. Clear explanation of the methods.", "weaknesses": "1. The paper claims that pre-training on a large dataset of human-object interaction videos without actions is useful for learning actions for robots. However from the experiments it is not clear how much the human videos are adding to the policy performance? There is a significant amount of data difference between the models trained with or without human videos. Yet the performance between the two models is close. How does the model without human data perform given some more robot trajectories? In Fig. 1, for comparison without human videos the drop in performance in marginal. In Fig. 5 seems there is a bigger drop in performance which might be compensated by having multiple camera views in the robot training data. \n\n2. Ego4D videos are very different visually from the tasks and environments shown. It is not clear which tasks from Ego4D are helping in transferring good features for the robotics tasks. How much of the video tokenizer codebook is shared between the human and robot videos. Because the dataset statistics look so different I wonder if enough of the codebook is being shared by the two datasets?\n\n3. Generating egocentric videos is challenging. If the model is good at generating egocentric videos then that in itself is a big contribution. What do the generated videos look like for the Ego4D dataset? Does performance on this task correlate with better downstream policy performance or is the fine-tuning step capable of recovering good policies on video prediction models that are not that good?\n\n4. Another baseline to compare against would be a VideoMAE[1] kind of method to learn the video features on just robot dataset and then do diffusion on that. This will be a good way to make the case for a tokenizer based diffusion model to learn video features. \n\n[1] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. Zhan Tong, Yibing Song, Jue Wang, Limin Wang", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough comments and valuable suggestions! Below, we carefully address your concerns:\n>1. ... how much the human videos are adding to the policy performance? ... How does the model without human data perform given some more robot trajectories?...\n\nThanks for your question. \n\n(1) Regarding the effectiveness of human video, we highlight that since the robot videos are used in pretraining, the predicted videos can directly benefit policy fine-tuning. As a result, even without using human data, VPDD remains a strong baseline as the video predictor is reliable for predicting such \"in-distribution\" scenes. \n\n(2) For experiments in unseen scenes, since the pretraining dataset does not contain videos with the same backgrounds or camera views, VPDD relies heavily on the generalization ability learned from large-scale human datasets. The superior performance in such unseen scenes verifies that large-scale human-data pretraining extracts commonsense knowledge and representations that can be shared between human and unseen robot scenarios. **Furthermore, we ablate the number of human videos used during pre-training and show the performance changes in Figure 1 of the global pdf**. We find that more human videos help increase the success rates, especially boosting the generalizability of unseen scenes.  \n\n(3) It is true that adding additional robot data in these unseen views will improve the performance without human-data pretraining. However, we cannot expect to collect all kinds of variations in robotic tasks, such as visual appearances, camera views, lighting changes, scene structures, embodiments, etc. This is a strong assumption and generally unrealistic. Large-scale human interaction data provides a desirable choice, as it learns a prior of visual representation and dynamic rules to facilitate effective generalization to unseen scenes.\n\n\n\n>2. ...How much of the video tokenizer codebook is shared between the human and robot videos...\n\nThanks for your interesting question. We would like to clarify that although Ego4d videos and robot videos show different visulization, they share substantial knowledge which can benefit robot learning. For example, (1) Human follows the dynamic rules of the physical world to interact with the objects, which is similar to how a robot system interacts with the environment. (2) The human videos cover a wide range of objects, visual features, and tasks, aiding robots in understanding and adapting to real-world situations and facilitating knowledge transfer and versatility in robot systems. As a result, it is not necessary for human videos to come from similar tasks and views. For example, although human videos are egocentric, they can still benefit the prediction of multi-view videos (left, right, front, and overhead), like those in RLBench. \n\nFollowing your advice, we sample 50 clips from both human and robot videos and then computed the shared VQ-VAE codes among them. We find that 1446 codes are shared, accounting for 70.6% of the codebook (with a size of 2048).\n\nTo further validate the knowledge transfer from human to robot, we include an ablation study that excludes robot data during pre-training. **The results presented in Table 1 of the global PDF indicate that VPDD-w/o-robot still maintains a satisfactory level of success rates and remains competitive compared to the baselines**. Specifically, for the 'drawer-open' task, which shares common behaviors and skills seen in human videos, this variant achieves a 100% success rate.\n\n>3. ...What do the generated videos look like for the Ego4D dataset? Does performance on this task correlate with better downstream policy performance or is the fine-tuning step capable of recovering good policies on video prediction models that are not that good?\n\n(1) We present the video prediction results of the Ego4d dataset on our project website. VPDD can also generate dynamically consistent future videos for the human domain. \n\n(2) It is expected that better video prediction will result in improved downstream policy performance, as the representation $z_{\\tilde{{x}}_{0}^{\\rm v}}$ provided to the fine-tuning stage will be more informative. **As shown in Figure 1 of the global PDF, we observe that VPDD's performance improves as more human videos are added**. One important reason is that video prediction performance improves with more data seen during pre-training. \n\n(3) Moreover, our method can also recover good policies on video prediction models that are not that good. We exclude robot videos in pre-training, which could lead to worse performance in robot video prediction. **As shown in Table 1 of the global pdf, VPDD can still achieve a satisfactory success rate**.\n\n\n>4. ... Another baseline to compare against would be a VideoMAE[1] kind of method to learn the video features on just robot dataset ...\n\nThanks for your suggestion. We finetune VideoMAE (ViT-Base pretrained on Kinetics-400) using our robot dataset and its official open-sourced codes, and then leverage the learned video features to learn actions like R3M-Diffusion baseline. We evaluate its final performance on 20 episodes per task in Metaworld. The average success rate is 46%, which is similar to the VC-1 diffusion baseline, which is also a ViT trained by MAE. However, VPDD still outperforms this baseline by about 23% performance improvement.\n\nOverall, we sincerely thank you for reviewing our paper and for your constructive suggestions again. We hope we have resolved all the concerns. We would appreciate it if you could kindly consider raising the score. We are always willing to address any of your further concerns. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Response
We thank all of the reviewers for their time and insightful comments. Furthermore, we are very glad to find that reviewers generally recognized our interesting idea, the superior performance of our method, and the clear presentation of our paper:
### Contributions:
- **Method**: The paper is well-motivated [**v4Bj,kd9D**]. The paper addresses a very timely problem [**ew7s**]. The proposed idea is very interesting [**wSvy**]. The approach is easy for researchers to implement and reproduce results [**ew7s**]. The design of the model and the training pipeline is sensible [**v4Bj**].
- **Presentation**: The paper is well written, clear and easy to understand [**kd9D**]. Clear explanation of the methods [**wSvy**].
- **Experiments**: The paper achieves strong performance on a number of manipulation benchmarks outperforming many baselines [**wSvy**]. Quantitative results for both Meta-World and RLBench are strong over baselines [**ew7s,kd9D**]. The results in are quite promising [**ew7s,v4Bj**]. The experiment results are statistically significant with multiple runs reported accompanied by error bars [**kd9D**].

We thank all the reviewers for their helpful and constructive feedback to improve the quality of our work. We will carefully update our paper in the next version to incorporate the valuable suggestions of the reviewers. In summary, we add the following new experiments to address the reviewers' concerns:

- [**v4Bj, kd9D, wSvy**] We report the performance of VPDD without seeing robot videos in the pretraining stage (see Table 1 of the global pdf). 
- [**v4Bj, wSvy**] We ablate the number of human videos used during pre-training (see Figure 1 of the global pdf).
- [**v4Bj**] The generalizability of VPDD for unseen tasks.
- [**kd9D, ew7s**] New experimental results on language table dataset compared with Octo-Base model.
- [**wSvy**] Video prediction performance on Ego4d domain.
- [**wSvy**] We report the performance of an added baseline of the VideoMAE-based diffusion model.

We hope to have addressed all the raised concerns and would be happy to respond to further questions and suggestions.

Meta-review: The paper proposes an approach to leverage action-less human videos to train robot policies. The paper received unanimous acceptance recommendation from the reviewers. The paper is very well motivated and presents a relatively simple approach with strong performance compared to the baselines. The AC follows the recommendation of the reviewers and recommends acceptance. However, reporting results beyond table-top manipulation would make the paper stronger. The authors are encouraged to incorporate the comments of the reviewers and revise the paper.
Decision: Accept (poster)