Title: Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training
Venue: NeurIPS 2024
Accepted: yes
Keywords: Imitation Learning

Actionless Video Pre-Training

Video-Based Policy Learning

Discrete Diffusion
Abstract: Learning a generalist embodied agent capable of completing multiple tasks poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. In contrast, a vast amount of human videos exist, capturing intricate tasks and interactions with the physical world. Promising prospects arise for utilizing actionless human videos for pre-training and transferring the knowledge to facilitate robot policy learning through limited robot demonstrations. However, it remains a challenge due to the domain gap between humans and robots. Moreover, it is difficult to extract useful information representing the dynamic world from human videos, because of its noisy and multimodal data structure. In this paper, we introduce a novel framework to tackle these challenges, which leverages a unified discrete diffusion to combine generative pre-training on human videos and policy fine-tuning on a small number of action-labeled robot videos. We start by compressing both human and robot videos into unified video tokens. In the pre-training stage, we employ a discrete diffusion model with a mask-and-replace diffusion strategy to predict future video tokens in the latent space. In the fine-tuning stage, we harness the imagined future videos to guide low-level action learning with a limited set of robot data. Experiments demonstrate that our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared to previous state-of-the-art approaches with superior performance.

Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training

Abstract

Learning a generalist embodied agent capable of completing multiple tasks poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. In contrast, a vast amount of human videos exist, capturing intricate tasks and interactions with the physical world. Promising prospects arise for utilizing actionless human videos for pre-training and transferring the knowledge to facilitate robot policy learning through limited robot demonstrations. However, it remains a challenge due to the domain gap between humans and robots. Moreover, it is difficult to extract useful information representing the dynamic world from human videos, because of its noisy and multimodal data structure. In this paper, we introduce a novel framework to tackle these challenges, which leverages a unified discrete diffusion to combine generative pre-training on human videos and policy fine-tuning on a small number of action-labeled robot videos. We start by compressing both human and robot videos into unified video tokens. In the pre-training stage, we employ a discrete diffusion model with a mask-and-replace diffusion strategy to predict future video tokens in the latent space. In the fine-tuning stage, we harness the imagined future videos to guide low-level action learning with a limited set of robot data. Experiments demonstrate that our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared to previous state-of-the-art approaches with superior performance. Our project webpage is available at https://video-diff.github.io/ .

1 Introduction

How do we derive a general-purpose robot agent that can complete a wide variety of tasks? We believe that recent advances in vision and language give us a clue, which delves into pre-training foundation models on extremely large and diverse datasets, followed by fine-tuning on specific domains. For instance, through pre-training on internet-scale datasets [65], large language models [81, 82, 63] and vision models [9, 69, 72] showcase impressive performance on various downstream tasks such as question answering, coding, and image generation. However, unlike general visual language tasks that can exploit copious amounts of data available on the Internet, embodied tasks necessitate high-quality egocentric data in robotics domains for precise control. Collecting such data can be expensive or time-consuming due to the reliance on robot interactions through teleoperation or kinematic solvers [31], and significant gaps in embodiments and dynamics persist when applying them to different robots.

In contrast to the limited availability of robot data, there is a wealth of human interaction videos capturing intricate tasks and varied interactions with the physical world [27]. These videos inherently

encapsulate rich semantic information regarding objects, environmental backgrounds, and handobject interactions across diverse scenarios, making them potentially valuable for acquiring shareable knowledge relevant to embodied tasks.

Motivated by this, many works have emerged to learn various objectives pre-trained on human actionless videos, aiming to capture useful knowledge that can be beneficial for embodied tasks. These approaches involve learning pre-trained image representations [61, 57, 68, 91], trajectory representations [4, 85, 74], reward functions [13, 56] and world models [59, 89]. However, they are still limited to comprehending the dynamic rules of the world or reasoning based on long-term behavior rather than relying solely on step-by-step transitions. We summarize three main

Large-Scale

Interaction

Videos

Limited

Number of

Self-Supervised

Pre-training

Fine-tuning

Visual Control Tasks

Actions for Robot Videos:

{𝑎

!

, 𝑎

"

, 𝑎

#

, ⋯ , 𝑎

$

}

Predicted

Future Videos

Discrete

Diffusion

videos

actions

Stage 1 & 2

Stage 3

{𝑎

!"#

, 𝑎

!"$

, ⋯ , 𝑎

!"%

}

challenges that bottleneck their performance: (i) The domain gap between humans and robots which hinders knowledge transfer; (ii) Complex, diverse and noisy behavior patterns hidden in human videos which are difficult to learn; (iii) Large-scale data from different modalities (e.g., videos, actions, texts) which requires a scalable and high-expressive model architecture to process.

To address these challenges, we propose a V ideo-based P olicy learning framework via D iscrete D iffusion (VPDD). VPDD bridges the visual gap between the human and robot domains by representing these two data types as unified latent representations. Then, VPDD performs video prediction as a pre-training stage with actionless videos, which acquires the commonsense knowledge shared between human and robot interactions, including dynamic rules and behavior patterns (e.g., pick, place, push) to understand and complete tasks. Then, VPDD performs policy learning via a fine-tuning stage with action-labeled robot videos, where VPDD learns to predict actions with foresight from future video predictions. The pre-training stage learns extensive knowledge from human video prediction, and the fine-tuning stage concentrates on training parameters specifically associated with actions. To tackle the challenge of modeling the noisy and complex distribution of large-scale videos while enabling the multi-modal generation of both videos and actions, we leverage the generative capability and flexible architecture offered by discrete diffusion models [2, 29]. We provide an overview of our method in Fig. 1.

In summary, we highlight our contributions as follows. (i) We propose VPDD, a novel pretrainingfinetuning paradigm for learning an actionable policy with limited robot data accessible. This paradigm demonstrates superior ability in transferring valuable knowledge from large-scale actionless human videos to downstream embodied tasks. (ii) We formulate both video prediction and action learning processes as unified discrete denoising problems, showing the supreme effectiveness in handling high-dimensional, multi-modal data. (iii) We conduct thorough experiments using human videos from Ego4D [27], as well as embodied datasets from Meta-World [98] and RLBench [43], showcasing its ability to predict dynamic-consistent future videos. Our actionable discrete diffusion policy also exhibits superior performance compared to previous state-of-the-art approaches [32, 61, 24, 75, 15], encompassing both seen and unseen scenes for multi-task robotic problems.

2 Preliminaries

2.1 Multi-Task POMDP

In this work, we consider a generalist vision-based agent that is capable of addressing multi-task predicaments, where the landscape is characterized by the inherent challenge of acquiring different skills across tasks and partial observability when dealing with image inputs. Given a specific task T ∼ p ( T ) , we further approach the problem as a task-specified Partially Observable Markov Decision Process (POMDP), defined as ( S T , O , A , P T , R T , µ T , γ ) . Here, O is a shared observation space as we use image observations for all tasks. We also assume all tasks share the same action space with the same embodiment.

2.2 Vector Quantized Model

In order to unify the feature space of both human videos and robot videos, we leverage the Vector Quantized Variational Auto Encoder (VQ-VAE) [83] to compress high-dimensional data points into information-rich discretized latent codes. Given a high-dimensional video segment x ∈ R T × H × W × C , the encoder E first converts it to the temporal-spatial features z = E ( x ) = { z m,i,l } ∈ R t × h × w × d , where t × h × w represents the encoded sequence length and is much smaller than T × H × W . Then we transfer the continuous features into discrete space by performing a nearest neighbors lookup in a codebook of embeddings Z = { e j } J j =1 ∈ R J × d to obtain the tokens

where the video tokens z q ∈ R t × h × w × d can be faithfully reconstructed via a decoder, i.e., ˆ x = G ( z q ) . The encoder E , decoder G , and codebook Z can be trained end-to-end via the following loss function L = ∥ x -ˆ x ∥ 1 + ∥ sg[ E ( x )] -z q ∥ 2 2 + β ∥ sg[ z q ] -E ( x ) ∥ 2 2 , where sg denotes stop gradient.

2.3 Discrete Diffusion Model

The discrete diffusion model was first proposed to deal with discrete state space with transitions converging to a binomial distribution [76], and then extended to multinomial diffusion with more options for transition matrices [36, 2]. In this work, we utilize discrete diffusion with the absorbing state for sequence prediction of discrete tokens. Besides J tokens from a codebook, an additional [MASK] token is introduced. We denote x k as a one-hot vector identifying the token index. The forward process from x k -1 to x k follows a Categorical distribution of Q k x k -1 , as

where [ Q k ] m,n = q ( x k = m | x k -1 = n ) ∈ R ( J +1) × ( J +1) is the Markov transition matrix from k -1 to k , which is formulated as:

where α k ∈ [0 , 1] is the probability of retaining the token, and each ordinary token has a probability of γ k to be replaced by [MASK] token, leaving a chance of β k = (1 -α k -γ k ) /J to be diffused. Importantly, due to the property of the Markov chain, we can derive the probability of x k at arbitrary timestep directly from x 0 as

Besides, the posterior of this diffusion process is tractable as q ( x k -1 | x k , x 0 ) = q ( x k | x k -1 , x 0 ) q ( x k -1 | x 0 ) q ( x k | x 0 ) = ( x T k Q k x k -1 )( x T k -1 Q k -1 x 0 ) x T k Q k x 0 . In the reverse process, rather than explic- itly predicting the posterior through a denoising neural network, the x 0 -parameterisation enhances stability and allows for fast inference (by skipping ∆ k steps per iteration). The reverse transition with reparameterisation is formulated as

where the neural network predicts the logits of the target data q ( x 0 ) .

3 Methodology

We commence with the pre-training of our model through future video prediction, enabling the learning of a general dynamic pattern across diverse domains. Subsequently, we fine-tune the model using a limited dataset of robot data for policy learning, leveraging foresight from predicted videos. Our framework is illustrated in Figure 2.

Conv3D Encoder

𝐸

Quantize

(⋅)

Conv3D Decoder

𝐺

Codebook

Large-Scale

Human Videos

Robot Videos

𝑪𝟏 𝑪𝟑

𝑪𝟒 𝑪𝟐

𝑪𝟏

𝑪𝟐

𝑪𝟑

𝑪𝟑

𝑪𝟒

𝑥𝐾

𝑥𝑘

𝑥𝑘-1

𝑥0

𝐌𝐌𝐌

⋯𝐌

⋯

𝐌𝐌𝑪𝟐

⋯ 𝐌

⋯

𝐂𝟏 𝐌𝑪𝟐

⋯ 𝑪𝟒

⋯

𝐂𝟏𝑪𝟑𝑪𝟐

⋯ 𝑪𝟒

⋯

𝑞(𝒙𝑘|𝒙𝑘-1)

𝑝𝜃(𝒙k-1|𝒙𝑘, 𝒚, 𝒍)

'Push and close a drawer'

'Handle and spread cards'

Instructions

𝒍

Historical Frames

𝒚

𝑡

𝑥𝐾

𝑥𝑘

𝑥𝑘-1

𝑥0

𝑞(𝒙𝑘|𝒙𝑘-1)

𝑝𝜃 (𝒙

𝑘-1|𝒙𝑘 , 𝒚, 𝒍)

Instructions

Historical Frames

𝐌𝐌𝐌

⋯𝐌

𝐌⋯𝐌

𝐌𝐌𝑪𝟐

⋯ 𝐌

𝐌 ⋯ 𝐂𝟕

𝐂𝟏 𝐌𝑪𝟐

⋯ 𝑪𝟒

𝐂𝟕⋯𝐂𝟕

𝐂𝟏𝑪𝟑𝑪𝟐

⋯ 𝑪𝟒

𝐂𝟕⋯𝐂𝟖

video

tokens

Stage 1: Video Tokenizing

Stage 2: Pre-Training with Mixed Data

Stage 3: Fine-Tuning with Robot Data

CLIP

𝑷𝜽𝟏

Denoise

Video

Tokens

Perceiver IO

Transformer

𝑷𝜽𝟐

Denoise

Action

Tokens

GPT2

Transformer

Predicted

Future Videos

𝑷𝜽𝟏

Denoise

Video

Tokens

Perceiver IO

Transformer

𝑷𝜽𝟐

Denoise

Action

Tokens

GPT2

Transformer

𝑡

CLIP

execute

ΔT =

0.2, -0.1,0.15

gripper = [1]

Predicted Actions

෥

𝒙0

v

෥

𝒙0

a

𝒙 0

v

𝐌

𝐌

𝐌

𝐌

𝐌

𝐌

𝐌

𝐌

3.1 Data Preparing and Tokenizing

Robot Data Collection. We use the rule-based script policy to rollout 20 expert demonstrations for each task in Meta-World [98]. We also run VPDD on 16 tasks from RLBench [43], a more challenging benchmark involving 6-Dof manipulation that necessitates multi-view images from a 3D scene to select actions. Following the multi-view manipulation frameworks [24, 75], we utilize the script motion-planner to collect 10 demonstrations for each task. Each demonstration from robot-data is formulated as τ i = { v 1 , a 1 , · · · , v t , a t , · · · , v T , a T } with v t = [ o t -I +1 , · · · , o t -1 , o t ] , where we set I = 4 throughout the paper and a denotes the actions. In the context of Meta-World, o t represents a single-view RGB image at timestep t . For RLBench, o t = { o front t , o left t , o right t , o wrist t } comprises 4 RGB multi-view images (i.e., front, left shoulder, right shoulder, and wrist). Consequently, in RLBench, v t is formulated as v t = { v front t , v left t , v right t , v wrist t } .

Human Data Collection. As for human data collection, we obtain untrimmed videos from the open-sourced Ego4D dataset [27], which contains massive-scale human-object interactions of various durations ranging from 5 seconds to 7 hours. We filter out videos without human-object interaction and segment each video into short clips with 8-frame intervals [60]. Thus each video is represented as τ i = { v 1 , v 2 , · · · , v n } , where v t denotes a clip containing 4 frames. This approach yields a total of 996,177 clips of human videos, comprising approximately 4M frames. More details on data collection and processing are given in §C.1.

VQ-VAE Encoding. To extract useful features from raw videos in both human and robot domains, a conventional approach is to directly encode them into an embedding space using pre-trained vision models like ViT. However, these models are usually specifically trained on image dataset [71], posing a significant challenge due to the domain gap with our interaction videos. Thus, we leverage VQ-V AE to compress the diverse and noisy videos into discrete latent codes, which provide a unified codebook for mixed videos and alleviate the domain gaps between human and robot videos. Formally, we adopt the VQ-VAE architecture introduced by VideoGPT [94] for encoding videos into a discrete latent space. The codebook comprises 2048 codes, each represented by 256-dimensional embeddings. The encoder architecture consists of a series of 3D convolutions that downsample by a factor of 4 over space-time (resulting in a 64 × total reduction), followed by 6 attention residual blocks. Consequently,

each video clip v t ∈ { τ i } is embedded into latent codes e t . The architecture for the decoder is the reverse of the encoder, featuring attention residual blocks followed by an equivalent number of 3D transposed convolutions for upsampling over space-time. The VQ-VAE is pre-trained on large-scale videos and remains fixed in the subsequent processes, providing flexibility for various downstream utilization methods.

Action Discretizing. For subsequent pre-training and fine-tuning, we process the collected continuous actions via uniform action discretization [46, 11]. In the case of Meta-World, the action space is a 2-tuple consisting of the change in the 3D space of the end-effector followed by a normalized torque that the gripper fingers should apply. Here all the continuous dimensions are discretized into 48 bins uniformly. Thus, the robot action can be represented using ordinals of the discrete bins as a 4 integer number. For RLBench, an action consists of the gripper open state and 6-DoF pose including position and rotation. The position is discretized into 360 bins, and rotation is discretized into Euler angles as 1-degree bins for each of the 3 rotation axes [75]. Gripper open state is a binary value.

3.2 Video Prediction via Unified Discrete Diffusion

Extracting general patterns useful for downstream decision-making from large-scale in-the-wild human videos is challenging, primarily because of the absence of labeled actions and the complexity of the underlying structure of human interactions. Different from previous ways of learning a visual representation, we propose a novel objective to further unleash the representation and temporal modeling ability of diffusion models. Specifically, after obtaining discrete tokens from VQ-VAE encoding, we train a unified discrete diffusion model on the latent space via a self-supervised objective. This objective involves predicting future videos based on observed historical videos for both humans and robots, while masking action tokens. Benefiting from the proposed objective and the x 0 -parameterisation of discrete diffusion, the diffusion model is incentivized to capture both the high-level temporal dynamics and the low-level visual commonalities between historical and future videos at each diffusion step. Then the acquired knowledge can be leveraged to guide action denoising at each step.

Unified Transition Matrix. The presence of a transition matrix determines the nature of the discrete diffusion model [2]. While the original discrete diffusion is limited to one modality, drawing inspiration from UniD3 [37], which enhances the transition matrix to encompass both images and text, we construct a unified transition matrix to capture global connections between the two modalities-videos and actions. The matrix [ Q k ] m,n below illustrates the unified transition process:

Q

k

=

































α

k

+

β

k

β

k

· · ·

β

k

0

0

· · ·

0

0

β

k

α

k

+

β

k

· · ·

β

k

0

0

· · ·

0

0

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

β

k

β

k

· · ·

α

k

+

β

k

0

0

· · ·

0

0

0

0

· · ·

0

α

k

+

β

∗

k

β

∗

k

· · ·

β

∗

k

0

0

0

· · ·

0

β

∗

k

α

k

+

β

∗

k

· · ·

β

∗

k

0

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

0

0

· · ·

0

β

∗

k

β

∗

k

· · ·

α

k

+

β

∗

k

0

γ

k

γ

k

· · ·

γ

k

γ

k

γ

k

· · ·

γ

k

1

































,

where β k and β ∗ k are the probabilities of a token to be replaced by any other accessible tokens in different modalities. The dimension of Q k is ( J + J ∗ + 1) × ( J + J ∗ + 1) , where J and J ∗ are the number of tokens in different modalities, i.e., J is the size of codebook in VQ-VAE and J ∗ is the number of action classes in discretization. The sum of each column in this transition matrix is one to preserve probability mass. Mathematically, we have β k = (1 -α k -γ k ) /J and β ∗ k = (1 -α k -γ k ) /J ∗ . All the mass of the stationary distribution falls on the [MASK] token, which satisfies the prerequisite for a discrete diffusion model transition matrix [2]. The details of the diffusion process are provided in §A.1.

Unified Objective. We cast both video prediction and action learning as a conditional generative problem, and the goal is to maximize E τ ∼∪ i τ i [ log p θ ( x 0 ( τ ) ∣ ∣ y ( τ ) , l ] . Here x = [ x v , x a ] , where x v = [ e t + h +1 , · · · , e t + h + M ] represents future video segments with M clips, and x a = [ a t , · · · , a t + H -1 ] denotes action sequences of H steps. y = e t serves as the condition containing historical video tokens. l is the language instructions describing current tasks. In practice, we train two separate denoising networks, namely p θ 1 ( x v k -1 | x k , y , l ) and p θ 2 ( x a k -1 | x k , z ˜ x v 0 , l ) , to learn videos and actions, respectively. Here, z ˜ x v 0 represents the hidden representation of predicted future videos given by p θ 1 at each diffusion step, which is utilized to guide action learning. Formally, ˜ x v 0 = Softmax ( MLP ( z ˜ x v 0 )) .

As the actions are absent during the pre-training stage, we mask x a and freeze p θ 2 , as illustrated in Figure 2. The network is trained to minimize the following variational lower bound (VLB) [76, 29]:

where

L K is a constant number that can be ignored in the training, as the prior distribution p ( x K ) is fixed:

Model Architecture. As in Eq. (8), the neural network p θ 1 receives x k , history y , language l and the diffusion timestep k as inputs. These inputs are individually embedded into embeddings h of size d via separate MLPs f , depicted as:

where language instructions l is encoded with CLIP's language encoder [66]. Afterwards, the embeddings are formulated as input tokens as h tokens = LN( h Ti × [ h l , h Ti , h x k , h y ] + E pos ) , where E pos is the positional embedding, and LN denotes layer normalization [3] for stabilizing training. The input sequence that represents a video can be extremely long so a standard Transformer with O ( n 2 ) complexity is hard to fit. We adopt Perceiver Transformer [41] to tackle this problem, as it has been widely utilized for modeling long sequences [75, 24]. Perceiver is a latent-space Transformer, where instead of attending to the entire input, it computes cross-attention between the input and a much smaller set of latent vectors (which are randomly initialized and trained). These latents are encoded with self-attention layers, and are cross-attended with the input to match the size for the final outputs. More details about the Perceiver Transformer are referred to §A.2.

3.3 Learning to Act via Few-Shot Fine-Tuning

During the fine-tuning stage, we leverage a limited dataset of robot data, including both videos and actions, for rapid adaptation. Both x v and x a attend to training the diffusion model. Given that p θ 1 has been trained sufficiently to capture fruitful information to predict future videos from history, we freeze p θ 1 and solely tune parameters of p θ 2 to minimize L vlb . As expressed in Eq. (8), the input of p θ 2 consists of x k , language l , hidden representation z ˜ x v 0 , and diffusion timestep k . In this case, we are tasked with predicting a sequence of action tokens x a 0 , considerably shorter than video-token sequence x v 0 , so we employ GPT2 [67] Transformer to process tokens embedded with MLPs. GPT2 has demonstrated an impressive ability to solve multi-task problems and model multimodal distributions. The model architecture of p θ 2 closely resembles that of MTDIFF-P [32]. More details of our method can be found in Appendix A.3.

4 Related Work

Robot Learning from Human Videos. Leveraging human videos [26, 23, 18, 27] for policy learning is promising to extract commonsense knowledge from human activities, which can be shared to embodied scenarios that suffer from scarce robot data [17, 22]. Since the human data is actionless and the domain gap between humans and robots exists, a main branch of research employs human video to learn shareable visual representations [12, 47] via time-contrastive [57], video-language alignment [61, 48], value function [8], and perceptual skills [40, 52]. Visual affordance like human-object interaction hotspots [54, 25] and the post-grasp trajectory [5] are also helpful for embodied agents in goal-conditioned imitation. Alternative methods involve extracting hand trajectories [4, 85] or keypoints [93] to transfer plans to robots. Different from the above methods, we eliminate the domain gaps by learning video tokens and representations for video prediction, which implicitly captures visual features, affordances, and long-term plans. Other works attempt to infer actions from videos via inverse kinematics [6, 50], whereas we learn action prediction through policy fine-tuning without external models.

Pretraining for Generalized Policy Learning. Early works of policy adaptation emerged in meta-RL [30, 99], while the pre-training and fine-tuning environments are assumed to be similar. Leveraging the transformer architecture, works perform pre-training in multi-task datasets by optimizing the multi-task policy [51, 70, 79, 80] or self-supervised objectives [78, 73, 59]. In tasks involving visual observations, methods adopt visual tokens for transformer-based multi-task policy learning [10, 11] and adaptation [53]. Additionally, some studies pre-train a reward function through video-language correspondence [13, 56] or diffusion models [21, 38] for downstream RL training. Different from the previous Transformer and continuous diffusion frameworks, our work first integrates visual tokens with discrete diffusion to predict consistent videos and actions simultaneously. Concurrently, GR-1 [88] utilizes human data to pre-train a GPT-style architecture for predicting future observations. In contrast, we perform video prediction instead of step-by-step image prediction using a unified discrete diffusion architecture.

Diffusion Models for RL. Diffusion models are a powerful family of generative models [72, 69] that can be categorized into continuous Gaussian diffusion models and discrete diffusion models that handle discrete visual tokens or symbols [29]. Continuous diffusion models have found extensive applications as multi-modal policies [87, 64, 16], environmental dynamics [95], and planners to generate action [45, 90] or state sequences [1, 14], guided by desired properties. Several methods also extend the diffusion models for multi-task learning [32, 62, 19] with low-dimensional states, while we specifically address the more challenging image-based setting. UniPi [20] and its following work [100] are related to our method by performing video generation via continuous diffusion, while we adopt a more flexible architecture with discrete diffusion to seamlessly connect the pretraining and fine-tuning stages, without relying on a task-specific inverse-dynamic model for acting. Additionally, we train the model on video tokens that maintain temporal consistency, while UniPi relies on super-resolution to improve the time consistency of generated frames.

5 Experiments

5.1 Bnechmarks and Baselines

After the video pertaining in Ego4D [27], we use the following robotic benchmarks to evaluate our method.

Meta-World. The Meta-World benchmark [98] contains 50 distinct manipulation tasks that require a Sawyer robot to interact with various objects with different shapes, joints, and connectivity. The action is the 3D position movements of the robot's end effector and the gripper openness. We follow recent works [96, 77] to extend the original environment to a more challenging setting with random goals, and refer to it as MT50-rand. We train the policy with 20 demonstrations per task, and report the average success rates on 50 evaluation episodes per task.

RLBench. RLBench [43] is a more challenging 3D manipulation benchmark with diverse tasks concerning interactions with a wide range of objects. We select 16 tasks from RLBench to evaluate our method, where each task has several possible variations, such as the shapes, colors, sizes and positions of objects. The input observations are captured from four RGB cameras positioned at the front, left shoulder, right shoulder, and wrist. The action is an 8-dimensional vector including 3-dimensional transitions, 4-dimensional quaternion, and a binary value about gripper openness. We follow the convention by using macro steps [42], which are key turning points in the action trajectory where the gripper changes its state (open/close) or the joint velocities approach to zero. We train the policy with 10 demonstrations per task and report average success rates on 25 evaluation episodes per task.

Baselines for Meta-World. We compare the proposed method VPDD with the following baselines: (i) R3M-Diffusion is a discrete diffusion model sharing identical architecture with p θ 2 , leveraging the R3M [61] ResNet50 encoder to encode images as input. R3M is also trained on Ego4D videos via a contrastive learning objective and stands as the state-of-the-art (SOTA) visual representation specifically designed for manipulation tasks; (ii) VC-1-Diffusion utilizes VC-1 [58] encoder (ViT-L) to extract image representations, which is also trained on large-scale egocentric videos [27] and ImageNet [71] using Masked Auto-Encoding [33]. (iii) MTDIFF-P [32] is the SOTA method for multi-task RL, which employs a continuous diffusion model with a Transformer architecture. Since it is designed to handle state-based input, we employ the R3M encoder to extract hidden embeddings from images, which are then fed into MTDIFF-P; (iii) Video-MTDT is an extension of Decision Transformer (MT) [15], learning from multi-task data with video tokens as input; (v) VPDD-w/o.-

human excludes human videos during pre-training, remaining other parts unchanged. This baseline helps to ablate the effect of pre-train on large-scale human videos; (vi) SODA [39] is a recently proposed diffusion-based representation method that employs an encoder to generate representations z from input images to support the denoising process. We pre-train the encoder by employing the video prediction objective on the same dataset and subsequently feed the learned z into p θ 2 during fine-tuning. See more details in §B.

Baselines for RLBench. Learning policies for RLBench is more challenging as it requires understanding the 3D scene structure for predicting the 6D poses of end-effectors. The baselines used in Meta-World all fail in the benchmark since they are disadvantaged with single-view observations. In contrast, VPDD can predict multi-view images, implicitly recovering the 3D geometry in manipulation. Thus, we use the following SOTA imitation architectures designed for 3D manipulation: (i) RVT [24] stands as the SOTA method, initially re-rendering visual observations into orthographic projections of cube views and subsequently predicting the next move based on these multi-view projections.; (ii) PERACT [75] encodes RGB-D images into voxel grid patches for 3D representation and predicts the action using the Perceiver Transformer.

5.2 Results Analysis

Meta-World

0

10

20

30

40

50

Success Rate (%)

56.57±0.41

52.93±0.55

38.59±0.74

48.61±0.62

44.72±0.6

30.27±0.48

45.90±0.67

VPDD (Ours)

VPDD-w/o-human

MTDiff-p

R3M-Diffusion

VC-1-Diffusion

Video-MTDT

SODA

Question 1. Does our pre-trained diffusion model (i.e., p θ 1 ) capable of generating dynamic-consistent future videos?

Although our primary contribution is not about video generation, it remains crucial to predict consistent future videos to aid in policy fine-tuning. The predicted raw videos are visually depicted in Fig. 3, with more video samples including real robots (trained by RoboSet data [7]) accessible at https://video-diff.github.io .

These videos are reconstructed from predicted video tokens by using the decoder of VQ-VAE. After pre-training, the video-prediction model p θ 1 demonstrates the capability to generate dynamically consistent single-view videos for Meta-World and multi-view videos for RLBench. Furthermore, we computed the Frechet Video Distance (FVD) score [34, 86] averaged across frames on 32 generated video samples from the Meta-World dataset. From the results in Table 5.2,

we observe that the FVD score of VPDD is considered acceptable and even lower than the hierarchical video synthesizing method UniPi [20] (score reported from its original paper). We attribute the

capability of generating high-quality videos to the well-trained video codebook and the proposed discrete diffusion model learned from large-scale human data.

Question 2. How does VPDD compare to other offline baselines for vision-based multi-task decisionmaking?

To evaluate the learned policy after fine-tuning, we take the first action generated by p θ 2 to interact with the environment. The results on Meta-World and RLBench are referred to Fig. 4 and Table 1 respectively, yielding the following key findings: (i) VPDD outperforms other SOTA methods in success rate by a large margin. For Meta-World, VPDD performs the best across 50 tasks with random goals. For RLBench, VPDD even outperforms the SOTA imitation architectures based on voxel and multi-view representations that are carefully designed for 3D manipulation, which usually require point clouds or 3D world rendering for scene understanding. Notably, VPDD achieves a remarkable success rate on put in cupboard , insert peg , stack cups and place cups , while both RVT and PERACT struggle on these challenging tasks. This verifies the efficacy of video pre-training for few-shot policy fine-tuning; (ii) According to Fig. 4, VPDD obtains a 6 . 9% relative improvement through pre-training on

Table T exture

Camera Pos

0

10

20

30

40

50

Success Rate (%)

VPDD (Ours)

VPDD-w/o-human

R3M-Diffusion

both human and robot videos compared with VPDD-w/o.-human. Furthermore, VPDD surpasses R3M and VC-1 with a notable 16 . 4% and 26 . 5% higher success rate, demonstrating the potential of our diffusion representation via video prediction; (iii) R3M-Diffusion outperforms MTDIFF-P which employ R3M encoder with continuous diffusion architecture by 26 . 0% , and Video-MTDT with Transformer architecture by 60 . 6% . This highlights the superior capacity of discrete diffusion models compared to other model architectures.

Question 3. How does VPDD generalize to unseen scenes?

As suggested in recent works [92, 97], we evaluate the generalization ability of our model in the two most challenging settings, i.e., camera view and visual background. Specifically, we alter the camera position and table texture in the visual scene of Meta-World to assess the generalization ability of our method. According to Fig. 5, VPDD exhibits superior generalizability, attributed to the training of the diffusion representation on large-scale diverse human videos. Regarding the shift in camera position, VPDD outperforms the VPDD-w/o.-human by 63 % and R3M by 252 % . Moreover, we show that VPDD can generalize on different tasks with a competitive performance, demonstrating its potential to serve as a foundation model. Detailed results and settings can be found in §C.3.































Question 4. Can VPDD maintain satisfactory performance when provided with fewer robotic demonstrations?

Leveraging the large-scale video pre-training, VPDD can learn the policy using only a small number of demonstrations. To validate the sample efficiency of our method, we conduct an ablation study on the number of demonstrations used in the fine-tuning stage. The results, depicted in Fig. 6, reveal that the performance of VPDD exhibits linear growth after training on 5 or more demonstrations, indicating the potential for VPDD to achieve better performance with increased demonstration data. Moreover, VPDD maintains a comparable success rate even when only 1 demonstration is used in the fine-tuning process.

Question 5. How does VPDD perform when trained on different amounts of human data?

0

1/3

2/3

1

Number of Human Data

35

40

45

50

55

Success Rate (%)

Original MT50

Shited Camera Position

The superior generalizability of VPDD, which is validated in Fig. 5, stems from large-scale human-data pertaining, which effectively extracts commonsense

knowledge and representations that can be shared between human and unseen robot scenarios. To further investigate the effects of human-data pre-training, we ablate the number of human videos used during pre-training and present the performance changes in Fig. 7. Our findings indicate that an increased number of human videos enhances success rates, particularly in improving generalizability by a large margin.

Question 6. Does VPDD outperforms other diffusion- based representation learning method?

To verify the representation capability inherent in our unified discrete diffusion framework, we reproduce SODA [39], which serves as a strong diffusion representation method. As shown in Fig. 4, VPDD outperforms SODA in the context of policy fine-tuning. We hypothesize that VPDD provides coarse-to-fine representations (i.e., z ˜ x v 0 ) throughout the action-denoising steps from K → 1 , exactly encapsulating useful information the denoising network focuses on at each step [35]. In contrast, SODA produces representation z that remains constant across all steps during action denoising.

6 Conclusion

We propose VPDD, a video-based policy learning framework via discrete diffusion. With a VQVAE tokenizer, we bridge the gap between human and robot videos by a discrete latent codebook. We leverage a unified discrete diffusion for pre-training on large-scale actionless mixture videos and subsequent fine-tuning the policy on a limited number of robot demonstrations. Experiments demonstrate that VPDD achieves superior performance on a variety of challenging manipulation tasks and showcases impressive generalization ability benefited from human video prediction. More Discussions of our work are given in §D.

Acknowledgments

This work is supported by Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102) and National Natural Science Foundation of China (62322603 & 62306242). We also thank the anonymous reviewers for their valuable suggestions.

References

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In International Conference on Learning Representations , 2023.

Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, 2023.

Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR , abs/1607.06450, 2016.

Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. 2022.

Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13778-13790, 2023.

Homanga Bharadhwaj, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint arXiv:2302.02011 , 2023.

Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In First Workshop on Out-of-Distribution Generalization in Robotics at CoRL 2023 , 2023.

Chethan Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh, Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic offline RL from internet videos via value-function pre-training. In NeurIPS 2023 Foundation Models for Decision Making Workshop , 2023.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robocat: A selfimproving foundation agent for robotic manipulation. arXiv preprint arXiv:2306.11706 , 2023.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, and et al. Rt-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems , 2023.

Kaylee Burns, Zach Witzel, Jubayer Ibn Hamid, Tianhe Yu, Chelsea Finn, and Karol Hausman. What makes pre-trained visual representations successful for robust manipulation? arXiv preprint arXiv:2312.12444 , 2023.

Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. arXiv preprint arXiv:2103.16817 , 2021.

Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Representations , 2023.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 , 2023.

Open X.-Embodiment Collaboration. Open x-embodiment: Robotic learning datasets and RT-X models. CoRR , abs/2310.08864, 2023.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 43(11):4125-4141, 2021.

Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via behaviorcustomisable diffusion model. In International Conference on Learning Representations , 2024.

Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Neural Information Processing Systems , 2023.

Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. In Neural Information Processing Systems , 2023.

Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@ CoRL2023 , 3:5, 2023.

Kuan Fang, Te-Lin Wu, Daniel Yang, Silvio Savarese, and Joseph J Lim. Demo2vec: Reasoning object affordances from online videos. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2139-2147, 2018.

Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. RVT: Robotic view transformer for 3d object manipulation. In 7th Annual Conference on Robot Learning , 2023.

Mohit Goyal, Sahil Modi, Rishabh Goyal, and Saurabh Gupta. Human hands as probes for interactive object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3293-3303, 2022.

Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision , pages 58425850, 2017.

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18995-19012, 2022.

Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.18259 , 2023.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10696-10706, 2022.

Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Metareinforcement learning of structured exploration strategies. Advances in neural information processing systems , 31, 2018.

Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning , pages 8387-8406. PMLR, 2022.

Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. In Neural Information Processing Systems , 2023.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems , 2021.

Minghui Hu, Chuanxia Zheng, Zuopeng Yang, Tat-Jen Cham, Heliang Zheng, Chaoyue Wang, Dacheng Tao, and Ponnuthurai N. Suganthan. Unified discrete diffusion for simultaneous vision-language generation. In International Conference on Learning Representations , 2023.

Tao Huang, Guangqi Jiang, Yanjie Ze, and Huazhe Xu. Diffusion reward: Learning rewards via conditional video diffusion. arXiv preprint arXiv:2312.14134 , 2023.

Drew A Hudson, Daniel Zoran, Mateusz Malinowski, Andrew K Lampinen, Andrew Jaegle, James L McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottleneck diffusion models for representation learning. arXiv preprint arXiv:2311.17901 , 2023.

Mingxiao Huo, Mingyu Ding, Chenfeng Xu, Thomas Tian, Xinghao Zhu, Yao Mu, Lingfeng Sun, Masayoshi Tomizuka, and Wei Zhan. Human-oriented representation learning for robotic manipulation, 2023.

Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In International Conference on Learning Representations , 2022.

Stephen James and Andrew J Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters , 7(2):1612-1619, 2022.

Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters , 5(2):3019-3026, 2020.

Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew J Davison. Coarse-to-fine qattention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13739-13748, 2022.

Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , 2022.

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems , 34:12731286, 2021.

Ya Jing, Xuelin Zhu, Xingbin Liu, Qie Sima, Taozheng Yang, Yunhai Feng, and Tao Kong. Exploring visual pre-training for robot manipulation: Datasets, models and methods. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 11390-11395. IEEE, 2023.

Siddharth Karamcheti, Suraj Nair, Annie S. Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. In Robotics: Science and Systems (RSS) , 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In International Conference on Learning Representations , 2024.

Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances in Neural Information Processing Systems , 35:27921-27936, 2022.

Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion-based task execution. arXiv preprint arXiv:2312.11598 , 2023.

Xingyu Lin, John So, Sashwat Mahalingam, Fangchen Liu, and Pieter Abbeel. Spawnnet: Learning generalizable visuomotor skills from pre-trained networks, 2023.

Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3282-3292, 2022.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems , 32, 2019.

Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. LIV: Language-image representations and rewards for robotic control. In International Conference on Machine Learning , volume 202, pages 23301-23320, 2023.

Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value-implicit pre-training. In International Conference on Learning Representations , 2023.

Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Information Processing Systems , 36, 2024.

Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. 2023.

Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. EmbodiedGPT: Vision-language pre-training via embodied chain of thought. In Neural Information Processing Systems , 2023.

Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning , 2022.

Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-rl. 2023.

OpenAI et al. Gpt-4 technical report, 2023.

Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In International Conference on Learning Representations , 2023.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. CoRL , 2022.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li FeiFei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115:211 - 252, 2014.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-toimage diffusion models with deep language understanding, 2022.

Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning , pages 19561-19579. PMLR, 2022.

Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. In 6th Annual Conference on Robot Learning , 2022.

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In 6th Annual Conference on Robot Learning , 2022.

Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015.

Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parametercompositional multi-task reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor. SMART: Self-supervised multi-task pretraining with control transformers. In International Conference on Learning Representations , 2023.

Adrien Ali Taiga, Rishabh Agarwal, Jesse Farebrother, Aaron Courville, and Marc G Bellemare. Investigating multi-task pretraining and generalization in reinforcement learning. In International Conference on Learning Representations , 2023.

Garrett Thomas, Ching-An Cheng, Ricky Loynd, Felipe Vieira Frujeri, Vibhav Vineet, Mihai Jalobeanu, and Andrey Kolobov. Plex: Making the most of the available data for robotic manipulation pretraining. In Conference on Robot Learning , pages 2624-2641. PMLR, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302.12422 , 2023.

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. arXiv preprint arXiv:1808.06601 , 2018.

Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations , 2023.

Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations , 2024.

Jialong Wu, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with in-the-wild videos for reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Zhou Xian, Nikolaos Gkanatsios, Theophile Gervet, Tsung-Wei Ke, and Katerina Fragkiadaki. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning , 2023.

Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv:2203.06173 , 2022.

Annie Xie, Lisa Lee, Ted Xiao, and Chelsea Finn. Decomposing the generalization gap in imitation learning for visual robotic manipulation. arXiv preprint arXiv:2307.03659 , 2023.

Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj, Samarth Sinha, and Animesh Garg. Learning by watching: Physical imitation of manipulation skills from human videos. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 7827-7834. IEEE, 2021.

Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers, 2021.

Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In International Conference on Learning Representations , 2024.

Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems , 33:4767-4777, 2020.

Sizhe Yang, Yanjie Ze, and Huazhe Xu. Movie: Visual model-based policy adaptation for view generalization. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning , pages 1094-1100. PMLR, 2020.

Haoqi Yuan and Zongqing Lu. Robust task representations for offline meta-reinforcement learning via contrastive learning. In International Conference on Machine Learning , pages 25747-25759. PMLR, 2022.

Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377 , 2024.

A The Details of VPDD

A.1 Discrete Diffusion

Discrete diffusion models with a mask-and-replace strategy generate sequences from [MASK] tokens. In this paper, they are trained by sampling a sequence x 0 = [ x v , x a ] , masking tokens according to a linear schedule [29] that corresponds to increasing the probability of being in the absorbing state linearly over time, and learning to predict the masked tokens given language l and historical videos. Specifically, for the proposed unified transition matrix Q k in Eq. (6), the computation of diffusion process q ( x k | x 0 ) in Eq. (4) can also be obtained in the following closed-form [37]:

x [M] = x ← argmax x = J + J ∗ and α k , β k , β ∗ k , γ k are the corresponding cumulative product. (11)

The reverse process predicts ˜ x v 0 and ˜ x a 0 by training denoising network p θ 1 (˜ x v 0 | x k , y , l ) and p θ 2 (˜ x a 0 | x k , z ˜ x v 0 , l ) , respectively. Then the forward process is used to compute p θ 1 ( x v k -1 | x k , y , l ) as expressed in Eq. (12) and p θ 2 ( x a k -1 | x k , z ˜ x v 0 , l ) as expressed in Eq. (13).

A.2 Perceiver Transformer

We use the Perceiver Transformer [41] to encode extremely long input sequences, which improves the computation efficiency. We maintain a set of latent vectors z Q of dimensions R 2048 × 256 which are randomly initialized. Then we compute cross attention between the input sequence and z Q . The process is illustrated in Fig. 8. z Q are processed with 6 self-attention layers, and then decoded into output with the same dimension space of input via cross attention.

Encode

Cross atten

Q

K

V

Process

×L

Self atten

Q

K

V

Decode

Cross atten

K

V

Q

Latents

Input

Output

A.3 Implementation Details

In this section, we give the pseudocodes of pre-training and fine-tuning in Alg. 1 and Alg. 2 respectively. Then we describe the details of the training process, architecture and hyperparameters:

Following previous works [36], we sample diffusion timestep k ∼ q ( k ) with a importance sampling strategy, where q ( k ) ∝ √ E [ L 2 vlb ] .

We set batch size as 20 for pre-training and 40 for fine-tuning. We train our model using Adam optimizer [49] with 2 e -4 learning rate for 2 e 6 training steps.

For pre-training, we represent our denoising network p θ 1 as a Perceiver Transformer described in §A.2. MLP f t , which processes the language embeddings given by the CLIP encoder, MLP f y , which processes the historical video tokens, and MLP f x k are 2-layered MLPs (prepended by a layer norm [3] and with Mish activation). MLP f Ti , which processes diffusion timestep k , is a 2-layered MLP (prepended by a Sinusoidal embedding and with Mish activation). Finally, we use a linear layer to project the encodings given by the Perceiver Transformer into the original dimension space of the input.

For fine-tuning, we represent our denoising network p θ 2 as a GPT2 Transformer. Similar to the architecture of p θ 1 , we first use MLPs with Mish activation to project different inputs into the same hidden dimension space, and then recover the encodings given by the GPT2 Transformer into the original dimension space via a linear layer.

We choose the sequence length H = 4 and M = 1 for future actions and videos prediction.

We set h = 20 which means that we predict future videos after 20 steps.

We set diffusion timesteps K = 100 .

We run all the experiments on a single RTX 3090 machine.

B The Details of Baselines

We describe the implementation details of baselines used in our experiments as follows:

R3M-Diffusion . We borrow the pre-trained R3M encoder from https://github.com/ facebookresearch/r3m , and leverage the encoder to encode sing-view RGB images in MetaWorld and multi-view RGB images in RLBench. Thus, we skip the pre-training process and directly train our discrete diffusion model on the robot data. The model architecture and hyper-parameters of R3M-Diffusion are almost the same as p θ 2 . The encoded hidden representation encoded by R3M is denoted as z R 3 M , then the denoising network can be written as p θ ( x a k -1 | x k , z R 3 M , l ) .

MTDIFF-P . We borrow the official codes of MTDIFF-P from https://github.com/ tinnerhrhe/MTDiff . In order to process high-dimensional images instead of low-dimensional states, we leverage the R3M encoder in R3M-Diffusion to obtain the visual representations.

VIDEO-MTDT . We use the language l to indicate tasks, which are encoded as a vector with size 2048 by the same CLIP encoder used in VPDD. We take the video tokens used in VPDD as the states. Then we follow the implementation from https://github.com/kzl/ decision-transformer/ to train Video-MTDT on the limited robot data.

VPDD-w/o-human . During the pre-training stage of VPDD, we remove the human videos and only use the robot videos for training.

SODA . Following the SODA paper [39] and open-sourced codes from https://github.com/ FutureXiang/soda , we first maintain an encoder E SODA equipped with the same Perceiver Transformer in p θ 1 . Then during the pre-training stage, a representation z is first encoded by E SODA conditioning on e t and l , i.e., z = E SODA ( e t , l ) . Then x k -1 is obtained via the following process:

where Attn is an attention operation [84, 55] where the queries are formed from x , the keys and values from y . The encoder is trained end-to-end and to minimize the same loss term L vlb . During the fine-tuning stage, z ˜ x v 0 is replaced of z output by E SODA . The model architecture and other hyper-parameters during fine-tuning remain the same.

Algorithm 1 Pre-Training Stage of VPDD

Initialize : given unified transition matrix { Q k } , well-trained VQVAE, training iterations N , initial network parameters θ 1 and θ 2 , learning rate η .

10:

1: for video clips v t in { τ i } do 2: e t ← VQVAE-Encoder ( v t ) 3: end for 4: for n = 1 to N do 5: Sample a batch B = ( x v , x a , y , l ) from human and robot videos, where x a ← ϕ 6: Sample diffusion timestep k from [1 , K ] with importance sampling strategy 7: x k ← q ( x k | x 0 ) , where x 0 = [ x v , x a ] ▷ Eq. (11) and (4) 8: L vlb = { L 0 if k = 1 , L k -1 otherwise. ▷ Eq. (6) 9: θ 1 ← θ 1 -η ∇ θ 1 L ▷ Adam optimizer end for

Algorithm 2 Fine-Tuning Stage of VPDD and Evaluation

# Fine-Tuning Process

Initialize : given unified transition matrix { Q k } , well-trained VQVAE, training iterations N , pre-trained network parameters θ 1 and initial parameters θ 2 , learning rate η .

1: for video clips v t and actions a t in { τ i } do

2: e t ← VQVAE-Encoder ( v t ) , a t ← Discretize ( a t )

3: end for

4: for n = 1 to N do

5: Sample a batch B = ( x v , x a , y , l ) from videos and discretized actions dataset.

6: Sample diffusion timestep k from [1 , K ] with a importance sampling strategy

7: x k ← q ( x k | x 0 ) , where x 0 = [ x v , x a ] ▷ Eq. (11) and (4)

9: θ 2 ← θ 2 -η ∇ θ 2 L ▷ Adam optimizer

10: end for

# Evaluation Process

1: Given a task, reset the environment

2: Obtain the initial video clips v 0 , language instructions l

3: for t = 0 to t max do

4: Initialize [ x v K , x a K ] ∼ p ( x K ) ▷ Eq. (10)

5: Construct y ← e t = VQVAE-Encoder ( v t )

6: for k = K to 1 do

7: Sample x v k -1 ∼ p θ 1 ( x v k -1 | x k , y , l ) and obtain z ˜ x v 0 from p θ 1 encoding ▷ Eq. (12)

9: x k -1 = [ x v k -1 , x a k -1 ]

8: Sample x a k -1 ∼ p θ 2 ( x a k -1 | x k , z ˜ x v 0 , l ) ▷ Eq. (13)

10: end for

11: Obtain predicted videos via VQVAE-Decoder ( x v 0 )

12: Reconstruct executable action sequence [ a t , · · · , a t + H -1 ] from x a 0

13: Execute the first action a t as the current action to interact with the environment

14: Obtain the next observed image(s), and update v t

15: end for

C Datasets and Experiments

C.1 Dataset

Meta-World. We use the official codes from https://github.com/Farama-Foundation/ Metaworld to collect 20 expert demonstrations for each task in Meta-World. The image size

is 260 × 260 . The dimension of action is 4 , representing the 3D position movements of the end effector and the variation of gripper openness. Every demonstration collected has 150 time steps.

RLBench. We use the official codes from https://github.com/peract/peract to generate a dataset for RLBench via a motion planner. Each demonstration consists of multi-view RGB images (i.e., front, left shoulder, right shoulder and wrist) and 8-dimensional actions including 3-dimensional transitions, 4-dimensional quaternion and a binary value about gripper openness. Following prior works [42, 75], we extract macro actions (keyframe actions) from collected demonstrations and leverage networks to predict the keyframe actions instead of consistent continuous actions. Specifically, a set of keyframe actions { k 1 , k 2 , · · · , k m } ⊂ A is captured with a simple heuristic: an action is a keyframe if (1) the joint velocities are near zero and (2) the gripper open state has not changed. Each data point in the demonstration τ can then be cast as a 'predict the next (best) keyframe action' task [44]. In this way, the sequence length of actions that need to be predicted is significantly reduced from hundreds of small steps to typically less than 10 macro steps.

C.2 Task Details

We take Meta-World as a main benchmark to evaluate our method and baselines, which consists of 50 diverse manipulation tasks. The poses and positions of goals are randomly generated during evaluation. These tasks require an agent to identify the observed sing-view RGB images and reach the goals with the correct behavior. See Fig. 9 for a sample visualization of the tasks.

We select 16 tasks out of 100 tasks from RLBench [43] that involve at least two or more variations to evaluate the multi-task and generalization capabilities of agents. Task variations include randomly sampled colors, sizes, counts, placements, and categories of objects. The set of colors include 20 instances: colors = { red , maroon , lime , green , blue , navy , yellow , cyan , magenta , silver , gray , orange , olive , purple , teal , azure , violet , rose , black , white } . The set of sizes include 2 instances: sizes = { short , tall } . The set of counts include 3 instances: counts = { 1 , 2 , 3 } . The placements and object categories are specific to each task. For instance, put in cupboard includes 9 YCB objects. In addition to these semantic variations, objects are placed on the tabletop at random poses. The tasks in RLBench require an agent to process multi-view RGB images properly and generalize to different variations that could be unseen in training. The details of variations for each task are referred to Table 3.

C.3 Experimental Setup and Additional Results

Generalize to unseen scenes. To evaluate the generalization ability of our method, we change the camera position and table texture in the Meta-World benchmark to generate out-of-distribution observations. Borrowing the official codes from https://github.com/RLAgent/factor-world ,

(a) window-open

(b) dial-turn

(c) close jar

(d) meat-off-grill

(e) faucet-open

(f) sweep-into

(g) stack blocks

(h) place cups

close the navy lar

take the steak off the grill

stack2maroon blocks

place2cups on the cup holder

we set the camera at another corner position and generate unseen table texture randomly while rendering. See Fig. 10 for visualization.

Generalize to unseen tasks. In the following, we show that VPDD can generalize on different tasks from the same domain. In experiments, during stage 2, we train the VPDD on 47 tasks on MetaWorld and leave 3 unseen tasks to test the generalizability. In stage 3, we fine-tune the pretrained model on these 3 tasks. We report the success rate of the final model trained over 0.2M gradient steps. The following experimental results in Table C.3 demonstrate the promising generalization ability of our model, as the performance gap compared with the oracle is very small.

(a) button-press

(b) shifted camera position

(c) shifted table texture

(d) handle-press

(e) shifted camera position

(f) shifted table texture

D Limitations and Future Work

We present illustrative samples of robot videos generated using discrete diffusion model p θ 1 in Fig. 3 and Fig. 11. More samples are available at https://video-diff.github.io . It is noteworthy that VPDD is able to generate dynamic consistent future videos, incorporating information beneficial for low-level control while maintaining coherence across distinct perspectives. However, it is imperative to acknowledge that our primary contribution is not on generating high-quality videos with exceptional resolution and meticulous semantic details. Consequently, some blurriness may be observed in our predicted videos, exemplified by deviations in the gripper's pose. See Fig. 12 for a failure example.

For future work, we could consider enhancing the coherence of videos across diverse views by leveraging the recently released Ego-exo4d data [28]. This extension encompasses considerations beyond solely temporal dynamics. Moreover, for the augmentation of video quality and the optimization of decision-making performance, it is worth exploring to scale up both the training dataset and model capacity.

E Broader Impact

This paper presents work whose goal is to advance the field of Machine Learning. Specifically, we propose a novel pretraining-finetuning framework to make better use of a copious amount of human actionless videos. Since this method is easy to reproduce (as we will release our code soon) and exhibits the SOTA performance, it encourages future research to further advance this field. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

(a) front

(b) front

(c) front

(d) front

(e) left shoulder

(f) left shoulder

(g) left shoulder

(h) left shoulder

(i) right shoulder

(j) right shoulder

(k) right shoulder

(l) right shoulder

(m) wrist

(n) wrist

(o) wrist

(p) wrist

二

二

(a) front

(b) front

(c) front

(d) front

(e) left shoulder

(f) left shoulder

(g) left shoulder

(h) left shoulder

(i) right shoulder

(j) right shoulder

(k) right shoulder

(l) right shoulder

(m) wrist

(n) wrist

(o) wrist

(p) wrist

小

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions and scope are introduced in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: It has been discussed in Appendix D.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the implementation details in Appendix A.3 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code and sufficient instructions of our approach in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide them in Appendix B and Appendix A.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the error bars for evaluation under different random seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide these details in Appendix A.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our work conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts of our method in Appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite them throughout our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We already provide sufficient implementation details of our approach in Appendix A.3. We also provide detailed instructions to run our method in supplementary material.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach that takes advantage of human-object interaction videos to improve policies. Their proposal involves three steps: 1) learn a video tokenizer common to both human and robot videos. 2) Learn a discrete-diffusion model that performs video token denoising on both human and robot videos. 3) Learn a discrete-diffusion model that performs action denoising on only robot data. They show this technique results in better performance as compared to baselines.\n\nSTRENGTHS:\n1. Proposed idea is very interesting. It brings together ideas from self-supervised pre-training and diffusion policies.\n2. They achieve strong performance on a number of manipulation benchmarks outperforming many baselines.\n3. Clear explanation of the methods.\n\nWEAKNESSES:\n1. The paper claims that pre-training on a large dataset of human-object interaction videos without actions is useful for learning actions for robots. However from the experiments it is not clear how much the human videos are adding to the policy performance? There is a significant amount of data difference between the models trained with or without human videos. Yet the performance between the two models is close. How does the model without human data perform given some more robot trajectories? In Fig. 1, for comparison without human videos the drop in performance in marginal. In Fig. 5 seems there is a bigger drop in performance which might be compensated by having multiple camera views in the robot training data. \n\n2. Ego4D videos are very different visually from the tasks and environments shown. It is not clear which tasks from Ego4D are helping in transferring good features for the robotics tasks. How much of the video tokenizer codebook is shared between the human and robot videos. Because the dataset statistics look so different I wonder if enough of the codebook is being shared by the two datasets?\n\n3. Generating egocentric videos is challenging. If the model is good at generating egocentric videos then that in itself is a big contribution. What do the generated videos look like for the Ego4D dataset? Does performance on this task correlate with better downstream policy performance or is the fine-tuning step capable of recovering good policies on video prediction models that are not that good?\n\n4. Another baseline to compare against would be a VideoMAE[1] kind of method to learn the video features on just robot dataset and then do diffusion on that. This will be a good way to make the case for a tokenizer based diffusion model to learn video features. \n\n[1] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. Zhan Tong, Yibing Song, Jue Wang, Limin Wang\n\nQUESTIONS:\nSee above weakness section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
In this paper, the authors focus on the problem of policy learning via leveraging large video data without action labels. To this end, they employ a discrete diffusion framework which is first employed to predict future quantized video frames. Afterwards, the model is fine-tuned on robot data to learn the final policy. The authors evaluate their approach on the Meta-World and RLBench and demonstrate promising quantitative results over other approaches.

STRENGTHS:
1. The paper addresses a very timely problem: Most available video data has no action labels. A major question is how to properly leverage this data to learn general policies for control tasks.

2. For an approach which utilizes a pre-training and a fine-tuning stage with frozen networks, the approach isn't overly complex. It's simple relative to other approaches. It therefore should be easy for researchers to implement and reproduce results.

3. Quantitative results for both Meta-World and RLBench are strong over baselines. The results in are quite promising.

WEAKNESSES:
1. Technical novelty is limited. The approach is similar to a lot of other approaches: quantize videos using VQVAE, predict future tokens using transformer, somehow account for missing action tokens.

2. The scope of domains for evaluation domains is somewhat limited. It would be interesting to see if the results hold on other robotic datasets.

QUESTIONS:
1. While I believe the current results in the paper are convincing, it still may strengthen the paper to demonstrate results on even more datasets.

LIMITATIONS:
Yes, the authors have properly addressed limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose utilizing large scale internet videos along with robot data to train a diffusion model which predicts video future frames conditioned on past video frames and language description of videos. This diffusion model operated on the vector quantized embeddings of the video frames. Subsequently, this diffusion model is adapted for action generation which is trained using a smaller amount of robot video data with action labels. The analysis is performed on the Metaworld suite and RLBench.\n\nSTRENGTHS:\n1. The paper is well written, clear and easy to understand\n2. The idea is well motivated. The question of utilizing internet scale knowledge for embodied intelligence is an open area of research.\n3. The experiment results are statistically significant with multiple runs reported accompanied by error bars. \n4. The method performs well in comparison to baselines.\n\nWEAKNESSES:\n1. **Missing Baselines:** How does the method compare against other generalist imitation learning methods like Octo, RT-1 etc? It's unclear how such methods compare in the finetuning setting. \n2. **Missing Ablations on amount of robot data used during pretraining:** While no action labels have been used during pretraining, the model still has access to robot trajectories. How does the method perform when no robot data is present during pretraining?\n3. **Lack of robot experiments:** Does the model extend to any real robot setups?\n\nQUESTIONS:\n1. What are the computational costs of training the backbone?\n2. Are the results on RLBench and Metaworld finetuned for every task or are in multi-task setting, i.e., do you have one model that yields the aggregate result on Metaworld or do you have one model per task?\n\nLIMITATIONS:\n1. The method is currently limited to simulation and it remains to be seen if the results will transfer to real robots. \n2. The method still requires 20 demonstrations per task during pretraining. It’s not clear if this is a kosher assumption. What if behavior cloning was used on these 20 demonstrations? Would this yield higher results? If so, its unclear why one would use VPDD?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper targets at transferring the knowledge from human videos to robotic manipulation policy. The paper proposes VPDD, a method that first pre-trains on video generation on both human and robot videos and then fine-tunes on robotic manipulation data with action labels. The intuition is, by learning to generate human and robot videos, the model obtains a \"prior\" (e.g., task planning, trajectory planning, etc.) of robotic manipulation which makes it easier to learn diverse manipulation tasks with even few-shot data. The authors compare VPDD to several robotic pre-training baselines and show non-trivial improvements over several benchmarks.\n\nSTRENGTHS:\nClear motivation that robot manipulation data is scarce and utilizing the large amount of human data will be helpful. The design of the model and the training pipeline is sensible and the results look promising.\n\nWEAKNESSES:\n1. The pre-training stage is trained jointly on human and robot data. The amount of human data is pretty abundant here but how many robot videos are we using here? If the robot videos are too few, will it affect the resulting pre-trained model from this stage? If the pre-training stage still needs a lot of robot videos, then it still faces the problem of lacking enough robot data. On the other hand, if the pre-trained model can still generalize pretty well without robot data in the pre-trianing stage, i.e., pre-training on human data only and fine-tuning on a small amount of robot data can work pretty well, it would be very helpful.\n\n2. Since the main message of the paper is training on human data can help learning robotic manipulation policies, it would be good to see how the performance changes when trained on different amounts of human data, for example, a scaling curve with x-axis being the number of human videos and y-axis being the performance. It would be interesting to see if the performance can keep going up by feeding more and more human data, or is there somewhere it saturates.\n\nQUESTIONS:\n1. In Figure 4, VPDD w/o human videos is still better than other baselines. What's the reason here? Assuming VPDD-w/o-human has seen the same robot data as other baselines, what's the main factor that causes such a difference?\n\n2. The authors tried VPDD on unseen scenes such as varying background and camera poses. What about unseen tasks? Assuming there are some tasks presented in human videos but not in robot videos, will the model learn to do these tasks, at least to some extent?\n\n3. As a bonus, it would be good to evaluate some off-the-shelf pre-trained video generation models such as Stable Video Diffusion for future exploration, but I don't think it affects the main conclusions of this paper.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}