Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
Authors: Yucheng Hu
Yanjiang Guo
Pengchao Wang
Xiaoyu Chen
Yen-Jen Wang
Jianke Zhang
Koushil Sreenath
Chaochao Lu
Jianyu Chen
Venue: ICML 2025 spotlightposter (2025)
Keywords: Robot policy learning
diffusion model
inverse dynamics model
Primary area: applications->robotics
TL;DR: We propose a generalist robot policy, Video Prediction Policy which conditioned on visual representaions inside video diffusion models.

Abstract: Visual representations play a crucial role in developing generalist robotic policies. Previous vision encoders, typically pre-trained with single-image reconstruction or two-image contrastive learning, tend to capture static information, often neglecting the dynamic aspects vital for embodied tasks. 
Recently, video diffusion models (VDMs) demonstrate the ability to predict future frames and showcase a strong understanding of physical world. 
We hypothesize that VDMs inherently produce visual representations that encompass both current static information and predicted future dynamics, thereby providing valuable guidance for robot action learning. 
Based on this hypothesis, we propose the Video Prediction Policy (VPP), which learns implicit inverse dynamics model conditioned on predicted future representations inside VDMs. 
To predict more precise future, we fine-tune pre-trained video foundation model on robot datasets along with internet human manipulation data.
In experiments, VPP achieves a 18.6\% relative improvement on the Calvin ABC-D generalization benchmark compared to the previous state-of-the-art, and demonstrates a 31.6\% increase in success rates for complex real-world dexterous manipulation tasks. For your convenience, videos can be found at https://video-prediction-policy.github.io/

Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
Yucheng Hu 12
Yanjiang Guo 13 Pengchao Wang*
Xiaoyu Chen 13 Yen-Jen Wang Jianke Zhang' 13
Koushil Sreenath
Chaochao Lu² Jianyu Chen 134
Abstract
Visual Representations: Contrastive Learning, Image Reconstruction
Visual representations play a crucial role in de- veloping generalist robotic policies. Previous vi- sion encoders, typically pre-trained with single- image reconstruction two-image contrastive learning, tend to capture static information, of- ten neglecting the dynamic aspects vital for em- bodied tasks. Recently, video diffusion models (VDMs) demonstrate the ability to predict future frames and showease strong understanding of physical world. We hypothesize that VDMs in- herently produce visual representations that en- compass both current static information and pre- dicted future dynamics, thereby providing valu- able guidance for robot action learning. Based on this hypothesis, we propose the Video Prediction Policy (VPP), which learns implicit inverse dy- namies model conditioned on predicted future rep- resentations inside VDMs. To predict more pre- cise future, fine-tune pre- pre-trained video foun- dation model robot datasets along with inter- net human manipulation data. In experiments, VPP achieves 18.6% relative improvement on the Calvin ABC-D generalization benchmark compared the previous state-of-the-art and demonstrates a 31.6% increase in success rates for complex real-world dexterous manipulation tasks. Videos and code are available at https: prwideo-peedietion----o
Vision Encoder
Predictive Visual Representations: Video Generation Pretraining
Shape:
Video Diffusion Model
(T, H, W)
T99
Instruction
Vision Encoder
Current Representation
Future Representations
Figure I. Visual representations inside video prediction models ex- plicitly express both current and future frames, providing valuable future information for embodied agent. Previous vision encoders did not have explicit future representations.
et al., 2023; Team et al., 2024; Wu et al., 2023a; Guo et al., 2025; Cui et al., 2025; Ding et al., 2024; 2025; Shi et al., 2025; Zhao al., 2025). crucial component in these gen- eralist policies the vision encoder, which captures visual information from pixel observations. Many studies have focused on optimizing vision representations for embodied agents, often leveraging internet video datasets (Ebert et al., 2021; Grauman et al., 2022) and self-supervised techniques such as single-image reconstruction (Majumdar et al., 2023; Karamcheti al., 2023; Gupta al., 2024), two-image con- trastive learning, and image-text contrastive learning (Nair et al., 2022; Ma al., 2022). Although these visual pre- training methods have demonstrated success for embodied tasks, they may not fully exploit the dynamic information en- coded sequential video datasets, as they typically operate on only one or two sampled images.
github. io.
1. Introduction
Building generalist robot policies capable of solving a vari- ety of tasks is rapidly advancing area of research (Brohan
Recently, powerful video diffusion models (VDMs) (Ho et al., 2022: Blattmann al., 2023a: Hong et al., 2022: Yang et al., 2024) have achieved impressive results in video generation tasks. Instead of performing pre-training oper- ation on single image or pairs of images, VDMs directly model entire video sequences. Text-guided video predic- tion models (TVPs) (Gu et al., 2023; Chen et al., 2023) can even predict future frames based on current observations
Eqqual contribution 'HIS Tsinghua University 'Shanghai AI Lab 'Shanghai Qi Zhi Institute 'RobotEra Uniiversity of California, Berkeley. Correspondence to: Jianyu Chen jjjanyuchen@tsingeee edu.cn">.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Vidco Prediction Policy
and instructions, demonstrating good understanding of the physical dynamics.
2. Related Works
Visual Representation Learning for Robotics. Self- supervised learning (SSL) techniques, such as con- trastive (Chen et al., 2021: 2020), distillation-based (Baevski al., 2022; Caron et al., 2020), and reconstruc- tive (He et al., 2022; Bao et al., 2021), have achieved signif- icant advancements in visual representation learning. Prior research has shown that these SSL techniques enable vision encoders to produce effective representations for embod- ied AT tasks (Yadav et al., 2023b;a; Parisi et al., 2022; Ra- dosavovic et al., 2023; Chen et al., 2024a), capturing both high-level semantic and low-level spatial information. No- tably, methods like R3M (Nair et al., 2022), vip (Ma et al., 2022), VC-1 (Majumdar et al., 2023), and Voltron (Karam- cheti et al., 2023) have specifically focused on embodied tasks by innovating pre-training approaches on human ma- nipulation video datasets (Goyal et al., 2017; Grauman et al., 2022). However, regardless of the training objective, the learned vision encoders primarily focus on extracting perti- nent information from current observations without explic- itly predicting future states. In contrast, our Video Predic- tion Policy leverages predictive representations within video prediction models to explicitly encapsulate both current and predicted future frames.
Inspired by the strong prediction capabilities of TVP mod- els, we hypothesize that they inherently contain valuable physical dynamics knowledge and can produce more effec- tive visual representations for embodied agent. We take a deeper look the visual representation inside TVP models. These representations are typically structured as a tensor with dimensions (T, H, W), explicitly representing 1 cur- rent step and ()) predicted future steps (Blattmann et al., 2023a), where H and W correspond to the height and width of the image representation. In contrast, previous vision encoders do not explicitly capture future representations, as shown in Figure 1. Based on this distinction, we refer to these latent variables within the video diffusion model as "predictive visual representations".
Our key insight is that the downstream policy can implicitly learn the inverse dynamics model by tracking the robot's movements within the predictive representation. As long as the video model accurately predicts future scenarios for diverse tasks, the policy can generate appropriate actions by tracking robot arm's 's position implicitly. In this way, we can transfer the generalization capabilities of the video pre- diction model to robotic policy. We only need few demon- strations to align the robot's action space with the visual space.
Future Prediction for Embodied Control Tasks. Existing research also explores the use of future prediction en- hance policy learning (Bharadhwaj et al., 2024; Chen et al., 2024b; Ye et al., 2024; Guo et al., 2024; Zhang et al., 2025; Song et al., 2025). For example, SuSIE (Black et al., 2023) conditions its control policy on a predicted future keyframe generated by InstructPix2Pix (Brooks et al., 2023), while UniPi (Du et al., 2024) learns the inverse dynamics between two generated frames. These methods rely single fu- ture prediction step to determine actions, which may not accurately capture the complexities of physical dynamics. Additionally, they denoise the final future image which is cosemio and lead to low control frequency. GR-1 (Wu et al., 2023a) generates subsequent frames and actions au- toregressively. However, it only generates one image per forward pass, and its prediction quality lags behind that of diffusion-based methods. Furthermore, GR-1 does not leverage pre-trained video foundation models. In contrast, VPP leverages representation fine-tuned from video founda- tion model, and predict a sequence of future frames to more effectively inform policy learning.
Building on this insight, we introduce the Video Prediction Policy (VPP), which employs a two-stage learning process: First, we fine-tune general-purpon into a text-guided video prediction (TVP) model using in- ternet human and robot manipulation data (Goyal et al., 2017; O'Neill et al., 2023). This step aims to develop a controllable video generation model that improves predic- tion capabilities in the manipulation domain. In the second stage, we learn inverse dynamics model conditioned on the predictive representations from the TVP model. Since we direct use the internal representation and avoid the need for multiple denoising steps as in previous work (Black et al., 2023; Du et al.. 2024), VPP can operate at high frequency in a closed-loop manner. We also visualize the representations within the VDM and confirm that they effectively capture key information about future evolution.
video diffusion model
In experiments, VPP consistently outperform other baseline algorithms across two simulated (Mees et al., 2022; Yu et al., 2020) and two real-world settings, demonstrating the effectiveness of our approach. Notably, the VPP achieves a 41.5% improvement in the Calvin ABC-D benchmark (Mees et al., 2022) compared to the previous SOTA method (Wu et al., 2023a). In real-world experiments, VPP shows a 31.6% improvement in success rate over the strongest baseline on high- nigh-dimmnsionall dexterous hand manipulation tasks.
Visual Representation inside Diffusion Models, Diffusion models have achieved remarkable success in the image and video generation tasks (Rombach et al., 2022; Blattmann et al., 2023a). Although diffusion models are trained as de- noisers, researches have shown that image diffusson models can also function effectively as vision encoders, generating meaningful visual representations that is linear-separable
2

---

Vidco Prediction Policy
for discrimination tasks (Xiang et al., 2023) and invalu- able for semantic segmentation (Luo et al., 2024). Gupta et al. (2024) also point out that representation inside image diffusion are versatile for embodied tasks. However, the capabilities of representations within video diffusion mod- els have not been extensively explored. He et al. (2024) try to use latent representation inside discrete VDMs to assist policy learning, however it need not leverage pre-trained video foundation models and train from scratch. Our find- ings suggest that representation within pretrained VDMs have a unique predictive property, making them especially useful for sequential embodied control tasks.
train the Text-guided Video Prediction (TVP) model across diverse manipulation datasets harness physical knowledge from internet data; subsequently, we design networks to aggregate predictive visual representations inside the TVP model and output final rohot actions.
4.1. Text-guided Video Prediction (TVP) Model for Robot Manipulation.
Recent advancements have focused on training general video generation models using extensive online video datasets, which encode abundant prior knowledge about the physical world's dynamics. However, we notice that these models are not fully controllable and fail to yield optimal results in specialized domains such as robot manipulation. To address this, we fine-tune the general video generation model into a specialized "Manipulation TVP Model" to enhance prediction accuracy.
3. Preliminaries
Video Diffusion Models. The core idea of diffusion models is to continuously add Gaussian noise to make video se- quences Gaussian and leverage the denoising process for generating videos. Let Zo represent a real video sample, the forward process aims to add Gaussian noise and result in a set of noisy data, i.e. q(xz|=;-1) = (zzz /a (1 IT) , where 22 and a, indicate the noisy data and noise amplitude at the timestep L. Let 0₂. IT Og, the above process can be simplified
We chose the open-sourced Stable Video Diffusion (SVD) model (Blattmann et al., 2023a) with 1.5 billion parameters foundation. observe that the open-sourced SVD model conditions only on initial-frame images 50. We aug- ment the model to incorporate CLIP (Radford et al., 2021) language feature lemb using gross-attention layers, Further- more, we adjust the output video resolution to 16x256x256 to improve training and inference efficiency. Despite these modifications, we preserve the other components of the original pre-trained SVD framework retain its core ca- pabilities. We denote this modified version as Vg. In this setup, the initial observation wise with each predicted frame as a condition. Then model Ve is trained with diffusion objective, reconstructing the full video sequence To = 80:T in dataset D) from noised samples 24 = Vaiza + /1 āec.
It V Va + VI - ōgtt.
(1)
The reverse process starts from the most noisy sample ør can be described in variational approximation of the prob- abilities (( (ze), as follows:
p(z1-1 (x)) =
ioo- t), (1 -
is concatenated channel-
(2)
where po( (z, - V1-vv t))/ Va is a learn- able neural network to estimate 11 Further, in text- guided video generation, the denoising process learns the noise estimator (x c) to approximate the score function -I v,, log (Ia (c), controlling the video generation based on the initial frame and language prompt.
LD D,ct 1701x,eeebb su) xoll ||
(3)
The video prediction objective offers a unified interface that directly generates future visual sequences, enabling the TVP model to harness physical knowledge from diverse datasets. These include internet human manipulation datasets DH- internet robot manipulation data DR, and also self-collected datasets Dc. Given the varying quality and scale of these datasets, we introduce specific coefficients À to appropri- ately balance the influence of different dataset types:
Diffusion Policy. The diffusion model has also proven effec- tive in action learning, known as diffusion policy (Chi et al., 2023). The diffusion policy aims to denoise the action se- quence a; (a,, 0++ .... di+m) based on observations s; and instruction. Chi et al. (Chi et al., 2023) point out that dif- fusion policy is capable of expressing complex multimodal action distributions and stabilizing training. Recent work (Reuss et al., 2024) further enhances the diffusion policy by incorporating the advanced diffusion transformer (DiT) block (Peebles & Xie, 2023), a technique we also adopt in the Video Prediction Policy to improve performance.
Losteo = AH + ARRDD + +CcLDe
(4)
Then we froze the fine-tuned manipulation TVP models in downstream action learning.
4.2. Action Learning Conditioned on Predictive Visual Representation
4. Video Prediction Policy
In this section, we describe the two-stage learning process of the Video Prediction Policy, shown in Figure 2. Initially, we
TVP Model as Vision Encoder. After training the TVP model specifically for manipulation tasks, can accurately
3

---

Video Prediction Policy
"Place orange
d CLIP
Video Diffusion Policy
We
Action Denolse C
Learnable tokens
DIT Block
Rerees
Us3e
NEa
XM
Interpolate & Stack
DIT Block
XN
Time
Tokenized and Time Position Embedding
t-0
t=T
Text Embedding
Stage 1: Predictive Visual Representations Learning
Stage Action Learning
Figure 2. In the first stage, VPP fine-tunes a general Prediction (TVP) model using robot and internet manipulation datasets. In the second stage, video-former aggregate the representations from the TVP model during the first forward pass, followed by the diffusion policy head. This approach enables VPP to learn implicit inverse dynamies model from the predicted future while maintaining high control frequency.
general-purpose -prppos video foundation model into manipulation- focusee Text-guided Vidco
predict future sequences based on image observations and instructions. However, denoising entire video sequence is highly time-consuming and may lead open-loop trol issues, discussed in (Du et al., 2024). Moreover, videos in their original pixel format often contain exces- sive, irrelevant information that can interfere with effective decision-making
RTx(E C. Cm)x given by:
F, = concate Li,..., L'm), dim =
For a robot with multiple camera views, such as a third-view and a wristed camera, we predict the future for each view independently. denoted as psttatic pourist,
Video Former. These predictive representations within the video diffusion model are still high-dimensional, they express a sequence of image features. To efficiently aggre- gate representations across spatial, temporal, and multi-view dimensions, we design a Video Former to consolidate this information into fixed number of tokens. The Video For- mer initializes learnable tokens Q(u:T,u-L] with fixed length T L, performing spatial-temporey attention (Blattmann et al., 2023b) each corresponding frame, followed by feed-forward layers. Formally, this branch can be expressed as follows where i is the index of frame:
To address these concerns, we employ the video diffusion model primarily
"vision encoder" rather than "de- noiser" by performing only single forward step. Our in- sight is that the first forward step, while not yielding clear video, still provides a rough trajectory of future states and valuable guidance. This insight verified in our experiment section and shown Fig 4. Specifically, we concatenate the current image so with the final noised latent (( (zn)) (typi- cally white noise) and input this combination into the TVP model. We then directly leverage the latent features. Previ- ous work (Xiang al., 2023) highlights that up-sampling layers in diffusion models yield more effective representa- tions. The feature at the m up-sampling layer, with width Wm, and height Hm. is expressed as:
Q' (Spat-Autn ( (Istattc [1), (
(5)
Q" -FN(TemmpAtnn(("))
Tim Vo lemb: 80) (m)) Tim E Tx. XW.. X.,,
Action Generation. After the Video-Former aggregates the Predictive feature into learnable tokens Q", diffusion policy is employed as the action head to generate the action sequence E A based on Q". We integrate the aggre- gated presentation Q" into diffusion transformer blocks using cross-attention layers. The diffusion policy aims to reconstruct the original actions dg from noised action ag = /Bgao + v1 - Bec, where represents white noise, and Bg is the noisy coefficient at step k. This step can be interpreted as learning a denoiser D& to approximate the
To effectively aggregate features from the up- sampling lay- and eliminate the need for manual layer selection, propose automatic method for aggregating features across different layers. First, we linearly interpolate each layer's feature map to the same height and width W, x Hp:
I/ = Interpolation I', C'x X
We then stack the features along the channel dimen- sion. The final predictive visual representation F, E

---

Video Prediction Policy
ith Task Success Rate
Category
Method
Annotated Data
2
3
5
Avg. Len 1
RT-1
100%ABC
0.533
0.222
0.094
0.038
0.013
0.90
Direct Action
Diffusion Policy
100%ABC
0,402
0.123
0.026
0.008
0.00
0.56
Learning Method
Robo-Flaningo
100%ABC
0.824
0.619
0.466
0.331
0.235
2.47
Uni-Pi
I00%ABC
0.560
0.160
0,080
0.080
0.040
0.92
MDT
100%ABC
0.631
0.429
0.247
0.151
0.091
1.55
Future Prediction
Susie
100%ABC
0.870
0.690
0.490
0.380
0.260
2.69
Related Method
GR-1
100%ABC
0.854
0.712
0.596
0.497
0.401
3.06
Vidman
100%ABC
0.915
0.764
0.682
0.592
0.467
3,42
3D Method
RoboUniview
100%ABC
0.942
0.842
0.734
0.622
0.507
3.65
Ours
VPP (ours)
100%ABC
0.965
0.909
0.866
0.820
0.769
4.33
Data
GR-1
10%ABC
0.672
0.371
0.198
0. 0.108
0.069
1.41
Efficiency
VPP (ours)
10%ABC
0.878
0.746
0.632
0.540
0.453
3.25
Table Zero-shot long-horizon evaluation the Calvin ABC-DD benchmark where agent is asked to complete five chained tasks sequentially based on instructions. VPP demonstrates a significant improvement the average task completion length (Avg. Len),
Emv
Env B
Easy Tasks
Task Level
Easy
Middle
Hard
Average 1 (50 tasks)
(Numbers)
(28 tasks)
(11 tasks)
(11 tasks)
Middle Tasks
RT-1
0.605
0.042
0.015
0.346
Diffusion Policy
0.442
0.062
0.095
0.279
Env
Unseen Enw
Susie 0,560
0. 196
0.255
0,410
GR-1
0.725
0.327
0.451
0.574
VPP (ours) 0.818
0.493
0.526
0.682
Table Multi-task success rate on Metaworld. use single language- conditioned policy to solve all 50 tasks.
Figure 3. CALVIN and Metaworld benchmarks.
noise c and minimize the following loss function:
setting, where the agent is trained in the ABC environment and evaluated in the unseen environment, as illustrated in Figure We use settings same as GR1 (Wu al., 2023a) which only the Language-annotatel ABC datasets for training.
Can(w; A) = Eao,e, II (aks Lemb. Q") aoll
(6)
5. Experiments
World Benchmark. Metaworld (Yu et al., 2020) fea- tures Sawyer robot performing various manipulation tasks and widely used to evaluate the precision and dexterity of robotic policies. As shown on the right Figure 3, includes 50 tasks with a rich array of operating objects at different levels of difficulty (Radosavovic et al., 2023). We use official Oracle policy collect 50 trajectories for each task as our training dataset.
In this section, we conduct extensive experiments on both simulated and real-world robotic tasks to evaluate the per- formance of the video prediction policy (VPP). We aim to answer the following questions:
1. Can VPP achieve a higher success rate manipulation tasks with predictive visual representations?
2. How do the video pre-training and internet manipula- tion datasets enhance the performance of VPP?
VPP Training Details. As outlined in Sec. 4, we use a two- stage training process. In the first stage, we fine-tune a video foundation model into a manipulation-focuse TVP model. The videos used in this stage include 193,690 human manip- ulation trajectories (Goyal et al., 2017) and 179,074 robotic manipulation trajectories (O' O''eill et al., 2023), along with downstream task videos, such as the official Calvin ABC videos, the MetaWorld videos, and real-world videos. Given the varying scales and quality of these datasets, we ap- ply different sampling ratios, following the approach in
3. How does predictive representation compare to previ- ous visual representations?
4. Which layer of the video diffusion model provides the most effective predictive visual representations?
5.1. Simulation Setups and Baselines
CALVIN Benchmark. CALVIN (Mees et al., 2022) is widely used benchmark designed to assess the instruction- following capability robotic policies in long-horizon ma- nipulation tasks. We focus on the challenging ABC-D
5

---

Vidco Prediction Policy
Octo (Team et al., 2024). Detailed dataset scales and sam- pling ratios can be found in Appendix B. Fine-tuning the video model takes 2-3 days on eight NVIDIA A100 GPUs.
Vidman (Wen et al., 2024). Pre-trained on the Open X- E--oodimennt dataset (OXE) video datasets and use a layer-wise self-attention adapter to transform video representation into policy model. However, Vidman did not finetune video model on down-stream tasks which lead to sub-optimal performance.
In the second stage, we train a generalist policy with Calvin or Metaworld dataset, which requires approximately 6-12 hours on four NVIDIA A100 GPUs.
Policy Roll-out Details. Previous works choose to denoise high-precision videos, a process that is conecconuuning and results in low-frequency (Black et al., 2023), even open- loop control (Du et al., 2024). In contrast, our approach uses the TVP model as an encoder rather than a denoiser, ensuring that each observation is processed through the TVP model only once, which takes less than 160 ms. Then down- stream policy generate action conditioned on the predictive representation. This modification allows us to achieve significantly higher frequency of 7-10 Hz with consumer- level NVIDIA RTX 4090 GPU. Additionally, we implement action chunking (Chi et al., 2023) with 10 steps to further improve the control frequency.
Quantitative Results. The comparisons on the Calvin benchmark are shown in Table 1. Results for Robo- Flamingo, Susie, GR-I, and 3D Diffuser Actors are recorded from their original papers. The MDT result is run on official implementation. The RT-1 result is sourced from (Li et al., 2023) and the Uni-Pi result from (Black et al., 2023). We also ran the Diffusion Policy based on the official open- source codebase with CLIP language conditions. Our pro- posed Video Prediction Policy significantly improved the previous state-of-the result from an average task com- pletion length of 3.65 to 4.33. Even with only 10% of the annotated Calvin ABC data used for training, our method still achieved a length of 3.25, which exceeds the results of related methods using full data. Furthermore, the Video Prediction Policy also achieved the best performance in the MetaWorid benchmark with 50 tasks, outperforming the similar strongest GR-1 baseline by 10.8% in average success rate.
Comparisons. Generalist robot policy has been widely explored in previous studies. In our experiments, we opted compare against a representative subset of prior methods that have either achieved state-of- off-the-art performance or share a similar approach with our methods.
Visualizations of Predictive Representations. Since we use the video prediction model a vision encoder and perform single forward pass to obtain predictive represen- tations, we are curious about the quality of these represen- tations. In Figure 4, we visualize the ground truth future, single-step predictions, and 30-step denoised predictions. We can observe that single-step representation already con- veys valuable information, such as the movement of objects and the robot arm, which effectively supports downstream action learning.
RT-1 (Brohan et al., 2022). A direct action learning robot policy that integrates semantic information us- ing Efficient-Net with condiinnniii,,,, followed by token learners for action learning.
Diffusion Policy (Chi et al., 2023). A direct action learning policy with novel action diffusers.
Robo-Flamingo (Li et al., 2023). A direct action learn- ing policy that leverages pre-trained LLM, incorpo- rating visual information into each layer in a flamingo style (Alayrac et al., 2022).
Uni-Pi (Du et al., 2024). Begins by learning a video prediction model to generate future sequences and then learns an inverse kinematics model between two frames to determine actions.
5.2. Ablation Study
VPP achieves significant improvements in simulated ex- periments. In this section, we conduct ablation studies to identify the effectiveness of different components of VPP. All ablation study are performed on Calvin ABC-D bench- mark and evaluated with average task completion length.
MDT (Reuss al., 2024). Learns diffusion trans- former policy along with an auxiliary mae loss to re- construct one masked future frame.
Effectiveness of Predictive Visual Representations. To verify the effectiveness of representation inside VDM, we replace the VDM vision encoder with several other pre- trained vision encoders designed for embodied tasks, while keeping all other components and settings unchanged.
Susie (Black et al., 2023). Uses a fine-tuned Instruct- Pix2Pix (Brooks et al., 2023) model to generate a goal image and learns a downstream diffusion policy condi- tioned on the goal image.
GR-1 (Wu et al., 2023a). Learns video and action se- quences jointly using an auto-regressive transformer. During policy execution, GR-1 outputs one future frame followed by one action.
1. Stable- VAE (Blattmann et al., 2023a), tre-trained with a VAE image reconstruction loss. Since the VAE encoder-decoder already performs well in reconstruct- ing images from video datasets, we did not perform further fine-tuning. The input 256x256 images are en- coded into 32x32 features with VAE, which are then
Robo-Uniview (Liu et al., 2024). Learns a 3d-aware visual encoder with 3d occupation loss to assist policy learning.
6

---

Vidco Prediction Policy
"Place the grasped object the drawer."
"Place the orange to blue plate."
Input
Ground Truth
30 Steps Denoise Prediction Step Direct Prediction
Figure 4. Visualization of one-step forward visual representations, We can observe that one-step representation already provide valuable information on physical evolution, although the textures and details are not precise.
Encoder
Pre-training Type
Avg. Length t
its ability to leverage the physical knowledge encoded in pre-trained video generation models and Internet manipu- lation datasets. We conducted experiments to verify the effectiveness of these two components. As shown in Table 4, removing the co-trained Internet manipulation data re- sulted a performance decrease from 4.33 to 3.97. Further removing the pre-trained SVD model and training the video prediction model from scratch the Calvin dataset led to a substantial performance drop. Notably, removing the video pretraining on Calvin alone also caused significant decline.
VDM (ours)
Video Generation
4.33
Stable-VAR VAE
VAE Reconstruction
2.58
VC-1
MAE Reconstruction
1.23
MAE Reconstruction+
Voltron
1.54 Language Generation
Table Ablation study different visual representations.
Ablation Type
Avg. Length
Latency
J
VPP
4.33
~ 140ms
VPP w/o Internet data
3.97
140ms
Effectiveness of Video Former. The Video Former module plays a pivotal role extracting predictive representations from the TVP model. To evaluate its effectiveness, we conduct ablation study by removing the Video Former and directly connecting the TVP features to the diffusion policy. The results, presented in Table 5, are obtained by evaluating the complete VPP model on single NVIDIA RTX 4090 GPU. The VPP score decreases from 4.33 to 3.86, while the inference time nearly triples. These findings indicate that the absence of the Video Former leads a substantial degradation in both accuracy and computational efficiency compared to the full model.
VPP w/o Calvin video
3.31
~ 140ms
VPP w/o Internet data
1.63
140ms
w/o SVD Pretrain
VPP w/o Video Former
3.86
~450ms
VPP w/o Feature Agg.
3.60
1440ms
Table Ablation study video pre-training and architecture.
resampled into 256 tokens via resampler (Jaegle et al., 2021) before passing to the diffusion policy, consistent with VPP.
2. VC-1 (Majumdar et al., 2023), pre-trained with a masked autoencoder loss. The authors note that fine- tuning ve-1 encoder with MAE loss on downstream task datasets can significantly improve performance. For fair comparison, we first fine-tuned the model the same video datasets used in VPP. The ve-1 features are resampled into 256 tokens with resampler and pass to policy head.
Effectiveness of Feature Aggregation Module. Many pre- vious works (Black et al., 2023; Wu et al., 2023a) directly use the final predicted image to learn policies. However, the image from the final layer often contains many irrelevant details that are not beneficial for the task. In contrast, we adopt feature aggregation mechanism leverage multiple layers of features within the up-sampling layers. We replace aggregated features with final layer features while keeping the other layers unchanged. This process lead to a decrease in the average task completion length on the Calvin bench- mark, from 4.33 to 4.05. More ablations on different layers can be found Appendix C.2.
3. Voltron (Karamcheti et al., 2023), pre-trained with both MAE future reconstruction and language generation tasks. We also fine-tuned the model on our video datasets and resampled the features into 256 tokens.
The results, presented in Table 3, indicate that replacing our predictive visual representations leads to a clear decline in performance.
5.3. Real World Experiments
We further verified the Video Prediction Policy on two real- world hardware platforms.
Effectiveness of Video Pre-training and Internet Manip- ulation Datasets. A significant advantage of the VPP is
7

---

Video Prediction Policy
Camera
"Pick the blue block?"
"Open
"Pick block."
"Tuiee the banara hand."
"Take spoonful of soup."
"Pour the black liquid into blue bowl."
Franka Panda
Third-feww Camera Wristed
Place to
"Place
"Retocate
Electrical
Dexterous Hand with over 100+ tasks
"Put the "close the drawee" into the
"Stack red block block?"
"Une tool to suck black liqult."
"Drop liquid Into
Seen Tasks
Tasks
Challenging Tool-use Tasks
Figure 5. Two real-wor world hardware platforms and visualizations of sampled tasks. We consider task "unseen task" if the operated object or the background scene do not appear in training datasets.
"Relocate the tennis to blue plate.
"Pouring coca cola into cup.
"Take spoonful liquid using spoon."
Figure 6. Predictions and executions on unseen tasks. Video prediction model generate reasonable futures (red). Real execution
trajectories (green) is also closely aligned the video predicted future (red).
Franka Panda Robot Arm On the Franka Panda platform, we collected 2,000 trajectories for over 30 tasks 6 cat- egories: picking, placing, pressing, routing, opening, and closing. We divided the tasks into seen and unseen cate- gories. A task is considered unseen if the operated object is new or the background scene is new.
Franka Panda
DP
Susie
GR-1
VPP(ours)
Seen Tasks
0.42
0.56
0.52
0.85
Unseen Tasks
0.25
0.46
0.38
0.73
Dexterous Hand
DP
Susic
GR-1
VPP(ours)
Seen Tasks
0.28
0.45
0.32
0.75
Unseen Tasks
0.11
0.28
0.15
0.60
Xarm with 12-degree Xhand Dexterous Hand. On the dexterous hand platform, we collected 4,000 trajectories over 100+ tasks 13 categories, including picking, placing, cup-upright, relocating, stacking, passing, pressing, unplug- ging, opening, closing, pouring, suction, and knocking. We also define task unseen if the operated object is or the background scene is new. Additionally, we included four challenging tool-use tasks, including the use of spoon, hammer, electrical drill, and pipette for chemistry tasks. More task details can be found in Appendix A.
Tool-use Tasks
0.05
0.23
0.15
0.68
Table 5. Success rates real-world tasks. Due space limit, only show the average success rate on each category. Detailed success
be found Appendix A
Quantitative Results. Due to the complexity of deploying methods on real-world hardware, we select the strongest baseline models-GR- Susie, and the widely-used diffu- sion policy-as as our baselines. For evaluation, we perform 200+ rollouts for Panda arm manipulation tasks and 500+ rollouts for dexterous hand manipulation tasks. The compar- isons are in the Table 5, which indicate VPP outperforms all the baselines with clear margin in both seen tasks, unseen tasks and tool-use tasks.
Training and Rollout Details. employ the same text- guided video prediction (TVP) model as in our simulated experiments, trained on both internet datasets and collected real-world data. Then a generalist robot policy is learned to solve all tasks in the domain conditioned instructions. The hardware platform and visualizations of some selected tasks are shown in Figure 5.
we take three unseen tasks as case studies: picking up a tennis ball, pouring Coca-Cola, and using a spoon. Notably, none of these objects-tennis tennis
Generalization Analysis.
8

---

Vidco Prediction Policy
ball, Coca-Cola, apoon-appear in collected dataset. As illustrated in Figure 6, the video prediction model fore- cast reasonable future states even on unseen tasks. More- over, we observe that the actual execution trajectory closely aligns with the predicted future state. We interpret the gen- cralization mechanism of the VPP model in two key aspects: First, video models can make correct visual predictions even on unseen tasks due internet-scale pre-training: Second, the low-level policy learns robust inverse dynamics model that only needs to implicitly track the movement of the robot in the predicted future, without the need to focus on new objects or backgrounds. In this way, the VPP model successfully generalizes to a wide range of unseen tasks.
pre-training of image transformers. arXiv preprint arXiv:2106.08254 2021.
Bharadhwaj. H., Dwibedi, D., Gupta, A., Tulsiani, S., Do- ersch, C., Xiao, T., Shah, D., Xia, F,, Sadigh, D., and Kirmani, S. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409 2024.
Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXi:22310.06639 2023.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231..5127, 2023a.
6. Conclusion
We introduce Video Prediction Policy (VPP), novel ap- proach for learning generalist robot policy. VPP learns implicit inverse dynamics model conditioned on predictive representations inside VDMs and yields consistent improve- ments across both simulated and real-world tasks. As video generation models are more and more powerful these days, we aim to fully unlock the power of video model building physical intelligence and highlight the potential of video generation models in embodied tasks.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler. S., and Kreis. K. Align your latents: High- resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575 2023b.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J.,
Acknowledgements
Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A.,
Hsu, J., et al. Rt-1: 1: Roboties transformer for real-world control at scale. arXiv preprint arXiv:2212.0688/ 2022.
This research is supported by III, Tsinghua University, Roboterax, UC Berkeley and Shanghai AlLab. And we gratefully acknowledge the Qizhi Research Institute for providing computational resources.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K. Ding, T.. Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-attion models transfer web knowledge to robotic control, arXiv preprint urXhv:2307.15881 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
References
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9922 29244, 2020.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M.. al. Flamingo: visual language model for few- shot learning. Advances in neural information processing 2022.
Chen, B., Xu, 7.,, Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., and Xia, F. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465. 2024a.
systems, 35:23716
Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M. Data2vec: A general framework for self-supervised learning in speech, vision and language. In Interna- tional Conference on Machine Learning, pp. 1298-1312. PMLR. 2022.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-16077 PMLR, 2020.
Bao, H., Dong, I.,, Piao, S., and Wei, F. Beit: Bert

---

Vidco Prediction Policy
Chen, W., Ji. Y. Wu, J., Wu, H., Xie, P., Li, J., Xia, X., Xiao, X., and Lin, L.. Control-a-video: Controllable text- to-video generation with diffusion models. arXiv preprint arXiv:2305.13840, 2023.
Gu, X., Wen, C., Ye, W., Song, J., and Gao, Y. Seer: Lan- guage instructed video prediction with latent diffusion models. arXiv preprint arXiv:2303. 14897 2023.
Guo, Y., Hu, Y., Zhang, J., Wang. Y.-J., Chen, X., Lu, C., and Chen, J. Prediction with action: Visual policy learning via joint denoising process. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024.
Chen, X., Xie, S., and He, K. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1640-9649, 2021.
Chen, X., Guo, J., He, T., Zhang, C., Zhang. P.. Yang, D. C., Zhao, I.,, and Bian, J. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai. arXiv preprint arXiv:2411.0077 2024b.
Guo, Y., Zhang, J., Chen, X., Ji, X., Wang. Y.J.,, Hu, Y., and Chen, J. Improving vision-lange vision-language- languagggaattnioonnn model with online reinforcement learning. arXiv preprint arXiv:2501. 2025.
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burch- fiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.c 0417, 2023.
Gupta, G., Yadav, K., Gal, Y., Batra, D., Kira, Z., Lu, C., and Rudner, T. G. Pre-trained text-to-image diffusion models versatile representation learners for control. arXiv preprint arXiv:2405.0mmmm 05.50555.5 2024.
Cui, C.. Ding, P. Song, W., Bai, S., Tong. X., Ge, Z., Suo, R., Zhou, W., Liu, Y., Jia, B., al. Openhelix: A short survey, empirical analysis, and open-source dual- system vla model for robotic manipulation. arXiv preprinl arXiv:2505.03911 2025.
He, H., Bai, C., Pan, L., Zhang, W., Zhao, B., and Li, X. Learning an actionable discrete diffusion policy via large-scale actionless video pre-training. arXiv preprint WrXiv:2402114407, 2024.
He, K., Chen, X., Xie, S., Li, Y.. Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Ding, P., Zhao, H., Zhang, W., Song, W., Zhang, M., Huang, S., Yang, N., and Wang, D. Quar-vla: Vision-language- action model for quadruped robots. In European Confer- ence on Computer Vision, pp. 352-367. Springer, 2024.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8666 8646, 2022.
Ding, P., Ma, J., Tong, X., Zou, B., Luo, X.. Fan, Y., Wang, T., Lu, H., Mo, P., Liu, J., al. Humanoid-vla: Towards universal humanoid control with visual integration. arXiv preprint arXiv:2502. 14795, 2025.
Hong. W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video gener- ation via transformers. arXiv preprint arXiv:2205.75868 2022.
Du, Y., Yang, S., Dai, B., Dai, H., Nachum, O., Tenenbaum, J., Schuurmans, D., and Abbeel, P. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. Perceiver: General perception with erative attention. International conference on machine learning, pp. 4651 4664. PMLR, 2021.
Ebert, F. Yang. Y., Schmeckpeper, K., Bucher, B., Geor- gakis, G., Daniilidis, K., Finn, C., and Levine, S. Bridge data: Boosting generalization of robotic skills with cross- domain datasets. arXiv preprini arXiv:2 13396, 2021.
Karamcheti, S., Nair, S., Chen, A. S., Kollar, T., Finn, C., Sadigh, D., and Liang, P. Language-driven representation learning for robotics. arXiv preprint arXiv:2302.12766 2023.
Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag M., et al. The" something something" video database for learning and evaluating visual com- mon sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Li, X., Liu, M., Zhang, H., Yu, C.,, Xu, J., Wu, H., Cheang, C., C.,jing Y., Zhang W., Liu, H., et al. Vision-language foundation models effective robot imitators. arXiv preprint arXiv:2311.013
Grauman, K., Westbury, A., Bymee E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the JEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
2023.
Liu, F., Yan, F., Zheng, L., Feng, C., Huang, Y., and Ma, L. Robouniview: Visual- -lnuuall model with unified view representation for robotic manipulation. arXiv preprint arXiv:2406. 18977, 2024.
10

---

Vidco Prediction Policy
Luo, G., Dunlap, L.,, Park, D. H., Holynski, A., and Darrell, T. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
Reuss, M., Ömer Erdinç Yağmurlu, Wenzel, F,, and Li- outikov, R. Multimodal diffusion transformer: Learning versatile behavior from multimodal goals, 2024. URL https: / http/aivvv.oo////// org/ abs 2tabs/24077 207.4009.9999
Luo, Z., Chen, D., Zhang. Y., Huang, Y., Wang, L.,, Shen, Y., Zhao, D., Zhou, J., and Tan, T. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint ar X001:223330083000 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Ma, Y. J., Sodhani, S., Jayaraman, D.. Bastani, O., Kumar, V., and Zhang, A. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint ar arXiv:2210.0000 00030, 2022.
Shi, L. X., Ichter, B.,, Equi, M., Ke, I.., Pertsch, K., Vuong, Q. Tanner, J., Walling, A., Wang, H., Fusai, N., et al. Hi robot: Open-ended instruction following with hier- archical vision- aasson-language-cttinn models, arXiv preprint arXiv:2502. 294117 2025.
Majumdar, A., Yadav, K., Arnaud, S., Ma, J., Chen, C.,, Silwal, S., Jain, A., Berges, V.PP., Wu, T., Vakil, J., al. Where are we in the search for an artificial visual cortex for embodied intelligence? Advances in Neural Informa- 2023.
Song, W., Chen, J., Ding, P., Zhao, H., Zhao, W., Zhong, Z., Ge, Z., Ma, J., and Li, H. Accelerating risionlaaggaae action model integrated with action chunking via parallel decoding. arXiv preprint arXiv:2503. 02.310, 2025.
tion Processing Systems,
Mees, O., Hermann, L., Rosete-Beaate E., and Burgard, W. Calvin: A benchmark for language-conditionee policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3)77327-7334, 2022.
Team, O. M., Ghosh, D., Walke, H.,, Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 122113 2024.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: 1812.01717 2018
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3m: universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 2022.
O'Neill, A., Rehman, A., Gupta, A., Maddukuri, A.,
Wen, Y., Lin, J., Zhu, Y., Han, J., Xu, H., Zhao, S., and Liang, X. Vidman: Exploiting implicit dynamics from video diffusion model for effective robot manipulation. Advances in Neural Information Processing Systems, 37: 41051-41075, 2024.
Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023.
Wu, H., Jing. Y., Cheang, C., Chen, G., Xu, Li, X., Liu, M., Li, H., and Kong, T. Unleashing large-scale video generative pre-training for visual robot manipula- tion. arXiv preprint
Parisi, S., Rajeswaran, A., Purushwalkam, S., and Gupta, A. The unsurprising effectiveness of pre-trained vision mod- els for control. In international conference on machine learning, pp. 17359-17371, PMLR, 2022.
1at::2222..111333 2023a.
Wu, J. Z., Ge, Y., Wang. X., Lei, S. W., Gu, Y., Shi, Y.. Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7623- 7633, 2023b.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference Computer Vision, pp. 4195-4205, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Xiang, W., Yang, H., Huang, D., and Wang, Y. Denois- ing diffusion autoencoders are unified self-supervised learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15802-15812. 2023.
Radosavovic, I., Xiao, T., James, S., Abbeel, P., Malik, J., and Darrell, T. Real-world robot learning with masked visual pre- pre-training. In Conference on Robot Learning, pp- 416 426. PMLR, 2023.
Yadav, K., Majumdar, A., Ramrakhya, R., Yokoyama, N., Baevski, A., Kira, Z., Maksymets, O., and Batra, D. Ovrl- v2: A simple state-of-art baseline for imagenav and ob- jectnav. arXiv preprint arXiv:2303.0777 2023a.

---

Video Prediction Policy
Yadav, K., Ramrakhya, R., Majumdar, A., Berges, V.-P. Kuhar, S., Batra, D., Baevski, A., and Maksymets, O. Offline visual representation learning for embodied nav- igation. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023b.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J.. Yang, Y., Hong, W. Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an ex- pert transformer. arXiv preprint arXiv:2408.0660 2024.
Ye, S., Jang, J., Jeon, B., Joo, S., Yang. Peng, B., Mandlekar, A., Tan, R., Chao, Y.-W., Lin, B. Y., et al. Latent action pretraining from videos. arXiv preprint arXiv:2410. 1758, 2024.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta- wotadworld: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Zhang, H., Ding, P., Lyu, S., Peng, Y., and Wang, D. Gevrm: Goal-expressive video generation model for robust visual manipulation. arXiv preprint wrXiv2202.00966 2025.
Zhao, W., Ding, P., Zhang. M., Gong, Z., Bai, S., Zhao, H., and Wang. D. Vlas: Vision-language-action model with speech instructions for customized robot manipulation. arXiv preprint arXiv:2502.13508 2025.
12

---

Video Prediction Policy
For your convenience, videos be found atttss///stee-ppallcitooooppl hitilg..itt.........g io
Code can be found at supplementary materiel.
A. Real-world experiments
A.1. Panda Maniplation
On the Franka Panda platform, we gathered demonstrations by telcoperating the Panda robotic arm using a space mouse. we collected 2k trajectories for over 30+ tasks of categories including picking, placing, pressing, routing, opening, and closing. Detailed success rates for each task in seen and unseen settings are shown Table 6.
Diffusion
Seen Tasks
Susie
GR-1
VPP
Policy
Pick
0.36
0.56
0.52
0.90
Place
0.40
0.42
0.38
0.86
Press
0.65
0.90
0.80
0.85
Route
0.40
0.55
0.50
0.75
Drawer
0.45
0.60
0.60
0.85
Average
0.425
0.563
0.519
0.856
Diffusion Policy
Unseen Tasks
Susie
GR-1
VPP
Pick
0.24
0.40
0.32
0.80
Place
0.12
0.44
0.32
0.72
Press
0.50
0.60
0.60
0.80
Route
0.20
0.50
0.50
0.70
Drawer 0.40
0.50
0.40
0.60
Average
0.250
0.463
0.388
0.737
Table Specific success rate at category level. seen tasks, We evaluate pick and place tasks 50 times and other tasks 20 times respectively. In unseen tasks, evaluate pick and place tasks 25 times and other tasks 10 times respectively
(a) Data collection with Space Mouse
(b) Data collection with Vision Pro
Figure 7. Data collection setups.
A.2. Dexterous Manipulation
To collect data for dexterous manipulation, employ Vision-Pro capture the finger joint movements of the human hand, which are then retargeted to our 12-degree-of-freede freedoo dexterous hand. This setup enables human operator to directly control the dexterous hand during various manipulation tasks. We collected 4.0k trajectories over 100+ tasks of 13 categories, including picking, placing, cup-upright, relocating, stacking, passing, pressing, unplugging, opening, and closing, low-level PD controller used to smooth the trajectories generated by VPP.
The detailed success rates for each task category in both seen and unseen settings are shown in Table 7.

---

Video Prediction Policy
Diffusion Policy
Seen Tasks
Susie
GR-1
VPP
Pick
0.38
0.61
0.48
0.83
Pick&Place
0.35
0.55
0.40
0.79
Cup-upright
0.00
0.00
0.00
0.64
Relocate
0.28
0.44
0.16
0.80
Stack
0.00
0.08
0.00
0.64
Pass
0.040
0.00
0.00
0.48
Press
0.68
0.96
0.64
0.96
Unplug
0.00
0.00
0.00
0.52
Drawer
0.40
0.64
0.48
0.72
Average
0.287
0.450
0.319
0.749
Diffusion Policy
Unseen Tasks
Susie
GR-1
VPP
Pick
0.12
0.42
0.26
0.75
Pick&Place
0.08
0.32
0.20
0.68
Cup-upright
0.00
0.00
0.00
0.40
Relocate
0.12
0.32
0.12
0.76
Stack
0.00
0.00
0.00
0.56
Pass
0.00
0.00
0.00
0.32
Press
0.44
0.76
0.40
0.88
Unplug
0.00
0.00
0.00
0.20
Drawer
0.28
0.44
0.24
0.56
Average
0.110
0.328
0. 0.159
0.605
Diffusion Policy
Tool-use Tasks
Susie
GR-1
VPP
Spoon
0.0
0.4
0.3
0.9
Hammer
0.2
0.2
0.1
0.6
Drill 0.0
0.1
0.2
0.8
Pipette 0.0
0.0
0.0
0.4
Average 0.05
0.23
0.15
0.68
Table 7. Specific success rate at category level. In seen tasks, We evaluate pick and place tasks 100 times and other tasks 25 times respectively. In unseen tasks, evaluate pick and place tasks times other tasks 20 times respectively. evaluate each tool-use use task for 10 times.
B. Video Prediction Model
B.1. Datasets Sample Ratios
Given the varying quality and scale of these datasets, have introduced different sample ratios to appropriately balance the influence of different datasets, similar to (Team et al., 2024). Detailed information is shown in Table 8.
B.2. Quantitative result on Prediction Quality
Our TVP models successfully predict future frames validation datasets across diverse manipulation tasks, with some prediction results visualized in Appendix B.3. Additionally, we evaluate the quantitative FVD metric (Unterthiner et al., 2018) on the bridge datasets (Ebert et al., 2021), following the evaluation settings in Seer (Gu et al., 2023). The results are shown in Table 9. Surprisingly, our model easily outperforms the previous TVP model. We attribute this improvement to our use of the pre- pre-traine trained video foundation model SVD (Blattmann et al., 2023a), which the earlier TVP model did not leverage, giving us a significant advantage.
2

---

Video Prediction Policy
Dataset Type
Name
Trajectory Numbers
Smaple Ratio
Internet
Something-
191,642
0.30
Human Maniplation
something-v2
RT-1
87,212
0.15
Bridge
23,377
0.15
Internet
BC-Z
43,264
0.08
Robot
Taco-Play
3,603
0.01
Datasets
Jaco-Play
1,085
0.01
Calvin-ABC
18,033
0.10
Metaworld
2,500
0.05
Self-Collected
Panda Arm
2,000
0.05
Datasets
Dexterous Hand
2,476
0.10
Total
375,192
1.00
Table We outline the dataset scales and sample ratios used for training our manipulation text-guided video prediction model. Following (Gu al., 2023), exclude 5,558 bridge trajectories and 2,048 someltinn--mmetiig--- trajectories during training, reserving them for validation. For all other datasets, 3% of the trajectories are excluded and used as validation datasets.
Bridge FVDI
VideoFusion 501.2
Tune- A Video
Seer
VPP
515.7
246.3
41.4
Table Quantitative evaluation of prediction quality on bridge datasets. The results of VideoFusion (Luo et al., 2023), Tune-A-Video (Wu al., 2023b), Seer (Gu al., 2023) copied from (Gu al., 2023).
B.3. More Visualization of Complete Prediction Results
We present additional visualizations of prediction results from our tne-tuned manipulation TVP model. Predictions on human manipulation datasets are displayed in Figure 8, and those on robotic manipulation datasets are illustrated in Figure 10. All trajectories are sampled from the validation datasets and are predicted using the same manipulation TVP model. Each sample was denoised in 30 steps using classifier-free guidance set 7.5, described in (Gu al., 2023). Our TVP model predicts a horizon of 16, and we visualize 8 frames at skip step of due to space constraints.
B.4. More Visualizations of Predictive Representations
We visualize the intermediate predictive representations through one-step direct predictions. Additional visualizations can be found in Figure 9. As discussed in the experimental section, while the textures and details in the one-step forward videos are not precise, they still offer valuable insights into physical evolution. The movements of objects and robot arm itself already can be reflected in the visualized representations.
C. More Details for Experiments
C.1. Structure details
We provide the VPP architecture and hyperparameter setting details in four evaluate environments, as shown in Table 13. The transformer block in TVP follows the setting in (Blattmann et al., 2023a), and the rest of the hyperparameter Diffusion Transformer follows the work (Reuss et al., 2024).
C.2. More ablation
In this section, we present additional ablation experiments conducted under the ABC-D setting of CALVIN (Mees et al., 2022).
Ablation on the video former entails the removal of the Temporal-attn module from the Video Former while maintaining all other configurations same as VPP. The results, displayed in Table 12, demonstrate that the Temporal-attn module could enhance the temporal comprehension capabilities of the Video Former.
Ablation 2 on the number of denoising steps introduces 2-step denoising process the TVP to derive the predictive
3

---

Video Prediction Policy
visual representation. The outcomes summarized in Table 12, revealing that the 2-step process did not yield superior performance. We hypothesize this is because single denoising step suffices to generate effective representation for trajectory prediction in our configuration. Additionally, the 2-step denoising process nearly doubles the inference time and reduces the control frequency by half. Due to these factors, we opted for one-step direct encoder in our main experiments.
Single-view Ablation evaluate the Calvin ABC-D task using only a single observation viewpoint (static view) and find that the success rate for Task reaches 3.58. This even surpasses the success rate achieved by the state-of-the-art 3D Diffuser Actor, which utilizes two viewpoints along with depth images.
Ablations on using different layers of features The average task completion length are listed in Table 10.
Ablations on using different diffusion time-step The average task completion length are listed in Table 11.
Calvin abc-d
Layer-3
Layer-6
Layer-9
Layer-12
VPP
Avg. Len
3.72
3.88
4.29
4.05
4.33
Table 10. Ablations different layers of features.
Calvin abc-d
Time-step 10
Time-step 20
Time-step 30
Avg. Len
4.21
4.33
4.25
Table 11. Ablations on the Use of different diffusion time-step.
Tasks completed in row
Method
1
2
3
4
5
Avg. Len
VPP(Ours)
0.965
0.909
0,866
0.820
0.769
4.33
VPP(Single-vieww
0.909
0.815
0.713
0.620
0.518
3.58
Ablation.
0.949
0.900
0.839
0.780
0.714
4.18
Ablation.2
0.951
0.904
0.840
0.777
0.718
4.19
Table 12. More ablation studies.
C.3. Baseline Implementations
The baseline methods, including RT-1 (Brohan al., 2022), GR-1 (Wu et al., 2023a), and Diffusion Policy (Chi al., 2023). implemented based their official repositories. For comparison with Susie (Black et al., 2023) in both the Metaworld and real-world manipulation scenarios, we adopt InstructPix2Pix (Brooks et al., 2023) as the future frame predictor and use an image-goal Diffusion Policy (Chi et al., 2023) to generate the state sequence.

---

Vidco Prediction Policy
Type
Name
Calvin
Metaworld
Franka Panda
Xhand
Video lens
16
8
16
Prediction
Action shape
10 7
4 4
10 7
10 18
Language shape
20 512
20 512
20 512
20 512
TVP
Image shape
256 256
256 256
256 256
256 256
Token shape
16 14 384
8 28 384
14 16 384
14 16 384
Input dim
1280
1280
1280
1280
Video Former
Latent dim
512
512
512
512
Num heads
8
8
8
num Layers
6
6
6
Latent dim
384
384
384
384
Condition shape
225 384
225 384
225 384
225 384
Num heads
8
8
8
Diffusion Transformer
Encoder Layers
4
4
4
4
Decoder Layers
4
4
4
4
Sampling Steps
10
10
10
10
TVP batchsize
4
4
4
4
Policy batchsize
76
64
128
128
Hyperparameter
Epoch nums
12
30
30
40
Learning rate
1 10-4
5
10-5
10-4
1 * 10-4
Table Hyper-parameter in Video Prediction Policy (VPP).

---

Video Prediction Policy
9
a moving bottle away from bottle
(b) moving silicone towards the camera
(c) moving toy closer plastic glass
(d) pouring liquid into cup
(e) pouring milk into glass
(f) turning the camera right while filming jeep
(g) pretending to pick a tennis ball up
(h) pulling toy car from left to right
putting something similar
tearing paper into two pieces
Figure Visualization of video prediction results on Internet human manipulation validation datasets with 30 steps de-noising. The green frames indicate the ground truth while the red frames indicate the predicted futures. 7oom in for better comparisons.
(a) place the grasped object in drawer.
(b) close drawer.
(d) stack the green block red block Figure Visualization Predictive representations. Green frames represent the ground truth, red frames correspond the predicted future states, and blue frames illustrate the visualized predictive representations. Zoom in for better comparisons.
(c) pick the mango and place in blue plate
6

---

Vidco Prediction Policy
(a) open bottom drawer
(b) place apple into bottom drawer
(c) place eraser in purple bowl
(d) place the white sponge in the ceramic cup
(e) put bowl on plate cardboard fence
(f) put eggplant into pot or pan
(g) Place the grasped object into the drawer
(h) close the drawer
(i) Put the grey circle on the red pole
open the drawer
(k) pick carrot
(l) pick the blue block
(m) Put the green block above the red block
(n) Place the purple onion onto the basket
(o) pick the yellow ball to the empty hole of the board
(p) Receive the orange from hand and place it into the basket
Figure 10. Visualization of video prediction results on robotic datasets with 30 steps de-noising. The green frames indicate the ground truth while the red frames indicate the predicted futures. (a)-()) are sourced from internet robotic while (k)-(p) are from self-collected datascts. Zoom in for better comparisons.
7


Reviews:
[{"review_id": "qMQMxUuhH5", "reviewer": "Reviewer_xoHf", "summary": "The paper introduces the Video Prediction Policy (VPP), a versatile robotic policy that enhances robot control by utilizing predictive visual representations generated by text-guided video prediction models (TVPs). VPP employs a two-stage methodology: first, fine-tuning a text-guided video prediction model on manipulation datasets, and second, integrating these predictive representations into a diffusion policy. The results show that VPP surpasses existing methods across both simulated and real-world benchmarks, achieving significant gains in task success rates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added detailed discussions and additional experiments: \n\n\n**Q1: Control frequency comparisons to UniPi**\n\nANS: We are afraid misunderstandings exist on the UniPi work. In page 5, sec 3.2 of the original UniPi paper, the authors state: \"we use an **open-loop** controller in all our experiments in this paper.\"The IDM model takes two predicted images as input without any feedback from the environment, making it an open-loop control system. In contrast, VPP can replan in close-loop based on current observation at high frequency(7-10Hz) by direct using representations in the first forward pass.\n\n---\n\n**Q2: About the comparison on Table 3: Voltron is only 83M while SVD is 1.5B**\n\nANS: We respectfully offer a different perspective on this comparison: Previous vision encoders for embodied control were trained with specifically designed objectives on small networks; Video diffusion models with billions of parameters are pretrained on large-scale internet datasets—why not leverage their power? Beside, pretraining these vision encoder with billions of parameters from scratch requires enormous computational resources, which we really can not afford. We hope this clarification can addresses your concern!\n\n---\n\n**Q3: About the SOTA on CALVIN benchmark**\n\nANS: We list several works that achieve strong performance on CALVIN, including some concurrent papers. Even considering these, to the best of our knowledge, VPP remains the SOTA method on CALVIN at the time of submission.\n\n|     CALVIN ABC-D  | Release Time|  Avg. Len | \n| :-- | -- |-- |\n|    3d-diffuser actor     | Feb. 16, 2024 |  3.35 |\n|      RoboUniview     |   Jun. 27, 2024  | 3.64 |\n|      CLOVER     |   Sep. 13, 2024  | 3.53 |\n|    Vidman (Concurrent*)      |  Nov.14, 2024   | 3.42|\n|    VPP      |   | 4.33|\n\n*In ICML guidelines, papers released within 4 months to the submission DDL are concurrent and author are not required to compare to them.\n\n---\n\n**Q4: Comparison and differences to Vidman**\n\nANS: (1) Performance: VPP significantly outperforms Vidman, even as a concurrent work. (2) Training strategy: Vidman does not fine-tune its video model on the downstream domain, likely leading to worse video predictions and lower performance. (3) Architecture: Vidman directly uses action tokens to attend to numerous video model tokens, whereas VPP employs a VideoFormer to aggregate tokens, assisting action learning. \n\nAblations confirm that removing downstream fine-tuning and VideoFormer both degrade performance:\n\n|               | Avg.Len|\n| :-- | -- |\n|    Vidman    | 3.42  |\n|    VPP w/o video finetuned in domain    | 3.31 |\n|    VPP w/o video former   | 3.86 |\n|    VPP   | 4.33 |\n\n---\n\n**Q5: Comparisons to the paper \"Video diffusion models are training-free motion interpreter and controller.**\n\nANS: We carefully reviewed this paper and found that it focuses on video motion generation, which is fundamentally different from robotics tasks. While it explores controlling motion generation via video latent, our work emphasizes the predictive capability of video representations, which is crucial for embodied AI.\n\n---\n\n**Q6: How well does video model generalize to different robotic embodiments? Consider evaluating VPP on CALVIN by training TVP model exclusively on CALVIN data**\n\nANS: Thank you for the insightful question! As mentioned in Q4, removing CALVIN video in TVP fine-tuning reduces performance. However, in this setting, representation inside TVP models still outperform the representations from static SVD VAE model. We argue that fine-tuning the TVP model on collected robot data better utilizes the available robotic data.\n\n|               | Avg.Len|\n| :-- | -- |\n|    SVD's VAE    | 2.58 |\n|    VPP w/o video FT   | 3.31 |\n|    VPP   | 4.33 |\n\n---\n\n**Q7: Is video model freeze in the second stage of training?**\n\nANS: Yes! We will make it more clear in the method section.\n\n---\n\n**Q8: Prediction horizon of TVP and affection on the downstream tasks?**\n\nANS: Typically we set the prediction horizon longer than the action horizon. In Section 4.3, VPP predicts 16 video frames at 0.2s intervals while outputting 10 actions at 0.1s intervals. Ablating to a 0.1s × 16 frame prediction had only a slight effect on performance (4.33 → 4.21).\n\n---\n\n**Q9: Which layer's representations provide the most informative features.**\n\nANS: Thank you for the insightful suggestion! We perform additional ablation study by conditioning policy on different layers inside up-sampling block. The results shows that the most informative ones lie in the middle of the SVD model. Our feature aggregation mechanism avoids manual selection while achieving the best results.\n\n|               | Avg.Len|\n| :-- |  --   |\n|    Layer-3    | 3.72  |\n|    Layer-6    | 3.88  |\n|    Layer-9    | 4.29  |\n|    Layer-12   | 4.05  |\n|    VPP   | 4.33 |\n\n---\n\nThank you again for your time and effort in reviewing our work!", "claims_and_evidence": "The claim regarding improved control frequency may be subject to scrutiny. For methods such as UniPi, which employ 'Video Prediction + Inverse Dynamics Model (IDM)', although multiple denoising steps are required to generate a full video sequence for enhanced downstream control, they typically only need to generate the video once (i.e., a video plan spanning a relatively long horizon). Subsequent multi-step closed-loop rollouts rely solely on the lightweight IDM, which can operate very efficiently. In contrast, VPP appears to invoke the large TVP model at every control step, potentially incurring higher computational costs. To substantiate claims about improved efficiency, evaluations focusing on the time consumed for task completion would provide a more meaningful and supportive metric.", "methods_and_evaluation_criteria": "Leveraging inherent dynamics within video diffusion models is a promising approach for robotic control.\nThe evaluation framework in this paper encompasses CALVIN, MetaWorld, and real-world assessments involving both gripper and dexterous hands, providing a comprehensive evaluation of the proposed method.", "theoretical_claims": "No theoretical claims are made in the paper.", "experimental_designs_or_analyses": "1. The analysis provided in Table 3 lacks persuasiveness and cannot be regarded as a fair comparison on the 'Pre-training Type'. It is noteworthy that the largest model in Voltron is a ViT-Base architecture with approximately 83 million parameters, while the Stable Video Diffusion (SVD) model employed in the proposed method exceeds 1.5 billion parameters. This substantial discrepancy in model scale undermines the validity of the comparison and calls into question the robustness of the findings.\n2. '3D Diffuser Actor' is no longer the state-of-the-art method in the CALVIN benchmark. Further inclusion and comparison with more advanced methods will highlight the superiority of the proposed method.", "relation_to_broader_scientific_literature": "Discussed in the 'Essential References' section.", "essential_references_not_discussed": "A notable similarity exists between the proposed method and VidMan [1], as both approaches involve fine-tuning video diffusion models using robotic data and leveraging features derived from the denoising process, integrated with cross-attention mechanisms, for diffusion-based action prediction. Furthermore, the core motivation of this paper—leveraging predictive visual representations from video diffusion models for robotic tasks—has also been extensively explored and discussed in prior literature [2].\n\n\n[1] Wen, Youpeng, et al. \"Vidman: Exploiting implicit dynamics from video diffusion model for effective robot manipulation.\" NeurIPS 2024\n[2] Xiao, Zeqi, et al. \"Video diffusion models are training-free motion interpreter and controller.\" arXiv preprint arXiv:2405.14864.", "other_strengths_and_weaknesses": "The evaluations with the dexterous hands are barely seen in recent literature, which is valuable for validating the proposed method.", "other_comments_or_suggestions": "Is line.117, 'we observe that the open-sourced SVD', \"W\" should be capitalized.", "questions_for_authors": "1. For the second stage of training as described in the paper, are the video former and DiT policy trained in an end-to-end manner with the TVP-based backbone, or can the TVP be frozen during this stage? Clarifying this would provide insight into the flexibility and efficiency of the training process.\n\n2. It is worth exploring whether the TVP-based backbone can generalize to unseen environments. In the current version, the TVP is extensively trained on specific downstream datasets (i.e., CALVIN and MetaWorld). How well does it generalize to novel scenarios or different robotic embodiments? The authors could consider evaluating VPP on CALVIN by training it exclusively on CALVIN data during the second stage. \n\n3. What is the prediction horizon (i.e., the time duration of predicted videos) of the TVP? Should this horizon be adjusted based on the control frequency of the system? To what extent will it affect the downstream performance?\n\n4. The feature aggregation process appears to have a significant impact on performance, as evidenced by the performance drop on CALVIN from 4.33 to 3.6. It would be valuable to investigate which layer's representations provide the most informative features for robotic control.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nhzcaoFyAk", "reviewer": "Reviewer_6CJo", "summary": "The paper is about fine-tuning a pretrained image to video diffusion model on video-caption datasets that focus on object manipulations. The image to video model is turned into an image plus text to video model through the fine-tuning. This diffusion model is then used as a feature extractor (no iterative de-noising) to obtain future frame representations for a given robotics task (defined by start frame and textual command). These future representations, tagged predictive visual representations, are used as inputs to train a separate diffusion policy model on various robotics tasks. The policy model is trained separately on ground-truth robot trajectories, generating action sequences given starting image and textual command. The two detached training phases allow the diffusion model to be trained on a large-scale with weak supervision, while the policy network is learned at a task level under full supervision. Experiments on two synthetic and two real-world benchmarks demonstrate the improved performance achieved by this method over existing work.\n\n## update after rebuttal\n\nThe authors have successfully addressed the concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper! Based on your review, we added a detailed discussion and additional experiments. \n\n\n**Q1: About the details on method: what diffusion time-step is used to extract TVP features during inference?**\n\nThank you for your question. In our pipeline, we set the SVD noise scale and diffusion timestep to 20. We conducted preliminary experiments with diffusion timesteps of 10, 20, and 30 and found that 20 yielded slightly better performance. However, the overall performance of VPP is not sensitive to the choice of timestep. The results for different timesteps are shown below:\n\n|   CALVIN ABC-D       | Avg.Len|\n| :-- | -- |\n|    VPP time-step 10  | 4.21  |\n|    VPP time-step 20  | 4.33  |\n|    VPP time-step 30  | 4.25 |\n\n---\n\n**Q2: In Table 4, do you have results with SVD pretrain, but only internet data training? I.e. the impact of SVD pre-training for the robotics tasks.**\n\nANS: Thank you for the insightful question! Following your suggestion, we fine-tuned the SVD model using only internet data, without incorporating downstream robotics data. As shown below, the results indicate a clear performance decline. We believe this is because **video prediction quality plays a crucial role in action learning**. In the VPP framework, fine-tuning the video model on robot datasets enhances video prediction quality within the specific domain, allowing it to better capture the dynamics of robotic data and potentially improve performance.\n\n\n\n|               | Avg.Len|\n| :-- | -- |\n|    VPP w/o internet data    | 3.97  |\n|    VPP w/o down-stream robot dataset (newly added)   | 3.31  |\n|    VPP   | 4.33 |\n\n\n---\n\n**Q3: Please add table comparing against inference speed of baselines.**\n\nANS: Thank you for the constructive suggestion! Follow your suggestion, We add a comparison of the inference speed here. All inference time is evaluated on single NVIDIA 4090 GPU and average on 100 runs. We can notice that VPP achieves the best performance while keeping high frequency. Other  method containing video/imgae diffusion (e.g., Susie/Uni-Pi) requires long time to denoise a complete video.\n\n|     CALVIN ABC-D  | inference Time|  Avg. Len | \n| :-- | -- |-- |\n|    Diffusion policy      | ~100ms |  0.56 |\n|    3d-diffuser actor     | ~600ms |  3.35 |\n|    Susie    |  ~5100ms | 2.69 |\n|    GR-1     |  ~90ms   | 3.06 |\n|    MDT      |  ~110ms  | 1.55 |\n|    Uni-Pi   |  ~5500ms  | 0.92 |\n|    VPP      |  ~140ms  | 4.33|\n\n---\n\nThank you again for your time and effort in reviewing our work! We hope our clarification can solve all your concerns, and we are always ready to answer any further questions!", "claims_and_evidence": "Yes. The key claim is that a video generation diffusion model, adequately fine-tuned, can provide representations useful for robotics downstream tasks. Extensive experimentation (on synthetic and real-world benchmarks) is provided as evidence to demonstrate usefulness of such representations.", "methods_and_evaluation_criteria": "Yes. The method of using future prediction is well motivated (and is well explored in concurrent literature). Evaluation criteria follows standard settings and aligns with results in prior works.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Follows standard evaluation settings from prior work.", "relation_to_broader_scientific_literature": "Highly relevant to several recent works which explore similar settings. This work could serve as a strong motivation for various future works.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n\n* Interesting idea of using future predictions in representation space with efficient, non-iterative feature extraction from a video diffusion model.  \n\n* Clear presentation of methodology: the representation learning as well as fine-tuning stages are explained well with good use of figures for explanation.  \n\n* Clear experimental setup following established benchmarks from prior work. Easy to observe the improvements from video diffusion representations for robotics tasks. \n\nWeaknesses\n\n1. Details on method\n\n    * What diffusion time-step is used to extract TVP features during inference? \n\n2. Additional ablations\n\n    * In Table 4, do you have results with SVD pretrain, but only internet data training? I.e. the impact of SVD pre-training for the robotics tasks.  \n\n3. Inference speed comparison missing\n\n    * Please add table comparing against inference speed of baselines", "other_comments_or_suggestions": "Proposing weak accept given the several minor concerns listed in weaknesses above.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xb9NKWXsKf", "reviewer": "Reviewer_uxTn", "summary": "This paper introduce Video Prediction Policy(VPP), a robotics framework that leverages video diffusion model to capture the dynamic presentation vital to policy training. VPP consists of a two-stage process: i) a general video model is fine-tuned into a text-guided video prediction (TVP) model using large-scale human and robotic manipulation datasets, ii) a robotic policy is trained on predictive representations from the TVP model, enabling implicit inverse dynamics learning.\nVPP is evaluated on both simulate and real-world environments against mainstream baselines. The comparison shows that VPP i) achieves a higher success rate in manipulation tasks ii) enable generalize in unseen tasks.\n\n## update after rebuttal\nI confirm my score. Authors addressed comments and added clarity and results to the original submission.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper! Thank you for your support on our work!\n\n\n**Q1: Regarding the data efficiency of VPP model**\n\nANS: Thank you for your constructive question! In Table 1 of the original paper, we conducted experiments on the CALVIN benchmark using only 10\\% of the standard dataset. Even with this limited data, VPP achieved performance comparable to previous SOTA models trained on the full dataset. These experiments demonstrate the data efficiency of VPP compared to previous methods, which stems from its powerful video pretraining process. Below, we provide additional comparisons with other methods:\n\n\n|     CALVIN ABC-D  | Data percentage|  Avg. Len | \n| :-- | -- |-- |\n|      GR-1          |   10\\%   | 1.41 |\n|      GR-1          |   100\\%  | 3.06 |\n| 3d-diffuser actor  | 100\\%    |  3.35|\n|      CLOVER        |   100\\%  | 3.53 |\n|Vidman (Concurrent*)|  100\\%   | 3.42 |\n|    VPP (ours)      | 10\\%     | 3.25 |\n|    VPP (ours)      | 100\\%    | 4.33 |\n\n\n---\n\n**Q2: While the results suggest that VDM-based representations capture useful motion dynamics, the paper does not explicitly demonstrate how these representations encode physical laws.**\n\nANS: Thank you for your insightful question! To directly illustrate the physical dynamics encoded in the VDM representation, we visualize the one-step denoised VDM features in Figures 4 and 9 of the original paper. These visualizations show that the learned representations consistently maintain physical consistency and align with intuitive physical laws.\n\nFor example, when a block is grasped, it moves along with the robot arm, and the color of the poured liquid matches the liquid in the bottle. We hope this analysis clarifies our claims.\n\n---\n\n**Q3: VPP's performance is highly dependent on the quality and predictive power of the video diffusion model. If the video diffusion model cannot accurately predict future scenes, VPP's performance may be limited.**\n\nANS: Thank you for your insightful comment—we agree with your observation! We have also noted that policy performance strongly correlates with video prediction quality. However, from another perspective, we believe **this dependency can be a strength**. As pretrained video models continue to improve, their enhanced prediction capabilities allow us to transfer generalization abilities learned from internet-scale data to embodied tasks, which is really a good thing in the data-limited robotics domain.\n\n---\n\n**Q4: Fine-tuning the video diffusion model requires a lot of computing resources, taking 2-3 days on eight NVIDIA A100 GPUs.**\n\nANS: Since the pretrained SVD model has 1.5 billion parameters, we use an A100 node for fine-tuning. However, by employing parameter-efficient techniques such as LoRA, we can further reduce the computational demand, which is also implemented in the code provided in the supplementary materials.\n\n---\n\n**Q5: It may still be difficult to accurately capture the complex physical dynamics over long time scales.**\n\nANS: Thank you for the insightful comment! As pretrained video models continue to improve, we believe their ability to predict complex dynamics will also improve. Furthermore, the VPP framework does not require long-horizon predictions, as the video model can perform high-frequency replanning (re-prediction). As described in Section 4.1, we predict 16 frames with a 0.2s time interval, resulting in a short prediction horizon of 3.2 seconds.\n\n---\n\n**Q6: Does the performance increase with the number of denoising steps?**\n\nANS: This is a great question! The number of denoising steps presents a trade-off: while more steps can lead to higher-quality RGB images, they also result in a much lower control frequency. We explored this in an ablation study presented in Table 10 (Ablation.2) of the original paper. Our findings show that using two denoising steps yields similar performance to a single-step approach but at nearly half the control frequency.\n\nTo balance performance and efficiency, we ultimately adopted the one-step direct encoder in our main experiments. We did not experiment with higher denoising steps, as they would further reduce control frequency, making the policy impractical for real-world deployment.\n\n\n---\nThank you again for your endorsement in our work! We will keep polishing the work to make it meet the highest standard!", "claims_and_evidence": "- In the introduction part, the paper suggests that VPP requires few demonstrations to align the robot's action space with visual features, but the extent of this data efficiency is not well quantified.\n- While the results suggest that VDM-based representations capture useful motion dynamics, the paper does not explicitly demonstrate how these representations encode physical laws.", "methods_and_evaluation_criteria": "VPP and evaluation crieria are generally appropriate and well-aligned with the problem of learning generalist robotic policies. VPP is evaluated in Calvin ABC-D, metaworld, and real-world robot arm and dexterous hand, which show its ability to generalize and do complex tasks.", "theoretical_claims": "The theoretical foundation of VPP is not formally established through rigorous proofs or mathematical derivations.", "experimental_designs_or_analyses": "The paper claims that VPP requires few demonstrations, but it does not test performance under limited-data conditions.", "relation_to_broader_scientific_literature": "The main contribution of this paper is the Video Prediction Policy (VPP), a new approach to learning general robotic policies using predicted visual representations inside video diffusion models (VDMs). VPP achieves significant performance gains in both simulation and real-world tasks by learning an implicit inverse dynamics model on future representations predicted by VDMs.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths:**\n- VPP achieves good performance in both simulation and real-world tasks by learning an implicit inverse dynamics model on future representations predicted by VDMs, and avoids the need for multiple denoising steps as in previous work, so it can run in closed-loop at high frequency.\n- Able to leverage physical knowledge from pre-trained video generation models and Internet manipulation datasets.\n\n**Weakness:**\n- VPP's performance is highly dependent on the quality and predictive power of the video diffusion model. If the video diffusion model cannot accurately predict future scenes, VPP's performance may be limited.\n- Fine-tuning the video diffusion model requires a lot of computing resources. Fine-tuning the video model takes 2-3 days on eight NVIDIA A100 GPUs.\n- It may still be difficult to accurately capture the complex physical dynamics over long time scales.", "other_comments_or_suggestions": "Some typos:\nIntroduction part: “Since we direct use the internal representation and avoid the need for multiple denoising steps as in previous work”, “direct” should be “directly”", "questions_for_authors": "Does the performance increase with the number of denoising steps increase?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HCU6jAZWHC", "reviewer": "Reviewer_Y6gd", "summary": "This paper proposes to use the intermediate features of video diffusion model as a visual encoder for generalizable robot action prediction. Firstly, a pre-trained video generation model is fine-tuned on robotic dataset. Then, the latent features of the first denoising step are aggregated for down-steam action prediction. Experiments in simulator and real world proves the effectiveness of the method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our paper! Thank you for your support in our work! \n\n**Q1: What's the percentage of each task category in the training and testing set?**\n\nANS: Thank you for the questions! The specific number of evaluated trajectories can be found in Appendix A in the original paper:\n\n|           Skill Name      | Pick(Grasp)|  Pick&Place | Cup-upright |Relocate/Pass    |Stack       |Drawer       |\n| :-- | -- |-- |-- |-- |-- |-- |\n|         Test percentage(evaluate)                   | 21.1\\%(100/475)|  21.1\\%(100/475) | 5.26\\%(25/475) |5.26\\%(25/475) |5.26\\%(25/475) |5.26\\%(25/475) |5.26\\%(25/475) |\n|           Press          | Unplug     |   Water pouring   |   Hammer     | Drill       |Pipette    | Spoon      |\n|               5.26\\%(25/475)          | 5.26\\%(25/475)|  5.26\\%(25/475) | 5.26\\%(25/475) |5.26\\%(25/475) |5.26\\%(25/475) |5.26\\%(25/475) |5.26\\%(25/475) |\n\n\nWe also count the trajectory numbers of each category in the training datasets:\n\n|           Skill Name      | Pick(Grasp)|  Pick&Place | Cup-upright |Relocate/Pass    |Stack       | Drawer       |\n| :-- | -- |-- |-- |-- |-- |-- |\n|         Train percentage                   | 15.2\\%|  40.5\\% | 2.5\\% | 7.6\\% |10.1\\% | 5.1\\% |\n|           Press          | Unplug     |   Water pouring   |   Hammer     | Drill       |Pipette    |  Spoon      |\n|               2.5\\%          | 2.5\\%|  5.1\\% | 1.2\\% | 1.2\\% | 2.5\\% |3.8\\% |\n\n---\n\n**Q2: Please clarify more about the improvement compared to GR-1 [1]. How much of the improvement is due to the use of the diffusion model?**\n\nANS: Thank you for your insightful question. To ensure a fair comparison between the diffusion-based approach and the autoregressive method, we conducted an ablation study in which we replaced the policy decoder in GR-1 with the same diffusion head used in VPP while retaining the original autoregressive video prediction component. The results on the CALVIN ABC-D benchmark are shown below:\n\n|               | Avg.Len|\n| :-- | -- |\n|    GR-1 (Auto-regressive)    | 3.06 |\n|    GR-1 (diffusion policy head)    | 3.42 |\n|    VPP (ours)   | 4.33 |\n\n We can summarize that the performance improvement brought by VPP stems from multiple factors, including the accurate future prediction enabled by the video diffusion model, the integration of the VideoFormer architecture, and the dynamic representations learned from diverse training data in pretrained SVD.\n\n---\n\n**Q3: I am curious about the tool-using tasks in real-world evaluation. Does the training set include tool-using tasks? Since both the text and figure show no tool-using tasks in training set (line 361, line 402).**\n\nANS: Initially, we did not include tool-use tasks in the training dataset. We observed that VPP could generate roughly correct trajectories (e.g., approaching the tool in Figure 6) in challenging tool-use tasks but consistently failed due to the high precision control required. \n\nTo address this, we collected 50–200 demonstrations for each tool-use task and fine-tuned both the video and action models. All baselines were also trained with the newly collected demonstrations.\nAfter fine-tuning, our model achieved a higher success rate in tool-use tasks. For your convenience, we have visualized more video prediction results before/after finetuning on our website: https://sites.google.com/view/vpp-rebuttal\n\n---\n\n**Q4: Besides, could you please show some visualization of video prediction on tool-using. The current ones only show the hand approaching the tool (Figure 6).**\n\nANS: Yes! We have also visualized the more video prediction results for tool tasks at different stages on our website: https://sites.google.com/view/vpp-rebuttal\n\n---\nThank you once again for endorsing our work! We will continue refining it to meet the highest standards.", "claims_and_evidence": "1. Features from video diffusion model could better encode the dynamic information, thus are beneficial for action prediction.\nThis claim is supported by the comparison with baselines and the ablation on different visual encoders.\n\n2. Efficiency. The prediction speed is faster than previous video generation model based methods, which is proved by the control frequency.\n\n3. Generalization. The setting of CALVIN and real-world experiments validate the generalization ability of this method.", "methods_and_evaluation_criteria": "This paper uses two simulation environments and real-world experiments to evaluate the proposed method and baselines with same robotic data (excluding data for video model training). The metrics include successful rate and task completion length. The experiments are thorough for generalizable robot manipulation problems.", "theoretical_claims": "Do not apply.", "experimental_designs_or_analyses": "This paper uses the CALVIN benchmark with ABC-to-D setting and Metaworld to validate the effectiveness of the method in simulation. Real-world experiments on two different embodiments further evaluate the proposed approach. In each setting, all methods are trained with same amount of manipulation data. Seen and unseen tasks are both evaluated to show the generalization ability.", "relation_to_broader_scientific_literature": "This approach is related to open-world manipulation. Specifically, the use of video diffusion model for robot manipulation and visual representation learning for manipulation.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strength\n\n1. Leveraging the features from pre-trained video generation model to better predict dynamics is well-motivated. Visual representations suitable for robot manipulation should encode both static and dynamic information.\n\n2. The design of using latent features instead of the denoised video improves the efficiency and robustness of action prediction.\n\n3. The experiments are thorough. The evaluations in real world are impressive, which shows the effectiveness of VPP.\n\n### Weakness\n\n1. The majority of tasks belongs to pick-and-place category, which is relatively simple. What's the percentage of each task category in training and testing set.", "other_comments_or_suggestions": "1. Please clarify more about the improvement compared to GR-1 [1]. How much of the improvement is due to the use of diffusion model?\n\n2. I am curious about the tool-using tasks in real-world evaluation. Does the training set include tool-using tasks? Since both the text and figure show no tool-using tasks in training set (line 361, line 402). Besides, could you please show some visualization of video prediction on tool-using. The current ones only show the hand approaching the tool (Figure 6).", "questions_for_authors": "Please see above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces Video Prediction Policy (VPP), which learns implicit inverse dynamics model conditioned on predicted future representations inside Video Diffusion Models (VDMs). VPP enhances robot control by utilizing predictive visual representations generated by text-guided video prediction models (TVPs). VPP consists of a two-stage process: (i) a general video model is fine-tuned into a text-guided video prediction (TVP) model using large-scale human and robotic manipulation datasets, (ii) a robotic policy is trained on predictive representations from the TVP model, enabling implicit inverse dynamics learning. VPP is evaluated on both simulated and real-world environments against mainstream baselines. The evaluation shows that the representations learned by VPP are effective for downstream robotics tasks (i) achieving a higher success rate in manipulation tasks (ii) enabling generalization in unseen tasks (iii) being more efficient (control frequency) than previous video generation based methods.

All reviewers agree that the proposed method of using future prediction is well motivated and its a promising approach for robotic control. The experiments are thorough and the evaluation criteria follows standard settings and aligns with results in prior works. The paper is well written with a clear presentation of methodology: the representation learning as well as fine-tuning stages are explained well with good use of figures for explanation. This work could serve as a strong motivation for various future works.

The authors have appropriately addressed all the concerns raised by reviewers. All additional results presented during the rebuttal should be included in the final version of the paper.
Decision: Accept (spotlight poster)