VideoVLA: Video Generators Can Be Generalizable Robot Manipulators
Authors: Yichao Shen
Fangyun Wei
Zhiying Du
Yaobo Liang
Yan Lu
Jiaolong Yang
Nanning Zheng
Baining Guo
Venue: NeurIPS 2025 poster ()
Keywords: Robot manipulation
video generation
Primary area: applications

Abstract: Generalization in robot manipulation is essential for deploying robots in open-world environments and advancing toward artificial general intelligence. While recent Vision-Language-Action (VLA) models leverage large pre-trained understanding models for perception and instruction following, their ability to generalize to novel tasks, objects, and settings remains limited. In this work, we present VideoVLA, a simple approach that explores the potential of transforming large video generation models into robotic VLA manipulators. Given a language instruction and an image, VideoVLA predicts an action sequence as well as the future visual outcomes.  Built on a multi-modal Diffusion Transformer, VideoVLA jointly models video, language, and action modalities, using pre-trained video generative models for joint visual and action forecasting. Our experiments show that high-quality imagined futures correlate with reliable action predictions and task success, highlighting the importance of visual imagination in manipulation. VideoVLA demonstrates strong generalization, including imitating other embodiments' skills and handling novel objects. This dual-prediction strategy—forecasting both actions and their visual consequences—explores a paradigm shift in robot learning and unlocks generalization capabilities in manipulation systems.

ViA: Video Generators Can Be Generalizable Robot Manipulators
Yichao Shen 1.2- Fangyun Wei²t, Zhiying Du Yaobo Liang?, Yan Lu?,
Jiaolong Yang"t, Nanning Zheng", Baining Guo?
IAIR, Xi'an Jiaotong University Miirosooft Research Asia FFudan University
Visual Imaginations
Observation
oO
VideoVLA
"Move blue plastic bottle near sponge.
Instruction
Executions
Figure 1: Illustration of VideoVLA. Given language instruction and the current visual observation, VideoVLA jointly predicts the appropriate sequence of next actions and generates video content that illustrates how these actions will influence physical interactions in the environment. In addition to delivering strong performance on in-domain tasks, VideoVLA demonstrates robust generalization to novel objects and unseen skills. This capability stems from its use of pre-trained video generation models--dstinet
distinct from prior vision-language language- action approaches [ B 4 6 7 that primarily rely on pre-trained vision-languse -anguage understanding models-an as well its duje-objective strategy.
Abstract
Generalization in robot manipulation is essential for deploying robots in world environments and advancing toward artificial general intelligence. While recent vision- iision-language-cce actignnncionnnn models leverage large pre-trained understanding models for perception and instruction following, their ability generalize to novel tasks, objects, and settings remains limited. In this work, we present VideoVLA, a simple approach that explores the potential of directly transforming large video generation models into robotic VLA manipulators. Given language instruction and an image, VLA predicts an action sequence as well as the future visual outcomes. Built on multi-modal Diffusion Transformer, VideoVLA jointly models video, language, and action modalities, using pre-trained video generative models joint visual and action forecasting. Our experiments show that high-quality imagined futures correlate with reliable action predictions and task success, highlighting the importance of visual imagination in manipulation. VideoVLA demonstrates strong generalization, including imitating other embodiments' skills and handling novel objects. This dual-predictio strategy- visual consequences-eeplllee a paradigm shift robot learning and unlocks generalization capabilities in manipulation systems.
both actions and their
Microsoft Research Interns. *Corresponding Authors.
$ National Key Laboratory of Sluman-Maching Hybrid Augmented Intelligence, National Engineering Re- search Center for Visual Informationand Applications, and Institute Artificial Intelligence and Robotics. Xi' an Jiaotong University
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Generalization has long been a central goal in robot manipulation, representing a critical step toward achieving artificial general intelligence. The aspiration is for robots not only to perform tasks encountered during training but also to handle unseen tasks, manipulate novel objects, and operate in unfamiliar environments. This capability is critical for deploying robots in open open-world settings, where unpredictability and variability are the norm rather than the exception. Recently, researchers have begun investigating whether robotic systems exhibit an emergence capabilities begin to surface- by scaling both the volume of training data and the size of model parameters, analogous to the scaling laws observed in Large Language Models (LLMs) [ 9 This line of inquiry has led to the development of SsiionLLanguage-ettiin
generalization
(VLA) models
3 4 [0]. Unlike conventional robot manipulation architectures [I 12), VLA models typically incorporate billions of parameters, perceive complex visual environments, follow natural language instructions, and, crucially, leverage large pre-trained vision-language 13 14 [55, vision [8] or language [9] 20 21 models. These pre-trained models serve two purposes: (1) reducing the need for extensive task- specific robotic data, and (2) enabling the transfer of knowledge from general- purpose models to robotic manipulation tasks. Existing works [2 4 10] [2] have demonstrated that using pre-trained ussoo-language understanding models as the VLA model backbone and finetuning them on robot action datasets can significantly reduce data requirements and achieve remarkable task performance. However, despite these advances, true generalization-ppin -artecularrttyy to unseen tasks, objects, or environments rerimas limited and is yet to be fully realized.
24 25 26 have demonstrated remarkable generalization capabilities when conditioned on novel textual or visual inputs. The generated videos exhibit excep- 26, which can be attributed the models' extensive knowledge learned from massive real- wolld videos. Notably, the situation of video generators han- dling novel text and novel image conditions shows a natural alignment with that of robot manipulators dealing with unseen instructions and unseen observations. The understanding of physical dynamics learned by video generators is also fundamental capability required for any high-performing robotic manipulator to reason about the physical consequences of their generated actions. Furthermore, video generators can predict future world states by following given instructions, which inherently reflects a planning capability that is also crucial for robotic manipulation models to anticipate and organize their interactions with the physical environment. Motivated by these observations, we aim to explore the following question: "Can large video generators be seamlessly adapted into generalizable robotic manipulators?"
Recently, large video generation models 23
tional outstanding physical plausibility 25
To answer this question, this work studies constructing and fine-tuning VLA models based on large video generation models. To bridge the gap between video generation and robotic manipulation, the key lies in enabling video generation models to produce instruction-follwiii executed by robots. Furthermore, to effectively transfer the strong generalization abilities of video generation models in the video domain to the action domain, it is essential to ensure a consistency between the actual exccution of generated actions and the visual imagination represented in the generated videos (see Figure for as illustration). This alignment allows the semantic and physical coherence learned in video generation to be naturally extended to embodied robotic behaviors.
actions that can be
We propose a simple yet effective approach called Video VLA, which transforms a Video Diffusion Transformer (24] to VitionAction Diffusion Transformer by adding actions as new output modality and jointly denoising video and action. The Diffusion Transformer (DiT) [27] architecture has demonstrated remarkable superiority in both video generation 24 28 26 and action generation [4 10 7 tasks, and in this work we unify them into a single, multi-modal DiT framework. In this multi-modal DIT, the video and language are encoded using specific tokenizers such as a causal standard practices in generative modeling. We do not apply tokenization to the action modality and simply use the 7-D vector representation encoding robot translation, rotation, and gripper state. The model operates by taking the language tokens and the latent of the current visual observation as conditions: it then jointly predicts the future actions and generates the corresponding future visual contents that would result from executing these ions in the current environment, supervised by a DDPM Diffusion loss [30].
video VAE 24 and a T5 (29) text
Our experiments reveal a strong correlation between the predicted actions and the generated video clips- when the imagined future (i.e., the generated videos) closely aligns with the actual outcome of the environment, the corresponding predicted actions consistently yield a higher task success rate. This observation suggests that the quality of visual imagination serves implicit indicator of

---

action reliability. In other words, when VideoVLA produces coherent and plausible future visual predictions, it is more likely that the associated actions are accurate for task completion. This finding underscores the importance of jointly predicting future visual dynamics alongside future actions.
One of the key advantages of VideoVLA lies in its generalization capability. Beyond performing in- domain tasks and manipulating objects seen during training, it demonstrates promising generalization in two challenging settings: (1) imitating skills from other embodiments, and (2) handling novel objects that do not appear the training set. This capability can be attributed to two factors: (1) the use of pre-trained video generation models, which enable the system to interpret language instructions and generate plausible imagined futures, and (2) Video dual- preddction strategy, which fosters a strong correlation between predicted actions and their corresponding visual consequences, VideoVLA explores a paradigm shift by pioneering the use of pre-trained video generation models for robot manipulation, in contrast to the dominant reliance on pre-trained understanding models in prior VLA works. The feasibility and potential of such new paradigm are clearly demonstrated by this work. As generative models continue to improve, we believe that robotic manipulation systems derived from video generation models, like VideoVLA, will exhibit increasingly robust generalization, moving us closer to the goal of artificial general intelligence.
2 Related Works
Models. Recent advances in lision-lngguage-ace (VLA) models [I [5 6 7 have enabled new paradigm for language-gus -angeed robot manipulation these models interpret language instructions, perceive the current environment, and control the robot to complete tasks in an end-to-end manner. This success is largely driven by the advancement of vision [6 17 18) language [19] 20 21, and vasion-language [13] [4] 15] foundation models. By leveraging these pre-trained understanding-orientee models, VLA systems have achieved notable improvements in task success rates and demonstrated a certain degree of generalization to novel 31 and unfamiliar environments [4] 6). Another key driver in the advancement of VLA models is the increasing availability of large-scale, diverse datasets. The RT series [32] 33
Aision-Lagguag--
2
4 10
objects
demonstrates that VLA models can effectively leverage large-scale datasets and deliver promising results. To support continued progress, the Open X-Embodinent (2) dataset was released, aggregating over 60 robot datasets- cnttaset--iclluing including Language Table [34.. Berkeley Bridge [5), and
unified benchmark for pre-training. VideoVLA builds upon the rapid progress in robot datasets and pre-trained foundation models but departs from prior VLA approaches that rely primarily on understanding-ories -oriented models. Instead, VLA leverages large-scale pre-trained video generation models and introduces a dual-prediction strategy-jointly predicting future actions and corresponding visual imaginations that depict the anticipated outcomes of those actions. Recent works, UVA (36] and VPP (37] represents initial steps toward this paradigm. Compared to these efforts, VLA differs in four key aspects beyond architectural distinctions: (1) we fully exploit the capabilities of large-scale pre-trained video generators: (2) our model demonstrates strong generalization to novel objects and unseen skills; (3) we reveal a strong correlation between predicted actions and visual imaginations: and (4) VidcoVLA achieves competitive performance against recent ftate-of-the-a VLA models such as zo [0] and CogACT (4) in both simulated and real-world settings.
Video Generation Models. Video generation 38
aims to synthesize realistic and temporally coherent video sequences. This field has experienced rapid progress. largely fueled by advances in image generation [39] and the development of diffusion models [27] 40 Modern video generation frameworks typically support multi-modal conditioning. For example, text-to-video
23
24
25
models [1
42 38
generate videos from natural language prompts, while image-to-videe and pose-
to-video models animate static inputs using learned motion priors
4
44
Most state-of-the-ar
video generation methods adopt two-stage pipeline: first, video VAE
is trained to encode each video into a sequence of latent representations: then, a diffusion transformer such as DiT [27]
24
25
or MM-DIT [39 is trained with a diffusion loss to model the spatiotemporal distribution over these video latents. Our model is built upon VideoX 24 one - of the most powerful video generation models to date. To our knowledge, this is the first work that adapts large-scale video generator to the domain of robotic manipulation, demonstrating that such models can serve not only generative purposes but also as effective tools for visual planning and action prediction.
Video Generation for Robot Manipulation. Using video generation models for robot manipulation has been an active area of research in recent years. However, most existing approaches incorporate video generation as visual planning component within modular frameworks to assist with action prediction. For instance, [45] 46, 47 extract actions from video predictions using inverse dynamics
3

---

models. [88] estimates end-effector actions by computing optical flow between frames and leveraging extract a goal frame from the predicted video and use it as the condition for action prediction. [5 and 52 focus on human hand-object interaction video generation and design models for
depth maps. Approaches like 49 50
transfer. [53] gathers future frame information by querying features from video generation model and uses diffusion head to generate action. [54] and [6] generate video frames autoregressively and predict one frame at time along with an action. Different from these works, our method introduces unified, end-to-end VLA model directly adapted from a large pre-trained video generation model, which whichjointly predicts the video and action modalities within single large transformer architecture, and it leverages the generalization capabilities learned from video generation pretraining on massive real-world videos.
3 Methodology
3.1 Problem Formulation
Given a text instruction T describing a task and the current visual observation O the objective is to jointly predict:
1. An action chunk A= = {a; E R is 7-D vector. The first three dimensions encode the wrist rotation, the next three dimensions encode the wrist translation, and the final dimension indicates the gripper state binary value, where 0 denotes a closed gripper and I denotes an open gripper.
Ji=1 consisting of K actions to be executed, where each action a;
2. A video clip F = }F composed of N frames that depict the anticipated future visual content resulting from executing A. In our implementation, we do not predict the raw visual frames directly; instead, we predict their latent representations, as detailed in Section 3.3
After executing the predicted action chunk A, a new observation is obtained, and the process is tepeated-that is, the model predicts the next action chunk based the updated ubservation-until the task is completed. Note that the frequencies of A and F may differ-each action a; in A can correspond to multiple frames in F.
3.2 Overview
Figure presents an overview of VideoVLA, which primarily consists of: (1) two encoders that transform each input modality-tth language instruction and the video clip-into tokens or latent representations; and (2) a DiT backbone that, conditioned on the language tokens and the latent repre- sentation of the current visual observation, jointly predicts future frame latents and the corresponding action sequence necessary to accomplish the task.
3.3 Data Preprocessing
VideoVLA processes language and visual inputs in the latent space to improve computational efficiency. To achieve this, employs text encoder and a video encoder, as shown in Figure 2 a).
Text Encoder. We adopt T5 29 as our text encoder to convert each language instruction T into a fixed-length token sequence of 226 tokens. The resulting encoded tokens are denoted as T.
Video Encoder. The video encoder aims to transform a video clip F {F,) { consisting of N frames, into a sequence of video latents V = {V; E R ソリ= latents after temporal downsampling, and (h, w) represents the spatial resolution of each latent V,,
where is the number of video
24! our video encoder. Owing to its We adopt the 3D-causal VAE encoder from CogVideoX
causal design, the first video latent V, in the generated sequence V = {V } encodes only the first frame F1. which corresponds to the current visual observation O. In other words, V1 serves as the latent representation of the current observation. During inference, we encode the current visual observation O alone obtain VI, 1, whereas during training, the entire video clip F is input to the encoder to obtain both the current observation latent V, and the future frame latents (V, (V,)z=>
3.4 Unified Future Modeling
Given the current observation latent V1 and the language instruction tokens T. our goal is to train VLA to jointly predict an action chunk A {a; E ? and the future frame latents (V, ; 2= As illustrated in Figure 2b), we employ a DiT-style 27 architecture, in which both the conditioning inputs and the target variables are concatenated into a unified sequence. The network composed of stack of sclf-attention blocks that model the interactions across modalities and time.

---

Robot Traectory
Denoised Future-
Denoised
Decoder
Frame Latents
Actions
Instruction:
"Place can into middle drawer."
Optional
2
Diffusion Transformer
Text Encoder
Video VAB Encoder
K
Language Tokens
First-Prume Latent
Noisy Future-
Nodsy 7-D Actions
Frume Latents
Language Tokens
Frame Latents
Actions
Conditions
Noisy Inputs
(a) Training Data Preprocessing
(b) Architecture RoboSeer
Figure 2: Overview of VLdeoVLA. (a) The text encoder converts the language instruction into fixed-length token sequence, while the video encoder transforms a video clip into a sequence of frame latents, where the first latent corresponds to the first frame (i.e., the current visual observation). (b) VLAeoVLA adopts a Diffusion Transformer (27) architecture that conditions on the encoded language tokens and the first frame latent to jointly predict the next action chunk required accomplish the task, along with the future frame latents that represent the anticipated visual outcomes of executing that action chunk. The video decoder, highlighted in pink, optional and only used when visualizing the imagined future frames.
Specifically, for each visual teitent-whetee it represents the current observation or future frames- flatten the spatial dimensions into a one-dimensional sequence using raster order. Let V'1 denote the flattened version of the current observation latent V1, and { {V's à ززز=2 denote the flattened future frame latents { {V, } Jj=2- We then construct multimodal input sequence by concatenating T, V'1, 22 and A. To model {V') ر and A, we adopt DDPM (30 Following its noise scheduling strategy, Gaussian noise is added both {V' j و and A. Prior inputting into the backbone, all modalities are projected into a common embedding dimension. The model is trained to denoise the noisy versions of (V', 1)== and A using DDPM's 'ss diffusion loss. Following DiT (27), the noise timestep embedding is incorporated via adaptive LayerNorm. The backbone is initialized using the pre-trained Cog VideoX 24 model.
Experiment
Dataset. We the Open X-Embodiment (OXE) (2) dataset for pre-training, which contains over 1 million real-world robotic trajectories collected from 60 datasets spanning 22 distinct robot embodiments. Following prior works such as Octo M1. Open 3], and CogACT [4], adopt similar subset of OXE, comprising 22.5 million frames for pre-training. For real-world experiments, collect dataset consisting of 5824 samples spanning three robotic manipulation tasks: "pick", "stack", and "place". The data is collected via teleoperation using Realman robot equipped with 7-DoF arm and gripper.
Evaluation. We conduct two types of ianattionii-ddmaaae and generalization across both simulation and real-world experiments. In-domain evaluation assesses scenarios where the skills (e.g., "put" and "stack") and objects (e.g., "green block") have been encountered by specific embodiment during pre-training or fine-tuning. Generalization evaluation, on the other hand, focuses on two key capabilities:
(I) executing previously learned skills on novel objects, and (2) transferring skills learned by other embodiments- unseen by the target embodiment-ine the target embodiment.
For simulation experiments, we train our model solely the OXE dataset, which includes data from the Google robot and WidowX robot, and evaluate on these two embodiments using the SIMPLER environment [55]. This simulation platform is designed closely mirror real-world conditions, effectively bridging the sim-to-real gap for both control and visual inputs (55]. For real-world experiments, further fine-tune the pre-trained model using our collected dataset.
Each task evaluated over multiple trials both simulation and real-world settings. Detailed setups provided in the appendix. We report the average success rate across all trials for each experiment.
5

---

Table I: In-domain evaluation of VideoVLA and prior VLA models using the WidowX robot and Google robot within the SIMPLER [55] simulation environment. All models are trained on the OXE 2 dataset. For To [00, retain only the visual observations as input, consistent with all other models in our comparison. The robot proprioception inputs are omitted because they are unavailable for most embodiments in the OXE dataset. For the WidowX robot, the four evaluated tasks are: "put spoon on towel", "put carrot on plate", "stack green block on yellow block", "put eggplant in yellow basket". We highlight the best results in bold and the second- best results with underline.
WidowX Robot (VM)
Google Robot (VM/VA)
Method
Put Sp.
Put
Stack
Put Ep.
Pick Up Coke Can
Move Near
Open/Close Drawer
Open and Place
Avg. (AII)
Ca.
Block
Avg.
Avg.
RF-1-X 2
0.0
4.2
0.0
0.0
1.1
56.7 /49.0
/
59.7 / 29.4
22.2/11J
42.7 /0.5
24.8
KT-2-X 2
78.7
82.3
77.9/79.2
25.0 35.3
3.7/
/
Octo-Base
[]
15.8
12.5
0.0
41.7
17.5
2..7/1.1 4.2/3.1 17.0/0.6
0.0/0.0
11.0/112
9.9
Octo-Small
[
41.7
8.2
0.0
56.7
26.7
-/-
-/-
-/-
-/-
-/-
OpenVLA B
4.2
0.0
0.0
12.5
4.2
18.0/608
56.3
63.0/2888
0.0/0.5
34.3/39.4
26.0
SpatialVL.A
56
20.8
20.8
25.0
70.8
34.4
81.0/8955
69.6/7177
59.3/36.2
8.37 12.2
52,4
47.1
No 10
29.2
62.5
29.2
91.6
53.1
84.0/7144
55.8/57.9
34.3/26.5
31.5
53.5/
50.0
CogACT
4
71.7
50.8
15.0
67.5
51.3
91.3 89.6
85.0/800
52.7 71.8/28.3
147.1
7
62.6
Ours
75.0
20.8
45.8
70.8
53.1
92.3/89k4
82.9 /7.3
66.2/
28.8
50.9/59.1
73.1
/2.88
62.8
63.0
Table 2: Evaluation of generalization to novel objects using the Google robot under the SIMPLER environment. The novel objects are selected from the YCB [57] and GSO (58] datasets.
Green Cube
Egg-- plant
Straw- herry
Tennis Ball
Cleaner Bottle
Toy Airplane
Flash- light
Method
Carrot
Wrench
Plum
Avg.
VLe
3
12.0
24.0
12.0
0.0
0.0
4.0
0.0
4.0
4.0
4.0
6.4
SpatialVL.A
56
76.0
52.0
84.0
32.0
60.0
56.0
32.0
56.0
32.0
28.0
50.8
To
10
48.0
60.0
44.0
16.0
12.0
44.0
24.0
16.0
0.0
24.0
28.8
CogACT
4
84.0
72.0
64.0
16.0
32.0
44.0
20.0
32.0
20.0
40.0
42.4
Ours
96.0
84.0
88.0
40.0
72.0
80.0
68.0
44.0
28.0
52.0
65.2
Implementation Details. We utilize CogVideoX-5B 24 our pre-trained backbone. By default, each inference step predicts 13 future frame coreets--crrrssooddiin to 49 video frames-and 6 action steps. The model is trained for 100K iterations during pre-training and 15k iterations during fine- tuning, using 32 AMD MI300X GPUs with batch size of 256. We employ the Adam W optimizer with learning rate of 1e-5 and a weight decay of lc-4. During inference, we use DDIM sampling with 50 denoising steps. For simulation experiments, we predict 13 future video latents corresponding to frames, whereas for real-world experiments, we predict future latents corresponding to 13 frames, for efficiency. In both settings, 6 future actions are predicted, but only the first 3 actions are executed during deployment.
4.1 Simulation Experiments
We use the SIMPLER (55) environment for simulation- evaluation.
In-Domain Evaluation. SIMPLER offers two evaluation protocols- Visual Matching (VM) and Variant Aggregation (V assess the performance of models using the Google robot and WidowX robot. Specifically, VM aims to closely replicate real-world tasks by minimizing discrepancies between the simulated and real environments. VA builds on VM by introducing variations such as changes in background, lighting, and table texture. For the Google robot, both evaluation settings are available in SIMPLER, whereas for the WidowX robot, only the VM setting is provided. Table [ summarizes the performance of the Google and WidowX robots under the SIMPLER environment. Our model, VLA,oVLA, achieves the highest average performance on the VM-WidowXX-robel (averaged over 4 tasks), the highest average performance on the VA- VA-Google-robe (averaged over 4 tasks), the second- -ighhest average performance on the TM-Google-robo (averaged over 4 tasks), and the highest overall average performance across all 12 tasks. These results highlight Video strong capabilities in in-domain tasks.
Generalization to Novel Objects. Table 2 presents the evaluation of generalization capabilities to novel objects. Specifically, we select objects from the YCB [57] and GSO (58) datasets that do not

---

Table 3: Evaluation of generalization skills using the Google robot within the SIMPLER environment. The new skills are transferred from the WidowX robot, as they are present in its training data but absent from the Google robot's training set. (L.R,U,B) denotes (Left, Right, Upper, Bottom).
Put Spoon on Towel
Put Carrot on Plate
Stack Green Block on Yellow Block
Take Out of Apple
Flip Cup
Pour Coke Can
Slide to (L.R.U.)
Method
Avg.
Open
8
0.0
12.5
0.0
26.7
0.0
4.0
0.0
6.2
SpatialVLA
56
6.3
31.3
0,0
31.3
20.0
40.0
4.0
18.9
no [0
18.8
18.8
0.0
66.7
8.0
12.0
4.0
18.3
CogACT
20.8
41.7
5.0
43.8
4.0
20.0
8.0
20.4
Ours
56.3
58.3
20.0
93.8
20.0
52.0
40.0
48.6
Table 4: Real-world in-domain evaluation using the Realman robot. All models are pre-trained on the OXE dataset and subsequently fine-tuned on our collected dataset. For the "Place" task, the embodiment is required to first perform a "pick up" action followed by "placing" the object at the specified location; therefore, we report the success rates for both stages separately.
Pick Up
Stack
Place
Task (All)
Method
Banana
Lemon
Avocado
Avg.
Cup
Bowl
Avg.
Pick Up
Place
Avg.
Avg.
VLe
3
12.5
0.0
12.5
8.3
12.5
0.0
6.3
29.2
0.0
14.6
9.7
SpatiaIVLA 56
37.5
25.0
50.0
37.5
25.0
16.7
20.8
20.8
0.0
10.4
22.9
To
10
62.5
62.5
75.0
66.7
58.3
50.0
54.2
45.8
16.7
31.3
50.7
CogACT
75.0
62.5
87.5
75.0
83.3
45.8
64.6
54.2
16.7
35.5
58.4
Ours
62.5
75.0
75.0
70.8
75.0
58.3
66.7
87.5
25.0
56.3
64.6
appear in the Google robot's training data and import them into the SIMPLER environment. We evaluate the "Pick Up" skill using the Google robot on 10 novel objects: green cube, carrot, eggplant, wrench, strawberry, plum, tennis ball, cleaner bottle, toy airplane, and flashlight. Our model achieves the highest average success rate and outperforms prior models on eight out of the ten novel objects.
Generalization to New Skills. Table presents the evaluation of skill generalization using the Google robot in the SIMPLER environment. The model is trained on the OXE dataset, which includes of skills.
diverse set of embodiments, each potentially associated with distinct, non-overlapping Notably, the training data for the Google and WidowX robots constitute a significant portion of the dataset, with the WidowX robot exhibiting a broader skill repertoire. To assess skill generalization, we evaluate the model's s ability to transfer skills from the WidowX robot to the Google robot. Specifically, all eight skills listed in Table 3 are present in the WidowX training data but absent from the Google robot's is training set. Our model outperforms the second-best model, CogACT (4), by 28.2 points in average success rate and achieves superior performance across all evaluated skills.
4.2 RorldWorld Experiments
For all experiments in this section, fine-tune each pre- mreitrinee Open B. Spa- tialVLA (56), CogACT [4], xy [10), and our Video our collected dataset using the Realman robot, which is equipped with 7-DoF and gripper.
In-Domain Evaluation. For real-world in-domain evaluation, we assess performance on three tasks: (1) Pick up the [Object] and place it onto the [Color] plate, where Object E [Banana, Lemon, Lemon, Avocado). and Color E [White, Blue, Ycllow]: (2) Stack the [Color] [Object] into the [Color] [Object], where Object E {Cup, Bowl} and Color (Pink, White, Blue, Yellow); J: (3) Place the [Color] block onto the [Color] block, where Color e (Red, Orange, Blue, Green, Yellow). To increase task difficulty, we introduce novel distractor objects into the scene. As summarized in Table 4 VideoVLA achieves the highest average success rate among all evaluated models, demonstrating strong in-domain performance in real-world settings beyond simulation.
Generalization to Novel Objects. Table 5 presents our evaluation of real-world generalization to novel objects. Using the Realman robot, we perform the task: "Pick up the [Novel Object] and place it onto the [Color] plate", where each Novel Object is chosen from set of 12 objects
7

---

Table 5: Evaluation of real-world generalization novel objects using the Realman robot. The task is: "Pick up the [Novel Object] and place onto the [Color] plate". B-1: an upright black bottle.
B-2: a horizontally placed black bottle. B-3: an upright yellow bottle.
Blue Ball
Clear Tape
Toy
Era-
Screw-
Man-
Cab-
Mou-
Pea- ch
Method
Pen
B-1
B-2
B-3
Avg.
Duck
ser
driver
go
le
sc
OpenVLA 3
50.0
0.0
8.3
25.0
0.0
0.0
0.0
0.0
8.3
8.3
0.0
8.3
16.7
9.6
SpatialVLA 56
58.3
0.0
8.3
33.3
0.0
0.0
0.0
0.0
16.7
8.3
8.3
25.0
25.0
14.1
mo
10
66.7
0.0
0.0
16.7
33.3
0.0
25.0
0.0
8.3
16.7
41.7
50.0
25.0
21.8
CogACT
91.7
16.7
58.3
50.0
8.3
16.7
0.0
58.3
33.3
26.9
Ours
83.3
75.0
75.0
58.3
58.3
41.7
41.7
33.3
25.0
16.7
41.7
66.7
41.7
50.6
Table 6: Evaluating real-world cross-embodimen skill transfer: our Realman robot performs novel skills learned only by the WidowX robot, using familiar objects.
Method
Move Block Move Fruit
Grab Fruit
Topple Bottle
Take Out Block
Wipe Table
Avg.
Open VLA 3
18.8
12.5
18.8
0.0
0.0
0.0
8.3
SpatialVLA
56
18.8
25.0
31.3
6.3
0.0
0.0
13.5
mo 10
43.8
31.3
62.5
12.5
12.5
8.3
28.5
CogACT
56.3
50.0
68.8
18.8
0.0
16.7
35.1
Ours
81.3
68.8
75.0
43.8
37.5
41.7
58.0
Table Ablation study on backbone choice. The eval- uation is conducted in the SIMPLER (Visual Match- ing) environment using the Google robot. * denotes models trained from scratch.
Table Ablation study on the number of predicted future frames. The evaluation is conducted the SIMPLER (Visual Match-
ing) environment using the Google robot.
Backbone
Coke Can
Pick Up
Move Open/Close Near
Drawer
Avg.
#Frames
Coke Can
Pick Up
Move Open/Close Drawer
Near
Avg.
OpenSora-1.1
23
67.7
57.1
25.9
50.2
13
88.7
75.4
61.6
75.2
CogVideoX 5B*
24
18.6
10.8
9.2
12.6
25
90.0
79.2
63.0
77.4
Cog CogVideoX-33 5B
24]
92.3
82.9
66.2
80.4
49
92.3
82.9
66.2
80.4
not seen during training, as listed in Table 5 and Color E (White, Blue, Yellow). Although all evaluated models exhibit some degree of real-world generalization to novel objects, their performance varies significantly. For instance, Ope [3] and SpatialVLA (56) achieve 0% success rate on nearly half of the novel objects. In contrast, our VideoVLA successfully handles all 12 novel objects, with success rates ranging from 16.7% % to 83.3%.
Generalization to New Skills. In this experiment, we train our model and all baseline models on combined dataset consisting the WidowX Robot (derived from the OXE dataset (2) and our own collected dataset. As result, our model is exposed to two distinct embodiments during training: the WidowX robot and our Realman robot. These embodiments possess partially non-overlapping skill sets. To evaluate skill transfer, we focus on skills that are observed by the WidowX robot during training never demonstrated by the Realman robot. We then assess whether the Realman robot can successfully perform these unseen skills. The set of non-overlapping skills includes: "Move", "Grab", "Topple", "Take Out", and "Wipe". The objects to be manipulated have been seen by the Realman robot during training; only the skills themselves are novel to the embodiment. As shown in Table Video achieves the highest success rates across all evaluated skills. Notably, even for skills such as "Topple" and "Wipe", which differ substantially from those encountered during training. VideoVLA is still able to complete them to a certain extent.
4.3 Ablation Study
Backbone. Table Table presents the results of two comparisons: (1) VideoVLA equipped with different pre-trained backbones, including Cog Video [24] and OpenSora-1.1 23]; and (2) VideoV VLA initialized with the pre-trained Cog VideoXX5B versus VideoVLA trained from scratch. The results highlight strong correlation between the quality of generated videos and task manipulation success rate: (1) using higher-quality video generator, such as Cog VideoX--D -BB, significantly improves task

---

Table 9: Ablation study on the dual-prediction strategy. In-domain evaluation conducted in the SIMPLER (Visual Matching) environment using the Google robot. Generalization tests follow the settings of Tables and
In-domain
Generalization
Method
Pick up Coke Can
Move Near
Open/Closse Drawer
Avg.
Novel Objects
New Skills
Default
92.3
82.9
66.2
80.4
65.2
48.6
No video loss
35.6
29.1
16.2
27.0
12.7
4.4
Action only
33.3
25.8
17.6
25.5
11.3
2.1
Table 10: Success rates for visual imagination and actual execution under generalization settings.
Visual imagination outputs are evaluated by human judges to determine success or failure.
Metrie
Novel Objects
New Skills
Visual Imagination Success Rate
84.0
63.4
Actual Execution Success Rate
65.2
48.6
performance; and (2) training from scratch without pre-training on large-scale general vidco data results in a dramatically lower success rate.
Time Horizon. Table 8 presents ablation study how the number of predicted future frames affects robot manipulation performance. We evaluate three settings: predicting 49, 25, and 13 future frames, which correspond to 13, and 4 video latents, respectively, given a video VAE encoder with a downsampling rate of 4. As the number of predicted future frames increases, the overall performance improves consistently. This suggests that having longer temporal horizon allows the model to better anticipate the consequences of its actions, leading to more accurate execution.
Dual-Prediction Strategy. Table presents an ablation study on our indeo-actiondual- pred-ccin strategy. Three variants are compared against each other: (i) Default, which jointly predicts future videos and actions with denoising losses on both modalities; (ii) No video loss, which retains joint modeling but applies the denoising loss only actions; and (iii) Action only, which predicts future actions without predicting future videos, using an action denoising loss. Both No video loss and Action only show a substantial performance decline across all tasks, underscoring the importance of the dual prediction for achieving robust task execution and strong generalization.
4.4 Craginnation-xxecttie Correlation Analysis
We further analyze the correlation between the predicted visual imaginations and the actual execution outcomes. For each task, record the video frames where the predicted actions executed, resulting in a sequence of M execution frames denoted as {F,) J Correspondingly, we generate M related imagination frames {F M by feeding the predicted video latents into the VAE decoder, as illustrated in Figure
Motion Similarity. To assess the motion similarity between these two videos, we proceed follows: For each video, we extract keypoints using SIFT [59] on the first frame, and then apply SAM [60] to segment the foreground (i.e., objects and the robot embodiment), retaining only keypoints within the foreground regions. Next, we use the SAM-PT [61] point tracking method to track these keypoints across frames. As result, we obtain two sets of point trajectories: {p,}f=1 from the actual
execution and {p)),"" from the visual imagination, where each P, and p'J represents trajectory of a foreground keypoint. We then apply the Hungarian matching algorithm to identify correspondences between trajectories in the two sets. For each matched pair, we compute the normalized cosine similarity between the trajectory vectors. Finally, we report the mean of all pairwise similarities as the overall metric for robot motion similarity between the predicted visual imaginations and the actual executions. The results in Figure 3 show that a higher robot motion similarity between the visual imaginations and the actual executions corresponds a higher task success rate. This indicates that (1) there is a strong correlation between the predicted visual imaginations and actual executions, and (2) predicting accurate future visual outcomes facilitates successful robotic manipulation.
Task Performance Comparison. We also compare the quality of visual imagination and actual exccution results in the generalization settings in Table I0] Since visual imaginations are video

---

Successful
Successfil
Execution
Exceution
Failed
Failed
Execution
Execution
Motion SimilaritCllaginationn, Exccution)
Motion Similaritytlnngnnation Exccution)
Google robot.
(h) WidowX robot.
Figure 3: Each sub-figure illustrates the relationship between robot motion similarity cumparity-oomparing visual imaginations with actual executions- and task success. Each point represents either a success- ful or failed execution. higher robot motion similarity corresponds an increased likelihood of successful execution. The plots show aggregated statistics across tasks in the SIMPLER environment using (a) the Google robot and (b) the Widow robot.
"Put the lemon the white plate."
"Put the white cup into the blue cup."
Visual Imaginations
Visual Imaginations
Actual Executions
Actual Executions
Figure 4: Visualization of VideoVLA's predicted visual imaginations and corresponding real-world executions during task completion, demonstrating a strong correlation between imagined and actual outcomes. Additional visualizations provided in the appendix.
outputs which cannot be automatically evaluated for success, we conduct human evaluation and deem an imagined trajectory successful if it (i) follows the instruction semantically and (ii) exhibits no salient geometric distortions or violations of physical plausibility. We observe that most visual imaginations yield reasonable outcomes, attaining 84.0% success on novel objects and 63.4% on new skills. As expected, actual exccution achieves lower success (65.2% and 48. 48.6%, respectively), reflecting the added difficulty of precise physical grounding, actuation noise, and perception errors in real environments. Additionally, we provide two qualitative visualizations in Figure
Conclusion
In this paper, we present VideoVLA, jision-lagguagc-atton framework that leverages large pre- trained video cenerators-unlinge prior approaches that rely primarily on large perception or vision- language models. VideoVLA jointly predicts future actions be executed and generates visual imaginations depicting the anticipated outcomes of those actions. Experimental results reveal strong correlation between imagined futures and actual execution: high-quality visual predictions are consistently associated with reliable action plans and task success. VideoVLA not only demonstrates strong performance on tasks involving seen objects and previously learned skills, but also exhibits remarkable generalization to novel objects and embodimnt skill transfer in both simulated and real-world environments. These capabilities stem from the power of large-scale pre-trained video generation and our dual-pme nual-prediction strategy, Our findings highlight the potential of generative video models as a scalable foundation for general-purpose -urpose robot manipulation.
10

---

Acknowledgments
We thank the reviewers for their valuable comments. This work was partly supported by National Natural Science Foundation of China under grant No. 62088102.
References
[1] Octo Model Team, Dihya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 12213, 2024.
[2] Abby O' O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lec, Acorn Pooley, Agrim Gupta, Ajay Mandlckar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint ar31::231008884, 2023.
[3] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open spen-source vision- sision-langage-acce model. arXiv preprint 240.:2240000924640 2024.
[4] Qixiu Li. Yaobo Liang. Zeyu Wang. Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng. Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision- cation-lnggagg-actiin model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19655 2024.
[5] Lirui Wang, Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling propriocepive--isute learning with heterogeneous pre-trained transformers. Advances Neural Information Processing Systems, 37:124420- 124450, 2024.
[6] Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, al. Gr-2: A generative lantuaggggggo model with web-scale knowledge for robot manipulation. arXiv preprint 2rX.v.2410.06158, 2024.
[7] Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-1b: a diffusion foundation model for bimanual manipulation. arXiv preprint arXiv:2410.0788 2024.
[8] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint 1.XXv:20011083660 2020.
[9] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks. Johannes Welbl, Aidan Clark, et al. Training cmmputeopptina large language models. arXiv preprint arxiv:2203. 15556, 2022.
[10] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. TO: A vision- aasiionlanguage-attiin flow model for general robot control, 2024. URL https://rxxiv. org/abs/2410.24164.
[11] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-2: Zero-shot task generalization with robotic imitation learning. In Conference Robot Learning, pages 991-1002. PMLR, 2022.
[12] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on robot learning, pages 894-906. PMLR, 2022.
[13] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916 2023.
[14] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvi: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 24185 24198, 2024.
[15] Peng Wang Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. et al. Qwen2-vl: Enhancing vision language model's perception of the world at any resolution, arxiv preprint 2r.iv:2499.12191 2024.

---

[16] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[17] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision. pages 11975- 1 11986, 2023.
[18] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719 2023.
[19] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ami Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[20] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08290 2024.
[21] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[22] Physical Intelligence, Kevin Black. Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, al. mo.st a lanage-agguage-tctioon model with open-world generalization, 2025.
[23] Zangwei Zheng, Xiangyu Peng. Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20404, 2024.
[24] Zhuoyi Yang, Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong. Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion madels with an expert transformer. arXiv preprint arXiv:2408.0607) 06072, 2024.
[25] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint 2412:2211.03603 2024.
[26] Team Wan, Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen Wei Xie, Di Chen, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.2033 2025.
[27] William Pechles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFF/CVF international conference on computer vision, pages 4195-4205, 2023.
[28] Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, et al. Step-vidco-E2v technical report: The practice, challenges, and future of vidco foundation model, 2025.
[29] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 11(14)):1667, 2020.
[30] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 1186840-8511,, 2020.
[31] Yifan Zhong, Xuchuan Huang, Ruochong Li, Ceyao Zhang, Yitao Liang. Yaodong Yang. and Yuanpei Chen. Dexgraspvla: A vision- vision-language language-action framework towards general dexterous grasping. arXiv preprint ar02: 20900, 2025.
[32] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishhan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint ar 162607221..06817. 2022.
[33] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding. Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision lawooo-llnnaggaattiaeo action models transfer web knowledge robotic control. arXiv preprint arXiv:2307.15811 2023.
12

---

[34] Corey Lynch. Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking robots in real time. IEEE Robotics and Automation Letters, 2023.
[35] Homer Walke, Kevin Black, Abraham Lee. Moo Jin Kim, Max Du Chongyi Zheng, Tony Zhao. Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A datasct for robot learning at scale. In Conference Robot Learning (CoRL), 2023.
[36] Shuang Ii, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. arXiv preprint mxXiv22533000200 2025.
[37] Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang. Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412. 14803, 2024.
[38] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint ar e/Xiv:2311.15127 2023.
[39] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[40] Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words; vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22669-22679 2023.
[41] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-vidco: Text-to-video generation without tixt-video data. arXiv preprint arXiv:2209. 14792, 2022.
[42] Ruben Villegas, Mohammad Babacizadeh, Picter-Jan Kindermans, Hernan Moraldo, Han Zhang, Moham- mad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint 2tl.:2220000999 2022.
[43] Duygu Ceylan, Chun Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image difffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23206-23217, 2023.
[44] Qiaosong Qi. Le Zhuo, Aixi Zhang, Yue Liao, Fei Fang, Si Liu, and Shuicheng Yan. Diffdance: Cascaded human motion diftusion model for dance generation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1374-1382, 2023.
[45] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Advances in Neural Information Processing Systems (NeurlPS), 2023.
[46] Sherry Yang. Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In International Conference on Learning Representations (ICLR), 2024.
[47] Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Din Yeung. and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. In Proceedings of the 41st International Conference Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, 2024.
[48] Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In International Conference on Learning Representations (ICLR), 2024. Spotlight.
[49] Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abheel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, and Jonathan Tompson. Vidco language planning. In International Conference Learning Representations (ICLR). 2024.
[50] Hongyin Zhang, Pengxiang Ding, Shangke Lyu, Ying Peng, and Donglin Wang. Gevrm: Goal-expressive video generation model for robust visual manipulation. 2025.
[51] Homanga Bharadhwaj. Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia. Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. In Proceedings of The 9th Conference on Robot Learning (CoRL), volume 305 of Proceedings of Machine Learning Research, pages 333619951, Sep 2025.
13

---

[52] Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao. Weikai Chen. and Xiaoguang Han. Taste-rob: Advancing video generation of task-oriented hand-object interaction for generalizable robotic manipulation. In IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR), 2025.
[53] Youpeng Wen, Junfan Lin, Yi Zhu, Jianhua Han, Hang Xu, Shen Zhao, and Xiaodan Liang. Vidman: Exploiting implicit dynamics from video diffusion model for effective robot manipulation. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[54] Hongtao Wu, Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-trainn training for visual robot manipulation. In International Conference on Learning Representations (ICLR), 2024.
[55] Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sich, Scan Kirmani, et al. Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941 2024.
[56] Delin Qu, Haoming Song. Qizhi Chen, Yuanqi Yao, Xinyi Ye. Yan Ding, Zhigang Wang, Yuan Gu, Bin Zhao, Dong Wang. et al. Spatialvla: Exploring spatial representations for lisual-angguage-aee model. arXiv preprint arXiv:2501. 15830, 2025.
[57] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The yeb object and model set: Towards common benchmarks manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEFE, 2015.
[58] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553-2560. IEEE, 2022.
[59] David G Lowe. Object recognition from local scale-invariant fcatures. In Proceedings of the seventh IEEE international conference computer vision, volume 2, pages 1150-1157. Ieee, 1999.
[60] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the JEEF/CVF international conference on computer vision, pages 4015-4026, 2023.
[61] Frano Rajië, Lei Kc, Yu- funn Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu. Segment anything meets point tracking. In 2025 JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 9302-9311. IEEE, 2025.
[62] Kevin Frans, Danijar Hafner, Sergey Levine, and Picter Abbeel. One step diffusion via shortcut models. arXiv preprint m:Xi::2410.125:5 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Section for more details.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our approach in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical assumptions or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details are provided in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We provide the code and plan to release it as open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We evaluate approach in both simulation and real-world settings to ensure reliable and faithful assessment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research presented in this paper fully conforms to the NeurlPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This discussion is included in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the relevant papers accordingly.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We perform real-world experiments using our own collected dataset, with detailed information provided in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Generative LLMs are not used in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: for what should or should not be described.
21

---

Appendix
A
Evaluation Details
We report the number of trials for each experiment in both simulation and real-world evaluations. For each model, including our VideoVLA and the baselines, we strictly follow consistent evaluation protocol.
Simulation Experiments. Table [] lists the number of trials for each experiment conducted in the SIMPLER environment [55], including Google robot experiments SIMPLER Visual Matching (VM) and Variant Aggregation (VA), WidowX robot experiments in SIMPLER Visual Matching (VM), and generalization experiments with the Google robot.
Real- World Experiments. Table 12 presents the number of trials conducted for each real-world
experiment.
Table 11: The number of trials for each simulation experiment.
Setting
Task
# Trials
Pick Up Coke Can
300
Move Near
240
Google Robot (SIMPLER-VM)
Open/Close Drawer
216
Open Top Drawer and Place Apple
108
Pick Up Coke Can
825
Move Near
600
Google Robot (SIMPLER VA)
Open/Close Drawer
378
Open Top Drawer and Place Apple
189
Put Spoon on Towel
24
Put Carrot on Plate
24
WidowX Robot (SIMPLER- VM)
Stack Green Block on Yellow Block
Put Eggplant in Yellow Baske
24
Google Robot (SIMPLER)
Each Novel Object
25
Google Robot (SIMPLER)
Each New Skill
20
Table 12: The number of trials for each world experiment.
Task
# Trials
Pick Up
24
Stack
48
Place
24
Each Novel Object
12
Each New Skill
16
B
More Analysis
Causal masking vs. bidirectional attention. To further examine whether a causal masking strategy offers advantages over our default bidirectional approach, conduct ablation in which causal mask is applied: action tokens can attend to video tokens, but video tokens cannot attend to action tokens. As reported in Table 13, we observe consistent drop in success rate with causal masking relative to the default bidirectional model. These results indicate that allowing video tokens to access action tokens helps the model better capture how specific actions drive physical changes in the visual scene. Furthermore, bidirectional interaction between action and vision enhances coherence and siinment-aalowing actions to more effectively guide video prediction, while video prediction, in turn, supports more accurate action prediction.
22

---

Table 13: Ablation study on causal masking strategy. The evaluation is conducted in the SIMPLER (Visual Matching) environment using the Google robot.
Method
Pick up Coke Can
Move Near
Open/Close Drawer
Average
Default
92.3
82.9
80.4 61.1
66.2
Causal mask
89.3
76.2
75.5
Table 14: Ablation study on diffusion schedules. The evaluation is conducted in the SIMPLER (Visual Matching) environment using the Google robot.
Pick Up Coke Can
Move Near
Open/Close Drawer
Diffusion Schedule
Avg.
Sync-training, Sync-inference (Default)
92.3
82.9
66.2
80.4
Async-training, Sync-inference Async-training, Async-inference
87.3
74.1
60.2
73.8
84.7
70.8
57.4
71.0
Asynchronous noising and inference. By default, our model applies synchronous diffusion schedule to both vision and action tokens during training and inference, which means that these two modalities share same diffusion timestep and strategy. To further examine the impact of alternative schedules model behavior, we evaluate two variants that decouple the action and video noising schedules during training: (i) asynchronous training, synchronous inference, which uses different noising schedules in training but jointly denoises actions and video at test time; and (ii) asynchronous training, asynchronous inference. which at test time first denoises vidco latents (stage- 1) and then conditions the denoised video to generate actions (stage-2). As demonstrated in Table [4 both variants exhibit inferior performance compared to the default joint (synchronous) strategy across all tasks, with average scores of 73.8% and 71.0% versus 80.4% reflecting drops of 6.6 and 9.4 percentage points, respectively. We hypothesize that the degradation arises because actions and video temporally aligned modalities; joint training and denoising allow cross- moosl information to be exchanged at every step, yielding complementary supervision.
C More Visualizations
Figures 5nd show visualizations VideoVLA's predicted visual imaginations alongside the corre- sponding executions during task completion, for real-world and simulation experiments, respectively.
D Limitations and Broader Impacts
One key limitation of this work its inference speed. For real-world deployment, VideoVLA predicts 4 future latents (corresponding to 13 video frames) and 6 future actions (of which the first 3 are executed), using DDIM denoising with 10 denoising steps. This process takes approximately 1.1 seconds single H100 GPU, resulting in effective control frequency of around 3 Hz. This limitation primarily stems from the reliance on a large pre-trained video Senerator-oog VideoX-SB in our case. Nonetheless, generation models, rather than the commonly used pre-trained perception and understanding models, to enable end-to-end caation-anguage-acce action (VLA) robotic manipulation. Future directions for improving VideoVLA's inference speed include pre-training a robot-oriented video generator with a smaller model size than reneral-purpose generators, adopting one-step denoising techniques such as the ShortCut model [62), and applying model distillation. We leave inference acceleration as an important avenue for future exploration.
work demonstrates the feasibility of leveraging large-scale pre- pre-trained
23

---

Visual
Imaginations
Actual Executions
"Put the blue ball into pink bowl."
Visual
Imaginations
Actual Executions
"Pick up the white cup."
Figure 5: Visualizations VideoVLA's predicted visual imaginations and the corresponding execu- tions during task completion in real-world experiments.
24

---

Visual
Imaginations
Actual Executions
"Move orange near pepsi can."
Visual
Imaginations
Actual Executions
"Move 7up can near apple."
Visual Imaginations
Actual Executions
"Pick coke can."
Visual Imaginations
Actual Executions
"Put spoon the towel."
Figure 6: Visualizations of VideoVLA's predicted visual imaginations and the corresponding execu- tions during task completion in simulation experiments.
25


Reviews:
[{"review_id": "zfwE0PV8e1", "reviewer": "Reviewer_kjNa", "summary": "RoboSeer introduces a vision-language-action model that, unlike prior work relying on pre-trained understanding backbones, leverages large-scale video diffusion models to perform dual prediction: it jointly forecasts the next sequence of 7-D robot actions and “imagines” the corresponding future video frames depicting the result of executing those actions. Built atop a multi-modal Diffusion Transformer initialized from CogVideoX, RoboSeer is conditioned on a language instruction and the current visual latent, and trained to denoise both future action chunks and frame latents simultaneously. Through in-domain and generalization experiments—in both SIMPLER simulation (WidowX and Google robots) and real-world with a Realman arm—RoboSeer not only achieves state-of-the-art success rates but also reveals a strong quantitative correlation between the fidelity of its visual imaginations and actual task success, suggesting that high-quality video prediction serves as an implicit reliability measure for action plans.", "questions": "* Action Conditioning: Is there a causal mask between future video latents and actions—i.e., can actions steer video prediction?\n\n* Causal Decoupling: Have you considered ablating or “masking” the video-prediction loss (e.g., training without the visual imagination objective) to causally isolate its effect on action accuracy? A controlled ablation would clarify whether visual generation truly drives performance improvements.\n\n* Inference Efficiency: What is the end-to-end inference latency on the Realman robot when sampling future frames and actions? If the model requires tens or hundreds of milliseconds per diffusion step, this could limit real-world responsiveness—please report wall-clock timings and discuss potential speed-accuracy trade-offs.\n\n* Video-Action Correlation Analysis: You demonstrate a strong empirical correlation between imagined and actual trajectories. Can you provide insights into why this holds? For instance, is it driven by dataset coverage (e.g., similar scenes during training) or by model generalization?\n\n* Generalization Alignment: Does the generalization behavior of video prediction mirror that of actions? In other words, when encountering novel objects or skills, is one modality easier to generalize than the other? Please consider separate generalization metrics for each modality to better understand their interplay.", "limitations": "Please discuss the latency of video diffusion-based inference on real robots.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Quality: The paper presents a rigorously evaluated approach, with clear experimental protocols covering in-domain tasks and two types of generalization (novel objects and cross-embodiment skills) across both simulation and real robots. \n\nClarity: The architecture and training objectives are described in a straightforward manner, with helpful figures illustrating both the model design and correlation analysis. However, some implementation details—such as the real-time inference latency on physical hardware—are omitted, which could affect reproducibility in real-world settings.\n\nSignificance: By shifting from perception-only to generation-augmented planning, RoboSeer leverages advances in video diffusion to unlock stronger generalization. The demonstrated video-action correlation insight is particularly impactful, suggesting a built-in self-evaluation mechanism.\n\nOriginality: Adapting a large pre-trained video generator for joint action and visual outcome prediction in robotics is relatively new, setting RoboSeer apart from concurrent VLA models that mostly focus on understanding. The dual-modality denoising approach represents an interesting fusion of diffusion modeling of actions and future forecasting.\n\nWeaknesses: The reliance on heavy diffusion sampling (50 denoising steps) raises questions about run-time efficiency in real applications. The paper does not explore whether simpler or faster generative backbones could achieve comparable performance. Additionally, while correlation is shown, the causal relationship between imagination quality and action accuracy remains unverified (e.g., through causal masking or ablation).", "was_revised": true, "final_justification": "I will increase my score to accept.", "rebuttal": "Dear Reviewer kjNa,\n\nWe appreciate your thoughtful feedback to our work. Please find our responses to your questions below.\n\n\n**Q1: Action Conditioning: Is there a causal mask between future video latents and actions—i.e., can actions steer video prediction?**\n\nThank you for the insightful question. In our current implementation, video latent tokens and action tokens are fully bidirectionally visible to each other; we do not apply a causal mask between them. This design choice is based on the fact that video and action sequences are temporally synchronized but belong to different modalities. Allowing mutual visibility enables each modality to inform and guide the prediction of the other. This setup is analogous to joint video-audio generation, where synchronized but distinct modalities benefit from cross-modal attention to improve overall generation quality.\n\n\nTo further examine whether a causal masking strategy offers advantages over our default bidirectional approach, we conducted an ablation study in which a causal mask was applied: action tokens could attend to video tokens, but video tokens could not attend to action tokens. This experiment was conducted in the SIMPLER (VM) environment, following the same setup as Table 1 in the main paper. The results are shown below:\n\n| Method | Pick up Coke Can | Move Near |Open/Close Drawer | Average |\n|:------|:--------:|:-------:|:-------:|:-------:|\n| Default |   92.3      |     82.9  |     66.2  | 80.4 |\n| Causal mask|   89.3    |     76.2  |    61.1 | 75.5 |\n\nWe observed a consistent drop in performance under the causal masking strategy. This indicates that allowing video tokens to access action tokens enables the model to better capture how specific actions drive physical changes in the visual scene. In other words, the bidirectional interaction between action and vision modalities enhances coherence and alignment—allowing actions to more effectively guide video prediction, while video prediction, in turn, supports more accurate action prediction.\n\nWe will include this analysis in the revised paper.\n\n\n**Q2: Causal Decoupling: Have you considered masking the video-prediction loss (e.g., training without the visual imagination objective) to causally isolate its effect on action accuracy?**\n\nThe following table presents the performance of our variants with and without the video prediction loss, evaluated in the SIMPLER (Visual Matching) environment. When the video prediction loss is removed, the average performance drops significantly from 80.4 to 27.0, highlighting the effectiveness of our default strategy, which jointly predicts visual imagination and actions.\n\n| Video Loss | Pick up Coke Can | Move Near |Open/Close Drawer | Average |\n|:------|:--------:|:-------:|:-------:|:-------:|\n| Without |   35.6      |     29.1  |     16.2 | 27.0|\n| With  |   92.3      |     82.9  |     66.2  | 80.4 |\n\nWe will include this study in the revised version.\n\n\n**Q3: Inference Efficiency: What is the end-to-end inference latency on the Realman robot when sampling future frames and actions?**\n\nAs shown in Lines 17–18 of the supplementary material, during real-world deployment, we predict 13 frames using 10 denoising steps, resulting in approximately 1.1 seconds to generate 6 action tokens (with the first 3 used for execution), corresponding to an effective control frequency of ~3 Hz.\n\nTo further improve efficiency, we integrate video generation acceleration techniques such as X-DiT and TEA-Cache. With inference distributed across 4 Nvidia A100 GPUs, the latency is reduced to approximately 0.38 seconds for generating  6 action tokens (with the first 3 used for execution), achieving a control frequency of ~8 Hz.\n\nWe acknowledge the trade-off between performance and accuracy. Training a large, high-performance video generator from scratch would require thousands of GPUs; therefore, we initialize our model with publicly available pre-trained video generators, whose architecture cannot be easily modified. In future work, we plan to explore distillation techniques to compress large, general-purpose video generators into compact, robot-specific ones, thereby improving inference efficiency.\n\n\n\n**Q4: Video-Action Correlation Analysis: You demonstrate a strong empirical correlation between imagined and actual trajectories. Can you provide insights into why this holds? For instance, is it driven by dataset coverage (e.g., similar scenes during training) or by model generalization?**\n\nFirst, our training data consists of temporally aligned robot manipulation videos and their corresponding action sequences. Our model learns the cross-modal correspondence—particularly temporal alignment—between the visual and action modalities, enabling it to jointly predict both. Each modality provides complementary information to the other. \n\nSecond, we leverage large-scale pre-trained video generators with strong generalization capabilities in video generation—i.e., the ability to generate novel content during inference. Our model can be viewed as a form of post-training on top of these video generators, allowing it to inherit their generalization capacity. As a result, the model can imagine plausible visual executions for novel environments, skills, and objects. Since it has already learned the alignment between imagined visuals and corresponding actions, this generalization ability transfers to the action domain, leading to improved generalization in robot manipulation tasks.\n\n\n**Q5: Generalization Alignment: Does the generalization behavior of video prediction mirror that of actions? In other words, when encountering novel objects or skills, is one modality easier to generalize than the other? Please consider separate generalization metrics for each modality to better understand their interplay.**\n\nWe conduct an analysis in the SIMPLER environment to examine the generalization behavior of video prediction and action prediction on novel objects and unseen skills. The experimental setup follows the same configuration as Tables 2 and 3 in the main paper.\n\nThe table below reports the success rates for both actual executions and visual imaginations. For actual execution, we can directly determine whether each task succeeds or fails and compute the success rate accordingly. In contrast, visual imagination produces video outputs, which cannot be automatically evaluated for success. Therefore, we manually inspect each imagined video to assess whether the task was successfully executed (a record is viewed successful if the visuals well follows the given instruction and there are no noticeable geometric distortions of objects or violations of physical plausibility).\n\nWe observe that most visual imaginations result in reasonable execution outcomes—for example, achieving an 84.0\\% success rate on novel objects and a 63.4\\% success rate on new skills. In comparison, the actual execution success rates are lower. This discrepancy is expected because real-world execution requires precise grounding of the robot in the physical environment, which introduces additional challenges. We also observed that in these generalization tests, a higher visual imagination success rate corresponds to a higher action execution success rate (84.0\\%$\\leftrightarrow$65.2\\%, vs 63.4\\%$\\leftrightarrow$48.6\\%).\n\n\n\n| Metric | Novel Objects | New Skills |\n|:------|:--------:|:-------:|\n| Visual Imagination Success Rate (human judge) |   84.0      |    63.4  |\n| Actual Execution Success Rate  |   65.2      |     48.6  |\n\nWe will include this analysis in the revised version.\n\n\n\n*Once again, thank you for your valuable time and effort in reviewing our work.*\n\nAuthors of Paper 7746"}, {"review_id": "31ks6KQ9Zc", "reviewer": "Reviewer_4hpC", "summary": "This paper proposes RoboSeer, a novel vision-language-action (VLA) framework that leverages pre-trained video generation models—rather than understanding models—for generalizable robot manipulation. RoboSeer adopts a dual-prediction strategy, simultaneously forecasting future actions and their visual consequences using a multi-modal diffusion transformer. By conditioning on current visual observations and language commands, RoboSeer predicts the next action chunk and generates imagined future frames. The model is trained using a DDPM-style diffusion loss and initialized with a large-scale video generator (CogVideoX-5B). Experiments in both simulation and real-world environments demonstrate state-of-the-art performance on in-domain tasks, strong generalization to unseen objects, and cross-embodiment skill transfer.", "questions": "- Efficiency: Can RoboSeer skip visual imagination inference when only actions are needed (e.g., for low-latency execution)?\n- Failure Modes: What happens when visual imagination is incorrect? Is action prediction always degraded?\n- Domain Transfer: Have you tried generalizing to new environments (e.g., different lighting or backgrounds)?\n- Release Plans: Will the model, code, and pre-trained weights be made available?\n- Smaller Models: How does RoboSeer perform with smaller video generators (e.g., CogVideoX-B)?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- Paradigm Shift: Moves from \"understanding\" to \"generation\" in robot policy learning.\n- Dual-Prediction Strategy: Joint action and visual outcome prediction improve accuracy and interpretability.\n- Comprehensive Experiments: Evaluation across multiple embodiments, domains, and tasks, including real-world deployment.\n- Strong Generalization: Outperforms baselines like π0, CogACT, and OpenVLA on novel objects and unseen skills.\n- Correlation Analysis: Demonstrates that visual imagination quality correlates with actual task success.\n- Ablation Studies: Carefully explores ithe mpact of video generator quality and prediction horizon.\n# Weaknesses\n- Lacks Theory: No theoretical analysis explaining why dual-prediction improves generalization.\n- High Compute Cost: Requires 32×192GB GPUs and large-scale training. \n- Visual Imagination Dependency: Unclear how much visual prediction helps over action-only baselines.\n- Skill Transfer Scope: Skill generalization tested on a limited set of tasks; broader testing would strengthen results.", "was_revised": true, "final_justification": "This paper is novel in robotics manipulation, meets SOTA, and is solid in experiment. Therefore, I confirm my rating as 5.", "rebuttal": "Dear Reviewer 4hpC,\n\nWe sincerely appreciate your valuable feedback. Please find our responses to your questions below.\n\n\n**Q1: Lacks Theory: No theoretical analysis explaining why dual-prediction improves generalization.**\n\nThank you for raising this question. We now provide the insight pertaining to why dual prediction improves generalization.\n\nFirst, our training data consists of temporally aligned robot manipulation videos and their corresponding action sequences. Our model learns the cross-modal correspondence—particularly temporal alignment—between the visual and action modalities, enabling it to jointly predict both. Each modality provides complementary information to the other. \n\nSecond, we leverage large-scale pre-trained video generators with strong generalization capabilities in video generation—i.e., the ability to generate novel content during inference. Our model can be viewed as a form of post-training on top of these video generators, allowing it to inherit their generalization capacity. As a result, the model can imagine plausible visual executions for novel environments, skills, and objects. Since it has already learned the alignment between imagined visuals and corresponding actions, this generalization ability transfers to the action domain, leading to improved generalization in robot manipulation tasks.\n\n\nTo further demonstrate the effectiveness of the proposed dual-prediction strategy, we compare three variants of our model:\n\n1. *Default* – jointly predicts future videos and actions, with denoising losses applied to both modalities;\n\n2. *No video loss* – jointly models videos and actions but applies the denoising loss only to actions;\n\n3. *Action only* – predicts future actions only, without predicting future videos, and applies a denoising loss on actions.\n\nWe evaluate the performance of these models in the SIMPLER environment, focusing on generalization to novel objects and new skills. The experimental setup follows the same configuration as Tables 2 and 3 in the main paper. The results are presented in the table below:\n\n\n| Method | Novel Objects | New Skills |\n|:------|:--------:|:-------:|\n| *Default*  |   65.2      |     48.6  |\n| *No video loss* |   12.7      |    4.39  |\n| *Action only* |   11.3      |    2.19  |\n\nBoth ablated variants— *No video loss* and *Action only* —exhibit a significant drop in generalization performance, especially on tasks involving novel objects and new skills. These results highlight the importance of the full dual-prediction framework, which plays a critical role in enabling robust generalization.\n\nWe will include this analysis in the revised paper and thank you for the thoughtful suggestion.\n\n\n**Q2: High Compute Cost: Requires 32×192GB GPUs and large-scale training.**\n\nVLA models typically require large-scale training data and sizable network architectures to achieve robust robot manipulation, which in turn demands substantial GPU resources. For instance, both OpenVLA [1] and SpatialVLA [2] are trained using 64 NVIDIA A100 GPUs. In contrast, our model achieves superior performance across both real-world and simulation experiments (as shown in Tables 1–6 of the main paper) while using fewer GPU resources. We train our model on AMD MI300 (192G) GPUs as AMD GPUs are the compute resources currently available to us. Nevertheless, our model is compatible with NVIDIA GPUs, such as the A100, and can be trained on them if such resources are available.\n\n**Q3: Visual Imagination Dependency: Unclear how much visual prediction helps over action-only baselines.**\n\nPlease refer to our response to **Q1** for the comparison with baselines.\n\n\n**Q4: Skill Transfer Scope: Skill generalization tested on a limited set of tasks; broader testing would strengthen results.**\n\nWe appreciate the reviewer’s suggestion. Due to the time constraints of the review cycle, we were unable to conduct a broader evaluation. However, we plan to expand our experiments to a wider range of tasks in future work to more thoroughly assess the model’s skill transfer capabilities.\n\n\n**Q5: Efficiency: Can RoboSeer skip visual imagination inference when only actions are needed (e.g., for low-latency execution)?**\n\nWe thank the reviewer for the insightful question. Currently, our model cannot entirely skip visual imagination inference, as the vision latent tokens are designed to interact with the action tokens during inference, mirroring the joint prediction strategy used during training. However, we can skip the computation in the transformer's vision output layers and omit the video frame decoding process to reduce computation.\n\nAs shown in Lines 17–18 of the supplementary material, our current model achieves a control frequency of approximately 3 Hz during real-world deployment on a single GPU. More recently, by integrating acceleration techniques including X-DiT and TEA-Cache and distributing inference across 4 GPUs, we have reduced latency to approximately 0.38 seconds—enabling a control frequency close to 8 Hz.\n\n\n\n**Q6: Failure Modes: What happens when visual imagination is incorrect? Is action prediction always degraded?**\n\nOur observations indicate that failed visual imagination often leads to incorrect action prediction. We analyzed 60 randomly sampled failure cases in which the imagined videos either failed to reflect the instruction intent, exhibited physically implausible changes, or appeared visually incoherent. Among these, 55 episodes (91.7\\%) resulted in failed actual executions. This suggests that inaccurate visual imagination is a major contributor to incorrect action predictions. \n\n**Q7: Domain Transfer: Have you tried generalizing to new environments (e.g., different lighting or backgrounds)?**\n\nAs noted in Lines 205–206 of the main paper, the SIMPLER (VA) benchmark [3] introduces substantial domain variations, including changes in background, camera viewpoint, lighting conditions, and table textures—collectively referred to as \"environment variants\" [3]. These variations differ significantly from the original training distribution. As shown in Table 1 of the main paper, our model achieves the top performance under the SIMPLER (VA) setting, demonstrating its robustness and strong generalization ability to novel environments.\n\nFurthermore, as illustrated in the third row of Figure 2 in the supplementary material, the SIMPLER (VA) setting includes previously unseen tables and backgrounds not present in the original Google Robot dataset. Despite these domain shifts, our model maintains high-quality visual imagination and preserves consistency between imagined and actual executions. These results indicate that our model enables effective domain transfer across diverse and challenging visual environments.\n\n\n\n\n**Q8: Release Plans: Will the model, code, and pre-trained weights be made available?**\n\nYes, we will publicly release the model, code and pre-trained weights to the community.\n\n\n\n**Q9: Smaller Models: How does RoboSeer perform with smaller video generators (e.g., CogVideoX-B)?**\n\nWe appreciate the reviewer’s suggestion on evaluating smaller video generation models. In our work, we adopt CogVideoX-5B as the backbone because it supports image-to-video generation, which is essential for our setting where the model predicts future visual imaginations (i.e., videos) and actions from a single observed image. We did not evaluate a smaller variant, such as CogVideoX-2B, because it only supports text-to-video generation and lacks the necessary image-to-video capability.\n\nAdditionally, in Table 7 of the main paper, we reported the results using OpenSora-1.1 which has 1B parameters. The performance is significantly lower compared to our model based on CogVideoX-5B. But it is important to note that there are many differences between these two models, including architecture, training data, and other factors, and model size alone may not fully explain the performance gap. There's rapid progress in the field of video generation and we plan to explore more efficient and capable image-to-video foundation models to improve our method's inference speed while maintaining high manipulation performance.\n\n\n**Reference**\n\n[1] OpenVLA: An Open-Source Vision-Language-Action Model\n\n[2] SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model\n\n[3] Evaluating Real-World Robot Manipulation Policies in Simulation (SIMPLER)\n\n\n*Once again, thank you for your valuable time and effort in reviewing our work.*\n\nAuthors of Paper 7746"}, {"review_id": "gdD6XTm3i5", "reviewer": "Reviewer_Knht", "summary": "The authors propose a method of incorporating action diffusion to a pretrained Video Diffusion model (CogvideoX) to jointly train on action and video generation for robotics tasks. The authors demonstrate significant generalization compared to existing methods like OpenVLA and $\\pi_0$ in both simulation and real-world experiments.", "questions": "see weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths \n\n1. The authors propose a new method that can leverage pretrained video diffusion models like CogVideoX to additionally generate actions to solve robotics tasks.\n2. The authors demonstrate huge improvements in generalization to new objects and skills in both simulation and real-world tasks.\n\n\nWeaknesses :\n\n1. It is not clear how much of the generalisation stems only from the pre-trained knowledge of the CogVideo model and how much from the joint modelling proposed by the authors. Two important ablations here are training from scratch without the cog weights and also a variant that only predicts (and is trained on) action instead of both video and action, using the pre-trained cogvideo weights.\n\n2. A very important generalization that should also be evaluated is generalisation to unseen environments, since this is a very hard challenge in robotics. It would be interesting to see how the current method performs in this setup. \n\n3. There needs to be ablations using a different noising schedule for the actions and video latents. One advantage of such a mechanism is that this would allow conditional prediction of both video based on action and action based on video by setting the corresponding denoising step to the min noise for action's or video's schedule while denoising only video or action respectively.\n\n4. One very interesting effect of being able to predict both video and actions, like in this case, allows us to visualize failure cases to see what the model \"expects to happen in the scene\" using such actions. Some analysis of failure cases inspecting the generated videos should be added to see if there are patterns to be noticed/insights to be gained from them about the cause of the failure.\n\n4. In L196-197the authors mention they predict 16 latents corresponding to 49 frames. This is not possible with a downsampling factor of 4, but 13 (=(49-1)/4 + 1) latents as mentioned elsewhere in the paper. If this is not a typing mistake, then the authors need to explain this in detail.", "was_revised": true, "final_justification": "The authors demonstrate significant generalization compared to existing methods and have also ablated different variations of their architecture and formulation showing consistently good results.", "rebuttal": "Dear Reviewer Knht,\n\nWe sincerely appreciate your thoughtful feedback. Please find our responses to your questions below.\n\n\n**Q1: It is not clear how much of the generalisation stems only from the pre-trained knowledge of the CogVideo model and how much from the joint modelling proposed by the authors. Two important ablations here are training from scratch without the cog weights and also a variant that only predicts (and is trained on) action instead of both video and action, using the pre-trained cogvideo weights.**\n\n\n\n**Q1.1: Effect of using pre-trained CogVideo weights.**\n\nAs shown in Table 7 and Lines 268–269, training the model from scratch without using the pre-trained CogVideo weights leads to a substantial performance drop—from 80.4\\% to 12.6\\%. This clearly highlights the importance of the prior knowledge embedded in CogVideo, which serves as a strong foundation for generalization and is critical for achieving high downstream performance.\n\n**Q1.2: Effect of joint video-action modeling.**\n\nTo further demonstrate the effectiveness of the proposed dual-prediction strategy, we compare three variants of our method:\n\n1. *Default* – jointly predicts future videos and actions, with denoising losses applied to both modalities;\n\n2. *No video loss* – jointly models videos and actions but applies the denoising loss only to actions;\n\n3. *Action only* – predicts future actions only, without predicting future videos, and applies a denoising loss on actions.\n\nWe evaluate these three variants under two experimental settings:\n\n1. In-domain evaluation in the SIMPLER VM environment: This setting assesses in-domain performance. The experimental configuration follows that of Table 1 in the main paper. Results are shown in the table below:\n\n| Method | Pick up Coke Can | Move Near |Open/Close Drawer | Average |\n|:------|:--------:|:-------:|:-------:|:-------:|\n| *Default* |   92.3      |     82.9  |     66.2  | 80.4 |\n| *No video loss* |   35.6      |     29.1  |     16.2 | 27.0 |\n| *Action only* |   33.3      |     25.8  |     17.6 | 25.5\n\n2. Generalization evaluation: This setting examines performance on novel objects and new skills, using the same configuration as Tables 2 and 3 in the main paper. Results are presented below:\n\n| Method | Novel Objects | New Skills |\n|:------|:--------:|:-------:|\n| *Default* |   65.2      |     48.6  |\n| *No video loss* |   12.7      |    4.39  |\n| *Action only* |   11.3      |    2.19  |\n\n\nBoth ablated variants— *No video loss* and *Action only*—show a substantial performance decline across all tasks. These results underscore the importance of the full dual-prediction framework, which is essential for achieving strong generalization and robust task execution.\n\nWe will include these studies in the revised paper.\n\n**Q2: A very important generalization that should also be evaluated is generalisation to unseen environments, since this is a very hard challenge in robotics. It would be interesting to see how the current method performs in this setup.**\n\nAs noted in Lines 205–206 of the main paper, the SIMPLER (VA) benchmark [1] introduces substantial domain variations, including changes in background, camera viewpoint, lighting conditions, and table textures—collectively referred to as \"environment variants\" [1]. These variations differ significantly from the original training distribution. As shown in Table 1 of the main paper, our model achieves the best performance under the SIMPLER (VA) setting, demonstrating its robustness and strong generalization ability to novel environments.\n\nFurthermore, as illustrated in the third row of Figure 2 in the supplementary material, the SIMPLER (VA) setting includes previously unseen tables and backgrounds not present in the original Google Robot dataset. Despite these domain shifts, our model maintains high-quality visual imagination and preserves consistency between imagined and actual executions. These results indicate that RoboSeer is capable of effective domain transfer across diverse and challenging visual environments.\n\n**Q3: There needs to be ablations using a different noising schedule for the actions and video latents. One advantage of such a mechanism is that this would allow conditional prediction of both video based on action and action based on video by setting the corresponding denoising step to the min noise for action's or video's schedule while denoising only video or action respectively.**\n\nFollowing your suggestions, we tried two variants of our model with different noising schedules for action and video latent.\n\n1. *Asynchronous training, synchronous inference*: This variant adopts different noising schedules for actions and video latents during training. However, during inference, both are denoised synchronously, as in our default method.\n\n2. *Asynchronous training, asynchronous inference*: This variant also uses different noising schedules during training. At inference time, we first input the noisy actions and video latents but denoise only the video latents (stage-1). The resulting denoised video latents are then combined with the original noisy actions to denoise the actions (stage-2).\n\nWe evaluate three models under the SIMPLER VM environment. The experimental configuration follows that of Table 1 in the main paper. Results are shown in the table below:\n\n| Method | Pick up Coke Can | Move Near |Open/Close Drawer | Average |\n|:------|:--------:|:-------:|:-------:|:-------:|\n| *Default*  |   92.3      |     82.9  |     66.2  | 80.4 |\n| *Asynchronous training, synchronous inference*|   87.3    |     74.1  |    60.2 |  73.8 |\n| *Asynchronous training, asynchronous inference*|   84.7    |     70.8  |    57.4 | 71.0 |\n\nWe observe that asynchronous training leads to a performance drop across all tasks compared to our default method. This is likely because actions and video are temporally aligned, synchronous modalities. Joint training and denoising allow the two modalities to complement each other, similar to video-audio generation models where both modalities are typically modeled jointly to enhance performance.\n\nUnder the *asynchronous inference* setting, we observe that the denoised video latents from stage-1 are often of decent quality and semantically consistent with real-world expectations. When using these denoised video latents as conditions to generate actions in stage-2, the resulting actions still yield reasonably high performance (i.e., the third row in the table above). This indicates that the model trained with asynchronous noising schedules indeed possesses a decent degree of video-conditioned action generation capability.\n\nWe are unsure if this satisfactorily addresses your questions, particularly regarding the \"advantage\" of allowing conditional prediction of both video based on action and action based on video. Please let us know if it doesn't or you have any further questions.\n\n**Q4: Some analysis of failure cases inspecting the generated videos should be added to see if there are patterns to be noticed/insights to be gained from them about the cause of the failure.**\n\nWe thank the reviewer for this insightful suggestion. We conducted an analysis of failure cases in which the visual imaginations were either inaccurate or visually unpleasant, and found that such failures often lead to incorrect action predictions. Specifically, we randomly analyzed 60 failure cases where the generated videos failed to reflect the instruction intent, exhibited physically implausible transitions, or appeared visually incoherent. Among these, 55 episodes (91.7\\%) resulted in failed executions. This indicates that inaccurate visual imagination is a major contributor to incorrect action prediction. \n\nWe indeed observed interesting pattens from the visual imagination which can explain the execution failures. For example, when the gripper is not visible in the initial frame, the visual imagination may plan a motion trajectory not consistent with the robot's physical state, leading to execution failure. This issue can be handled by incorporating robot state into the model in our future work. In a few cases where the robot confuses the target objects to be grasped with other similarly colored ones, the visual imagination is found to have the same confusion in the generated videos.\n\nDue to rebuttal constraints, we are unable to provide examples here. However, in the revised version of the paper, we will include additional examples of both visual imagination and actual execution in failure cases.\n\n**Q5: In L196-197 the authors mention they predict 16 latents corresponding to 49 frames. This is not possible with a downsampling factor of 4, but 13 (=(49-1)/4 + 1) latents as mentioned elsewhere in the paper. If this is not a typing mistake, then the authors need to explain this in detail.**\n\nWe thank the reviewer for carefully pointing this out. This was indeed a typographical error—the model predicts 13 vision latents corresponding to 49 frames. We will correct this in the revised version of the paper. Other instances in the paper referencing 13 latents corresponding to 49 frames, such as Line 192, are correct.\n\n\n**Reference**\n\n[1] Evaluating Real-World Robot Manipulation Policies in Simulation (SIMPLER)\n\n*Once again, thank you for your valuable time and effort in reviewing our work.*\n\nAuthors of Paper 7746"}, {"review_id": "T4AWnON1Ir", "reviewer": "Reviewer_cgvg", "summary": "This paper proposes Roboseer which leverages diffusion models to output actions and future outcomes at the same time.", "questions": "see above", "limitations": "see above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "* There are a lot of experiments\n\nWeaknesses:\nI seriously doubt that the authors had zero knowledge of using video generation models for robot manipulation, for example, those works that use inverse dynamics. They did not mention those works in the related works section, and not compare to them in the experiments section as well. They even claimed:\n\n*this is the first work that adapts a large-scale video generator to the domain of robotic manipulation*\n\nSuch papers, not to say hundreds, but at least tens of papers are in this field (but really I think there are already hundreds of such papers). To name a few:\n\nLearning Interactive Real-World Simulators\nLearning to Act from Actionless Videos through Dense Correspondences\nLearning Universal Policies via Text-Guided Video Generation\nVideo Language Planning\nGEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation\nUnleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation\nTASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation\nGen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation\nUnleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation\nGR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation\nRoboDreamer: Learning Compositional World Models for Robot Imagination\nVidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation\n\nOf course the authors can say that there is at least some level of novelty, even not prominent novelty, which distinguishes themselves from these papers. But the authors did not mention any of these papers and discuss about the differences so we are unable to tell as well.", "was_revised": true, "final_justification": "I wanna emphasize again that it's not that I did not appreciate the merits of the paper, but that the paper over-claimed too much and missed potentially paragraphs of related works - this is a serious major problem of this paper. this also makes it hard to grasp the true contributions of this paper.\n\nHowever, the authors discussed these related works and their differences in the rebuttal phase. I assume that they will add all these to their final version and revise their paper so it does not over-claim; and therefore will raise the point to 4.", "rebuttal": "Dear Reviewer cgvg,\n\nWe appreciate your feedback to our work. Please find our responses to your questions below.\n\n**Q1: I seriously doubt that the authors had zero knowledge of using video generation models for robot manipulation, for example, those works that use inverse dynamics. They did not mention those works in the related works section, and not compare to them in the experiments section as well. They even claimed: this is the first work that adapts a large-scale video generator to the domain of robotic manipulation.**\n\nThank you for highlighting these previous works. In our paper, we propose a new large vision-language-action (VLA) model and primarily compare it with existing end-to-end VLA models such as RT-2, Pi0, CogACT, SpatialVLA, as shown in Table 1 of the main paper.\n\nBelow, we first provide a discussion of the papers you mentioned, which will also be incorporated into the revised version of our paper. We then outline the key differences between our work and these prior approaches, followed by the rationale for not including direct comparisons. Finally, we describe our revision plan to better integrate these related works.\n\n**Discussion of the Mentioned Papers.**\n\nSeveral works such as [1,2,3] train video generation models on robot video data and extract actions using an inverse dynamics model based on adjacent frame pairs. However, these inverse dynamics models are typically trained with limited data and simple architectures, making them less robust to changes in environment appearance (e.g., lighting, background, textures). Moreover, extracting actions from only two frames often fails to capture fine-grained or complex physical dynamics.\n\nThe work of [4] estimates end-effector actions by computing optical flow between frames and leveraging depth maps. However, since the gripper’s motion often produces weak optical flow signals, the authors resort to heuristic rules (e.g., large z-axis motion) to infer gripper closure, which can lead to inaccuracies in predicting precise gripper actions.\n\nOther approaches [5,6] extract a goal frame from the predicted video and use it as input to a goal-image-conditioned policy for action prediction, which introduces additional training and engineering overhead due to the need for a separately trained goal-image-conditioned policy.\n\nGen2Act [7] and TASTE-Rob [8] focus on human hand–object interaction video generation, where actions are produced via a human-to-robot transfer module based on synthesized human hand trajectories.\n\nGR-1 [9] and GR-2 [10] generate video frames autoregressively, predicting one frame at a time. This design limits their capacity for global planning and foresight. Moreover, they do not leverage pre-trained video foundation models, thereby missing the benefits of large-scale visual pretraining for generalization. \n\nVidMan [11] inserts implicit inverse dynamics adapters into a video diffusion transformer to integrate future frame information and then applies a separate diffusion-based action head for action generation. In their framework, the video and action modalities are trained in two stages without joint training.\n\n**Key Differences from the Mentioned Works and Rationale for Not Including Direct Comparisons.**\n\n***The key difference of our approach compared to prior works lies in the design of a unified, end-to-end vision-language-action (VLA) model directly adapted from a large pre-trained video generation model. Unlike modular approaches, our method jointly predicts the video and action modalities within a single large transformer architecture, and it leverages the generalization capabilities learned from video generation pretraining on massive real-world videos. No such model exists to our knowledge.***\n\nIn contrast, most of the mentioned works rely on additional modules to produce final actions. As mentioned above, [1–3] introduce separate inverse dynamics modules. [4] employs hand-crafted heuristic rules to generate action trajectories. [5–6] require an additional module to infer final actions from goal images generated in a separate stage. [7,8] first train models on human-object interaction data and then design task-specific algorithms to transfer the learned human-centric representations to robot action prediction. [11] gathers future frame information by querying features from video generation model and then uses a separate diffusion action head to generate action. The work of [9] and [10] do not generate video chunks as global planning and foresight; nor do they leverage pre-trained video foundation models.\n\nIn our method comparison, we focus exclusively on end-to-end VLA models and ensure fair comparison by reproducing open-sourced baselines such as OpenVLA, Pi0, CogACT, SpatialVLA. We consider these recent large-scale VLA models to be the primary competitors to our approach. Among the works mentioned by the reviewer, [9-11] are the more related as they fit better to the category of end-to-end VLA models. However, none of these works have released training code, making it infeasible to compare with these methods in our experimental setup.\n\n**Plans to Incorporate the Mentioned Papers in the Revised Version.**\n\nThank you again for pointing out these excellent works. We will incorporate a detailed discussion of the mentioned related studies in the revised version of the paper to provide a more comprehensive comparison and context. Specifically, we plan to add a paragraph in the **Related Work** section to discuss prior efforts that apply video prediction to enhance action policies. A preliminary draft is provided below and we will expand it to include additional relevant works:\n\n> **Video Generation for Robot Manipulation.** The use of video generation models for robot manipulation has been an active area of research. Most existing approaches incorporate video generation as a visual planning component within modular frameworks to assist with action prediction. For instance, [1–3] extract actions from video predictions using inverse dynamics models. [4] estimates end-effector actions by computing optical flow between frames and leveraging depth maps. Approaches like [5,6] extract a goal frame from the predicted video and use it as the condition for action prediction. [7] and [8] focus on human hand–object interaction video generation and design models for human-to-robot transfer. [11] gathers future frame information by querying features from video generation model and uses a diffusion head to generate action. [9] and [10] generate video frames autoregressively and predict one frame at a time along with an action. Different from these works, our method introduces a unified, end-to-end VLA model directly adapted from a large pre-trained video generation model. Unlike modular approaches, our method jointly predicts the video and action modalities within a single large transformer architecture, and it leverages the generalization capabilities learned from video generation pretraining on massive real-world videos. \n\nPlease let us know if this would be an acceptable solution.\n\n**Reference**\n\n[1]  Learning Universal Policies via Text-Guided Video Generation Video Language Planning\n\n[2]  LEARNING INTERACTIVE REAL-WORLD SIMULATORS\n\n[3]  RoboDreamer: Learning Compositional World Models for Robot Imagination\n\n[4]  Learning to Act from Actionless Videos through Dense Correspondences\n\n[5] VIDEO LANGUAGE PLANNING\n\n[6] GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation\n\n[7] Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation\n\n[8] TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation\n\n[9] Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation\n\n[10] GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation\n\n[11] VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation\n\n---\n\nOnce again, thank you for your valuable time and effort in reviewing our work. Please let us know if you have further questions.\n\nAuthors of Paper 7746"}]

Author rebuttal:

Meta-review: ## Summary
This paper introduces RoboSeer, a novel vision-language-action (VLA) model that leverages pre-trained video generation models to jointly predict future actions and their visual outcomes for robot manipulation. The approach represents a paradigm shift from "understanding" to "generation" in robot learning. All four reviewers lean towards acceptance, and after a thorough discussion period, their initial concerns were largely addressed by the authors' detailed rebuttal and additional experiments.

## Strengths identified by the reviewers
- Novelty and Paradigm Shift: The core idea of using a generative model to imagine future visual outcomes alongside predicting actions was seen as highly original and a significant departure from existing VLA models that focus primarily on perception (Reviewers kjNa, 4hpC).

- Strong Results: The paper presents comprehensive experiments across simulation and the real world, demonstrating state-of-the-art performance and impressive generalization to novel objects, unseen skills, and new environments (Reviewers 4hpC, Knht).

- Insightful Analysis: The demonstrated correlation between the quality of the "imagined" future video and the actual task success rate was highlighted as a particularly impactful finding, suggesting a built-in mechanism for self-evaluation (Reviewer kjNa).

## Weaknesses and Rebuttal

- Causality and Importance of Visual Generation: Reviewers kjNa and Knht questioned the actual contribution of the visual generation component to the model's performance. They requested ablations to isolate the effect of joint modeling and pre-trained weights.

  - Rebuttal: The authors provided extensive new ablation studies. They showed that removing the video prediction loss or training from scratch without pre-trained weights led to a massive drop in performance. Reviewers kjNa and Knht were satisfied with these results. 

- Missing Related Work: Reviewer cgvg raised a major concern about the paper's claim to be the "first" in this area, pointing out a significant body of existing work using video generation for manipulation that was not cited or compared against.

  - Rebuttal: The authors acknowledged this oversight. They provided a detailed discussion of the papers mentioned by the reviewer. They committed to adding this discussion to the related works section and toning down their claims of novelty. Reviewer cgvg was satisfied with the authors' commitment to revise the paper, stating they would raise their score assuming the changes are incorporated.

- Inference Efficiency and Latency: Reviewer kjNa raised concerns about the real-world applicability given the computational cost of diffusion models.

  - Rebuttal: The authors provided concrete latency figures for their real-world deployment (~3 Hz control frequency), along with details on how they improved this to ~8 Hz using acceleration techniques. They acknowledged the trade-off and discussed future work on model distillation. This was part of the detailed response that led Reviewer kjNa to raise their score.

- Generalization to New Environments: Reviewer Knht suggested evaluating generalization to unseen environments.

  - Rebuttal: The authors clarified that the SIMPLER (VA) benchmark used in their experiments already includes significant domain variations (lighting, backgrounds, textures), and their model achieved top performance in this setting, demonstrating this capability. Reviewer Knht was satisfied with the authors' response and maintained their positive rating.

## Recommendation Justification
This is a strong paper with a novel and impactful idea. The core contribution—leveraging large-scale video generators for joint action and outcome prediction—is a significant step forward for robot manipulation. The authors provide compelling empirical evidence of their method's effectiveness and strong generalization capabilities. The initial weaknesses, particularly the missing related work and the need for more rigorous ablations, were thoroughly addressed by the authors in their rebuttal. The reviewers were satisfied with the responses, and the consensus is a clear acceptance. The authors have committed to incorporating the additional related work, discussions and experiments into the final version, which will further strengthen the paper. I recommend accepting this paper.
Decision: Accept (poster)