Title: VideoVLA: Video Generators Can Be Generalizable Robot Manipulators
Venue: NeurIPS 2025
Accepted: yes
Keywords: Robot manipulation

video generation
Abstract: Generalization in robot manipulation is essential for deploying robots in open-world environments and advancing toward artificial general intelligence. While recent Vision-Language-Action (VLA) models leverage large pre-trained understanding models for perception and instruction following, their ability to generalize to novel tasks, objects, and settings remains limited. In this work, we present VideoVLA, a simple approach that explores the potential of transforming large video generation models into robotic VLA manipulators. Given a language instruction and an image, VideoVLA predicts an action sequence as well as the future visual outcomes.  Built on a multi-modal Diffusion Transformer, VideoVLA jointly models video, language, and action modalities, using pre-trained video generative models for joint visual and action forecasting. Our experiments show that high-quality imagined futures correlate with reliable action predictions and task success, highlighting the importance of visual imagination in manipulation. VideoVLA demonstrates strong generalization, including imitating other embodiments' skills and handling novel objects. This dual-prediction strategy—forecasting both actions and their visual consequences—explores a paradigm shift in robot learning and unlocks generalization capabilities in manipulation systems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Roboseer which leverages diffusion models to output actions and future outcomes at the same time.\n\nSTRENGTHS AND WEAKNESSES:\n* There are a lot of experiments\n\nWeaknesses:\nI seriously doubt that the authors had zero knowledge of using video generation models for robot manipulation, for example, those works that use inverse dynamics. They did not mention those works in the related works section, and not compare to them in the experiments section as well. They even claimed:\n\n*this is the first work that adapts a large-scale video generator to the domain of robotic manipulation*\n\nSuch papers, not to say hundreds, but at least tens of papers are in this field (but really I think there are already hundreds of such papers). To name a few:\n\nLearning Interactive Real-World Simulators\nLearning to Act from Actionless Videos through Dense Correspondences\nLearning Universal Policies via Text-Guided Video Generation\nVideo Language Planning\nGEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation\nUnleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation\nTASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation\nGen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation\nUnleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation\nGR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation\nRoboDreamer: Learning Compositional World Models for Robot Imagination\nVidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation\n\nOf course the authors can say that there is at least some level of novelty, even not prominent novelty, which distinguishes themselves from these papers. But the authors did not mention any of these papers and discuss about the differences so we are unable to tell as well.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method of incorporating action diffusion to a pretrained Video Diffusion model (CogvideoX) to jointly train on action and video generation for robotics tasks. The authors demonstrate significant generalization compared to existing methods like OpenVLA and $\\pi_0$ in both simulation and real-world experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n\n1. The authors propose a new method that can leverage pretrained video diffusion models like CogVideoX to additionally generate actions to solve robotics tasks.\n2. The authors demonstrate huge improvements in generalization to new objects and skills in both simulation and real-world tasks.\n\n\nWeaknesses :\n\n1. It is not clear how much of the generalisation stems only from the pre-trained knowledge of the CogVideo model and how much from the joint modelling proposed by the authors. Two important ablations here are training from scratch without the cog weights and also a variant that only predicts (and is trained on) action instead of both video and action, using the pre-trained cogvideo weights.\n\n2. A very important generalization that should also be evaluated is generalisation to unseen environments, since this is a very hard challenge in robotics. It would be interesting to see how the current method performs in this setup. \n\n3. There needs to be ablations using a different noising schedule for the actions and video latents. One advantage of such a mechanism is that this would allow conditional prediction of both video based on action and action based on video by setting the corresponding denoising step to the min noise for action's or video's schedule while denoising only video or action respectively.\n\n4. One very interesting effect of being able to predict both video and actions, like in this case, allows us to visualize failure cases to see what the model \"expects to happen in the scene\" using such actions. Some analysis of failure cases inspecting the generated videos should be added to see if there are patterns to be noticed/insights to be gained from them about the cause of the failure.\n\n4. In L196-197the authors mention they predict 16 latents corresponding to 49 frames. This is not possible with a downsampling factor of 4, but 13 (=(49-1)/4 + 1) latents as mentioned elsewhere in the paper. If this is not a typing mistake, then the authors need to explain this in detail.\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RoboSeer, a novel vision-language-action (VLA) framework that leverages pre-trained video generation models—rather than understanding models—for generalizable robot manipulation. RoboSeer adopts a dual-prediction strategy, simultaneously forecasting future actions and their visual consequences using a multi-modal diffusion transformer. By conditioning on current visual observations and language commands, RoboSeer predicts the next action chunk and generates imagined future frames. The model is trained using a DDPM-style diffusion loss and initialized with a large-scale video generator (CogVideoX-5B). Experiments in both simulation and real-world environments demonstrate state-of-the-art performance on in-domain tasks, strong generalization to unseen objects, and cross-embodiment skill transfer.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- Paradigm Shift: Moves from \"understanding\" to \"generation\" in robot policy learning.\n- Dual-Prediction Strategy: Joint action and visual outcome prediction improve accuracy and interpretability.\n- Comprehensive Experiments: Evaluation across multiple embodiments, domains, and tasks, including real-world deployment.\n- Strong Generalization: Outperforms baselines like π0, CogACT, and OpenVLA on novel objects and unseen skills.\n- Correlation Analysis: Demonstrates that visual imagination quality correlates with actual task success.\n- Ablation Studies: Carefully explores ithe mpact of video generator quality and prediction horizon.\n# Weaknesses\n- Lacks Theory: No theoretical analysis explaining why dual-prediction improves generalization.\n- High Compute Cost: Requires 32×192GB GPUs and large-scale training. \n- Visual Imagination Dependency: Unclear how much visual prediction helps over action-only baselines.\n- Skill Transfer Scope: Skill generalization tested on a limited set of tasks; broader testing would strengthen results.\n\nQUESTIONS:\n- Efficiency: Can RoboSeer skip visual imagination inference when only actions are needed (e.g., for low-latency execution)?\n- Failure Modes: What happens when visual imagination is incorrect? Is action prediction always degraded?\n- Domain Transfer: Have you tried generalizing to new environments (e.g., different lighting or backgrounds)?\n- Release Plans: Will the model, code, and pre-trained weights be made available?\n- Smaller Models: How does RoboSeer perform with smaller video generators (e.g., CogVideoX-B)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRoboSeer introduces a vision-language-action model that, unlike prior work relying on pre-trained understanding backbones, leverages large-scale video diffusion models to perform dual prediction: it jointly forecasts the next sequence of 7-D robot actions and “imagines” the corresponding future video frames depicting the result of executing those actions. Built atop a multi-modal Diffusion Transformer initialized from CogVideoX, RoboSeer is conditioned on a language instruction and the current visual latent, and trained to denoise both future action chunks and frame latents simultaneously. Through in-domain and generalization experiments—in both SIMPLER simulation (WidowX and Google robots) and real-world with a Realman arm—RoboSeer not only achieves state-of-the-art success rates but also reveals a strong quantitative correlation between the fidelity of its visual imaginations and actual task success, suggesting that high-quality video prediction serves as an implicit reliability measure for action plans.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The paper presents a rigorously evaluated approach, with clear experimental protocols covering in-domain tasks and two types of generalization (novel objects and cross-embodiment skills) across both simulation and real robots. \n\nClarity: The architecture and training objectives are described in a straightforward manner, with helpful figures illustrating both the model design and correlation analysis. However, some implementation details—such as the real-time inference latency on physical hardware—are omitted, which could affect reproducibility in real-world settings.\n\nSignificance: By shifting from perception-only to generation-augmented planning, RoboSeer leverages advances in video diffusion to unlock stronger generalization. The demonstrated video-action correlation insight is particularly impactful, suggesting a built-in self-evaluation mechanism.\n\nOriginality: Adapting a large pre-trained video generator for joint action and visual outcome prediction in robotics is relatively new, setting RoboSeer apart from concurrent VLA models that mostly focus on understanding. The dual-modality denoising approach represents an interesting fusion of diffusion modeling of actions and future forecasting.\n\nWeaknesses: The reliance on heavy diffusion sampling (50 denoising steps) raises questions about run-time efficiency in real applications. The paper does not explore whether simpler or faster generative backbones could achieve comparable performance. Additionally, while correlation is shown, the causal relationship between imagination quality and action accuracy remains unverified (e.g., through causal masking or ablation).\n\nQUESTIONS:\n* Action Conditioning: Is there a causal mask between future video latents and actions—i.e., can actions steer video prediction?\n\n* Causal Decoupling: Have you considered ablating or “masking” the video-prediction loss (e.g., training without the visual imagination objective) to causally isolate its effect on action accuracy? A controlled ablation would clarify whether visual generation truly drives performance improvements.\n\n* Inference Efficiency: What is the end-to-end inference latency on the Realman robot when sampling future frames and actions? If the model requires tens or hundreds of milliseconds per diffusion step, this could limit real-world responsiveness—please report wall-clock timings and discuss potential speed-accuracy trade-offs.\n\n* Video-Action Correlation Analysis: You demonstrate a strong empirical correlation between imagined and actual trajectories. Can you provide insights into why this holds? For instance, is it driven by dataset coverage (e.g., similar scenes during training) or by model generalization?\n\n* Generalization Alignment: Does the generalization behavior of video prediction mirror that of actions? In other words, when encountering novel objects or skills, is one modality easier to generalize than the other? Please consider separate generalization metrics for each modality to better understand their interplay.\n\nLIMITATIONS:\nPlease discuss the latency of video diffusion-based inference on real robots.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-04"}