Vid2World: Crafting Video Diffusion Models to Interactive World Models
Authors: Siqiao Huang
Jialong Wu
Qixing Zhou
Shangchen Miao
Mingsheng Long
Venue: ICLR 2026 Poster (2026)
Keywords: World Models
Video Diffusion Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Vid2World,  a general approach for leveraging and transferring pre-trained video diffusion models into interactive world models.

Abstract: World models, which predict future transitions from past observation and action sequences, have shown great promise for improving data efficiency in sequential decision-making. However, existing world models often require extensive domain-specific training and still produce low-fidelity, coarse predictions, limiting their usefulness in complex environments. In contrast, video diffusion models trained on large-scale internet data have demonstrated impressive capabilities in generating high-quality videos that capture diverse real-world dynamics. In this work, we present _Vid2World_, a general approach for leveraging and transferring pre-trained video diffusion models into interactive world models. To bridge the gap, Vid2World systematically explores _video diffusion causalization_, reshaping both the architecture and training objective of pre-trained models to enable autoregressive generation. Additionally, it incorporates a _causal action guidance_ mechanism to enhance action controllability in the resulting interactive world models. Extensive experiments across multiple domains, including robot manipulation, 3D game simulation, and open-world navigation, demonstrate that our method offers a scalable and effective pathway for repurposing highly capable video diffusion models into interactive world models.

Published as a conference paper at ICLR 2026
VID2WORLD: CRAFTING VIDEO DIFFUSION MODELS TO INTERACTIVE WORLD MODELS
Siqiao Huang's Jialong ;u'; Qixing Chouu",hanneeee Miao', Mingshhng Long" 1D4 Tsinghua University CChongging University huang- sq23@mails. t talssstsinnggn singhua. edu. en, wwjjallong2229gmaa com mingsheng@t s singhua. edu.
ABSTRACT
World models, which predict future transitions from past observation and action sequences, have shown great promise for improving data efficiency in sequen- tial decision-making. However, existing world models often require extensive domain-specific training and still produce low-fidelity, coarse predictions, limiting their usefulness in complex environments. In contrast, video diffusion models trained on large-scale internet data have demonstrated impressive capabilities in generating high-quality videos that capture diverse real-world dynamics. In this work, we present Vid2 World, general approach for leveraging and transferring pre-trained video diffusion models into interactive world models. To bridge the gap, Vid2World systematically explores video diffusion causalization, reshaping both the architecture and training objective of pre-trained models to enable au- toregressive generation. Additionally, it incorporates causal action guidance mechanism to enhance action controllability in the resulting interactive world mod- els. Extensive experiments across multiple domains, including robot manipulation, 3D game simulation, and open-world navigation, demonstrate that our method offers scalable and effective pathway for repurposing highly capable video dif- fusion models into interactive world models. Code and models are available at attps: /kni ght github. io /vwiwwwrdddd./.oo
INTRODUCTION
World models (Ha & Schmidhuber 2018 Dawid & LeCun 2023) have emerged as pivotal com- ponents for sequential decision-making enabling agents predict future states and plan actions by simulating environment dynamics. Despite their success in numerous domains, including game simulation (Hafner et al.) 2020) Alonso al.) 2024). autonomous driving (Wang et al., (002b)). and robotics (Yang et al. 2024 these models conventionally rely solely on in in-domain action- labeled data, necessitating meticulous and labor-intensive data collection, yet still often yielding relatively coarse predictions with limited physical realism, constraining their applicability in complex environments.
To mitigate this data-hungry nature, recent works (Bar et al., 2025 Wu et al. 2024) have drawn inspiration from the success of foundation models (Bommasani et al. 2021). exploring pre-training on broader, cross-domain action-labeled data. While this strategy improves data efficiency and generation quality to some extent, it does not solve the fundamental problem. The high cost of acquiring any form of action-labeled data persists, and the resulting models still struggle generate visuals with high fidelity and realism. This indicates that merely expanding the scope of action-labeled data is insufficient. A more fundamental paradigm shift thus imperative to truly unlock the full capabilities of world models.
We argue that the requisite paradigm shift is to leverage the largest yet overlooked data source: internet-scale action-free video data. Abundant, easy to collect, diverse with rich world priors, these data constitute the most prominent part of the data pyramid for world models (shown in Figure D. While prior work (Gao et al. (2025) has explored co-traie co-training with such data, we highlight more
"Equal contribution.

---

Published as a conference paper at ICLR 2026
Domainn Action Labeled
Robol
Manipulation
Vid2World
Game
Labelelabeedd Datk
Simulation
Internet- Seale
Internet-icale Attiom-Frve Data
Pretrained
Opes-Word
Diffissio Model
Navigation
Figure I: Vid2World repurposes video diffusion models for interactive world modeling. From the perspective of the data pyramid for world models, leverages vast pre-trained knowledge from internet-scale action-free video data to achieve high-fidelity, action-conditionee generation across diverse downstream domains with limited interaction data.
direct and cost-efficient path: transferring the physical priors and generative capabilities learned by video diffusion models (Ho et al., 2020 OpenAl 2024; DeepMind 2024) into interactive world model-level transfer not only avoids the
models. This transition from data-level exploitation prohibitive cost of training on massive video corpora but also extracts physical priors more smoothly. as non-causal generative modeling could be inherently easier than causal counterpart.
Despite profound potential, two significant challenges arise in bridging the gap between passive video diffusion models and interactive world models, as shown in Figure The first key challenge lies in enabling causal generation. Standard video diffusion models, designed for full-sequence denoising with bidirectional context, inherently introduce non-causal temporal dependencies. This makes them unsuitable for causal rollouts, where future predictions must strictly depend past information. The second challenge, equally critical, is enforcing action conditioning. While causalization enables autoregressive rollout, these models still lack the ability for counterfactual reasoning saasonnng--redictinng how different actions influence future states. This necessitates injecting fine-graine fine-grained, frame-level action signals into the generation process. Especially in diffusion models, despite that flasifier-free guidance (Ho & Salimans 2021) offers the freedom of balancing sample diversity and fidelity, extending it action guidance still requires careful algorithmic and architectural designs.
predicting
In this paper, we present Vid2World, a general approach to effectively transform internet-scale pre-trained video diffusson models into interactive world models capable of autoregressive, action- conditioned generation. To causalize video diffusion models, we systematically explore and discover better weight transfer schemes that adapt temporal attention and convolution layers into their causal counterparts, enabling fine-tuning under causal training objective (Chen et al.) (2024). For action conditioning, we inject action signals into model inputs corresponding frames and design an extended training objective that supports action guidance during diffusion sampling at each frame in principle. We evaluate Vid2World by transferring an extensively pre-trained, 1.4B 44B-parameter video diffusion model (Xing et al. 2024) to diverse domains, including robot manipulation (Brohan et al.) 2023). 3D game simulation (Pearce & Zhu 2022), and open-world world navigation (Shah et al.2(2022) Experimental results demonstrate significant improvements as staat--of-the-aa world models.
existing transfer approaches well
To summarize, contributions (1) To the best of our knowledge, are the first to systemati- cally explore the problem of transferring full-sequence, non-causal, passive video diffusion models into autoregressive, interactive, action-conditioned world models. (2) We propose Vid2World, general and effective approach for this problem, featuring novel techniques for the causalization and action conditioning of video diffusion models. (3) State-of-the-an performance Vid2World across domains establishes new benchmarks for this critical problem and facilitates future research.
2 RELATED WORKS
Diffusion for World Modeling. Due to the high fidelity offered by diffusion models in image and video generation, utilizing diffusion for world modeling has garnered growing interest. Prior works fall primarily into two categories. The first treats world modeling problem, where history observation and action sequences serve entirely as conditions. While these approaches follow autoregressive framework and have shown promise in domains such game simulation (Alonso et al., 2024; Decart et al. 2024) and navigation (Bar et al. 2025), they typically rely fixed-length context window, limiting their applicability in environments that demand
conditional image generation
2

---

Published as a conference paper at ICLR 2026
Causal Liansfomm
Action Condition
Video Diffusion Models
Causal Diffusion Models
Interactive World Models
Figure 2: Transforming video diffusion models into interactive world models involves two key challenges: (1) Causal generation: converting full-sequence diffusion models into causal diffusion models; (2) Action conditioning: adapting causal diffusion models into interactive world models.
long-term temporal reasoning. The second category formulates the problem as a full-sequence video generation task (Yang al., 2024] Yu et al. 2025 Zhou et al. 2022), often achieving better temporal coherence between frames. Yet, these models operate on full video segments, precluding autoregressive rollout, and thus hindering their
interactive environments.
Leveraging Foundation Models to World Models. Foundation models (Bommasani et al. 2021
trained on large-scale and diverse data, have shown revolutionary potential across modalities such as text (OpenAl et al. 2023 Guo et al.) (025), image (Rombach et al., [022] Betker et al. (023)), and video (OpenAl| 2024 Meed 2024). In the text domain, large language models are prompted to act world models for spatio-temporal reasoning in agentic tasks (Hao et al. 2023 Gkountouras
ctt al. 2025 Hu et al., 2025). In the video domain, adapting pre-trained generative models into world models typically involves architectural modifications. For instance, He et al. (2025) integrate
action-conditioning module into the generative backbone, while Rigter et al. (2024) introduce an
action-aware adapter to modulate the output of frozen video model. However, these approaches often overlook the critical need for interactivity and temporal causality, limiting their applicability in sequential decision-making and interactive environments.
PRELIMINARIES
World Models. A world model is internal model learned by agent model the dynamics of its environment. This environment is typically formalized as (discrete-time) Partially Observable Markov Decision Process (POMDP) (Kaelbling al. 1998). defined At each time step I, the agent receives an observation o, Φ(*)). where 8, E S is the underlying state that satisfies the Markov property. Upon taking an action a; A, the next state sampled from the transition distribution models, the agent learns to estimate this transition function through history observation and action sequence: Po Po(o++1 10g,, (.gz). While world models can be applied to wide range of observation modalities, including proprioceptive signals (Yin et al., (205)). text (Wang al. 2024a Wu al. 2025), 3D meshes (Zhang et al., 20024), and pixel-based inputs (Wu et al. 2024
tuple (S, O Φ, A, p, 7)).
S x A - A(S), i.. 8t+1 p(- St, az). In the context of world
Zhu et al. (025).
here we focus on learning in the pixel space, where observations are defined over O |
Diffusion Models. Diffusion models (Ho al. 2020 Song al. 2021a) highly expressive
generative models that learn to approximate a target data distribution g(x), where x E Rd by progressively denoising Gaussian noise. At its core, the model makes use of two Markov Chains: forward process and a backward process, to transport between the noise distribution x ~ (0(1 and the distribution of interest x qx).. The forward (noising) process is defined as:
(x( = Nx""; /1- 3k k 1- 1),
where K is pre-defined noise schedule. Starting from pure noise x N I), the learned reverse (denoising) process aims to recreate x ~ q(x) using the following factorization:
pa (x (xx) ~ * (x(" k), 7h 1).
In practice, it is common to reparameterize the objective in terms of noise prediction, i.e., learning to
predict e (VT (ik) -1x)-xx+ xx Vou J4, where og A 1 Bg and on A I Og. This simplifies to
3

---

Published as a conference paper at ICLR 2026
minimizing the mean square loss:
[()) =
,0(xx,))13)
where X - V x + vI a e and E ~~ N( Sampling is performed via iterative denoising
through Langevin dynamics: x² Vau (x* La-aee ex(x", k) + o.w).
vi-az
Video Diffusion Models. In video diffusion models (Ho al. 2022). the sample x is represented sequence of frames (xx") ISEST, where f denotes the frame index and k indicates the noise level at that frame. Conventional approaches (Blattmann et al.) 2023) apply one uniformly sampled noise level across all frames, treating each frame identically within the denoising process. To relax this constraint, Chen et al. (2024) propose to sample noise levels independently for each frame, i.c., kg 2(( Kj) during training. Intuitively. this formulation captures more diverse set of noise level combinations across frames during training, opening up new capabilities. At inference time, the model follows a denoising schedule E R.Mx where M is the number of denoising steps and each row Km E IH" specifies the per-frame noise levels at step M. By setting k = 0 for history frames, k. K for masked future frames, and progressively denoising the current frame ki E {K,...,)) the model is capable of autoregressive generation.
4 METHODS
While video diffusion models excel at generating high-fidelityy physically plausible sequences, their default formulation is fundamentally incompatible with interactive world modeling. Concretely, two key transformation barriers stand out:
1. Inability of causal generation: Typical video diffusion models generate frames using bidirec- tional temporal context, allowing future frames to influence the past;
2. Lack of action conditioning: These models are typically conditioned on coarse, video-level inputs (e.g.. text prompts) and lack mechanisms for fine ame-grained, frame-level action conditioning.
To overcome these transfer barriers, propose Vid2World with two key modifications, contributing to a systematic, multi-level framework for converting video diffusion models into interactive world models, shown in Figure 2 In Section [.1] present the strategy of video diffusion causalization, which converts non-causal architectures into temporally causal variants compatible with the post- training objective (Chen al., (202)), by exploring weight transfer mechanisms maximally preserve the representations learned during pre-training. In Section 4.2 we extend the training objective to enable causal action guidance during inference for step-wise, interactive rollouts.
4.1 VIDEO DIFFUSION CAUSALIZATION
To causalize video diffusion models, modifications required on both architectures and training objectives. From an architectural standpoint, while bidirectional temporal modules in standard video diffusion models, which allow information flow across all timesteps, effective for full-sequence generation, they are fundamentally incompatible with autoregressive world modeling, where the current observation must not depend on future observations actions. This necessitates architectural surgery to enforce temporal causality, specifically in the computation and parameters of temporal attention (HaCohen et al. 2024 or non-causal convolutions (Blattmann al. 2023 Guo al.) 2024).
Temporal Attention Layers. Non-causal temporal attention layers can be converted into their
causal counterparts by straightforwardly applying causal masks. Since attention operates through dot products between queries and keys. it is inherently adaptive to variable sequence lengths; therefore. restricting the receptive field to exclude future frames does not alter the underlying computation of inter-token relationships. Consequently, this does not mandate parametric modifications.
Temporal Convolution Layers. In contrast, causalizing temporal convolution layers is more challenging. These layers employ symmetric kernels that aggregate features from both past and future frames, and simple adaptations may lead to suboptimal utilization of pre-trained kernel weights. To achieve this we systematically investigate three different strategies, detailed below.
4

---

Published as a conference paper at ICLR 2026
Temporal Convolution
Shift Weight Transfer
Masked Weight Transfer
Extrapolative Weight Transfer
Arreel ierght
Causalization
Zere-Zeit
Stuft 121
0 00000
0000000
Illustration of weight transfer mechanisms for temporal convolution layers: (1) Shift: shifts all weights into the past. (2) Masked: retains only past weights. (3) Extrapolative: leverages local linear feature relationships more in principle(example shown with 1,p 2).
Figure 3:
A naive approach, which we term Shift Weight Transfer, directly reuses the full pre-trained kernel {w, - by shifting it m steps into the past, resulting in a new causal kernel {w]]p_-m: While this preserves all kernel weights, it introduces temporal misalignment: the kernel's 's i-th position aggregates information at timestep i m, giving no guarantees of producing similar representations.
An alternative strategy, Masked Weight Transfer, truncates the kernel by retaining only the weights corresponding to past and present timesteps { { while setting the rest to zero {wi}r=-- O. This resembles applying hard causal mask the kernel at initialization. Although causality is enforced, it discards potentially useful information encoded in the future-facing weights.
Finally, we propose more principled and robust mechanism. Extrapolative Weight Transfer, based on local linear extrapolation of features along the temporal dimension. Formally, we posit that the feature a future timestep Z+++ can linearly approximated over a window of p past timesteps:
Z y= T Zt-j Bk,
where " 33 are determined by a linear extrapolation from the past p features. Our core principle is maximally preserve the output representation of the original convolution, such that the new causal computation produces similar result to the original non-causal one:
Σ Wi
-,
w w/zts b'.
This is achieved by re-distributing the weights {w;)i>o that originally acted on future frames, back onto the past part of the kernel, according to the linear feature relationships:
w [2-m] w + 1[-+11550 Σn WW b' - b Σ W.B:.
These architectural adaptation strategies are illustrated in Figure 3 with didactic example of Extrapolative Weight Transfer for m = 1,p = A detailed mathematical derivation and analysis of the error bounds provided in Appendix
Training Objectives for Causal Generation. Architectural changes alone are insufficient to enable causal generation. In this setting, future frames predicted step by step, conditioned previously p
fully denoised frames. i.e., under noise levels (k+)=1 = (0, O,...... .,....,, 0, k), E (o..... K). Hence, the model must be trained handle these inference-time noise-level distributions. In conventional video diffusion models, the training procedure follows a homogeneous noise schedule, where all frames share the same noise level. This limited subset of noise-level combinations makes them naturally incompetent for noise levels at autoregressive inference. Therefore, it becomes vital to train the model with different noise levels across frames. Here, we adopt Diffission Forcing (Chen) et al. 2024). where we sample noise levels to be independent and uniform in different frames, Le.. ke U(0 K), Vt. This training scheme exposes the model to the full space of noise-level combinations in the history frames, thereby enabling flexible and robust causal rollouts.
4.2 CAUSAL ACTION GUIDANCE
Causal video diffusion models alone are not yet interactive world models, as they still fall short of
action-conditioned generation. Prior works (Alonso al.) 2024 Bar et al. 2025 primarily explore
Whiie throughout the paper, primarily use "causal" to denote temporal causality, in this section, mechanism implicitly involves interventional causality via action guidance. See Appendix A.5 details.
5

---

Published as a conference paper at ICLR 2026
Sampled
d
Nuise Prodiction
1ef +ell
00
00
Action Guidance
Diffusion Forcing
x
u10, Mr
Causalized Architecture
Causalized Architecture
Action Drupout
P. Paeivoy
Ø
R6-1
Ø
4-1
Re-1
(a) Training
(b) Auto-regressive Rollout
Figure 4: Training and sampling of Vid2World, initialized by architecture causalization, (a) During training, we add independently sampled noise levels each frame, as well randomly drop out each action with fixed probability. (b) For autoregressive rollout, we denoise the latest frame while setting history clean. Action guidance is added for the current action. See Appendix for details.
integrating action condition at the video level, where the entire action sequence is encoded to a single embedding, analogous text embeddings. However, this approach has two major drawbacks: (a) Lacking the ability to perform fine-grained, frame-level eation-conditioned predictions; (b) Incompatibility with interactive settings, where actions arrive sequentially an online fashion during inference. In this section, we introduce both conceptual and methodological advances that extend action conditioning into causal action guidance, enabling principled steering of the generative process toward action-aligned predictions in interactive environments.
Causal Action Injection. To address these limitations, we equip the model with frame-level conditions through architectural modifications. When predicting ot, the embedding of a:-1 is added the model's s latent representation temporal position This allows each frame be conditioned directly on its preceding action in a temporally aligned manner, opening up the potential for precise, fine-grained control interactive settings. Specifically, this is implemented by feeding the action inputs into the denoising network using a lightweight multi-layer perceptron (Haykin 1994).
Training and Sampling with Guidance. For more targeted control over the generated dynamics, we adopt classifier-free guidance (Ho & Salimans [021] in our autoregressive action-conditioned setting, realizing Causal Action Guidance. Classifier-fres free guidance trains model to jointly learn a conditional and an unconditional score function, allowing for amplified guidance at inference time by steering the output toward the conditional distribution. In our setup, the score function c [a,], [kt]]) takes in tuple of noised observations [x]]. actions [a,]. noise levels [k,] as input, and the conditioned variable is the most recent action. Therefore, the model should be capable of capturing both the conditional score function: Eoond co ([x)+))st, (ar)r<e, [br),se), well its unconditional
counterpart, where the most recent action is masked: Cucond co cx 1 [a,<t-1, Ø), [kr)r
To achieve this, we extend our training objective by incorporating an action dropout mechanism, where a, for each timestep / is independently dropped with fixed probability p:
Σ 1-0 lle < Isn [ārlst [ ā 0,
W.P. P, az otherwise.
(()) E,R,)..,,
At its core, this mechanism compels the model to learn score functions conditioned on all subsets of the action sequences, including the effect of the immediate action the predicted transition. This, in turn, enables classifier- free guidance for sampling via: Eguided (1 + A) Econd X Euromd where + the guidance scale. Intuitively, since the score function represents the gradient of the log- ogg-robbbbility. this linear composition in the score space corresponds to multiplicative steering mechanism in the probability space. We formalize this equivalence in the following theorem.
Theorem 4.1 (Causal Action Guidance Probability Steering). Let H4 ([x)lr<t, (arlr<t-1) denote the history context excluding the current action. Under standard score- score-based formulation, the proposed score composition is mathematically equivalent sampling from the following steered

---

Published as a conference paper at ICLR 2026
posterior distribution, where (1 X) is a fixed constant:
p(x; ar-1, He)
x(x: a Hi) &
pxx:
He)
p(xe He) p(aa-
Xt, He)a
p(x, He)
History-Comm Consistent Prior
Action Alignment
In other words, the guidance term acts as an implicit classifier, steering the generation towards regions aligned with the user's most recent action, while preserving the high-fidelity generation capabilities in sequential modeling. As result, through varying X, the model is equipped with the test-time flexibility of controlling responsiveness towards fine-grained action variations. Ultimately, this transformation better aligns the model with its objective of world modeling- merely to capture average behavioral trends, but to reason about an agent's 's immediate actions. We provide the detailed proof for Theorem 4.1 in Appendix A.4
Summary. Vid2World transfers full-sequence, passive video diffusion models into autoregressive, interactive world models. Through video diffusion causalization, we open up the model's capability to perform causal generation, and through causal action guidance, incorporate and strengthen action signals for interactive settings. Pseudocode of our approach is provided Algorithms and
EXPERIMENTS
We leverage DynamiCrafter (Xing et al., 2024), a state-of-the-art I.IB U-Net-based video diffusion model pre-trained on internetsccale videos, as our base model. We evaluate Vid2World across multiple domains, spanning real-robot manipulation, 3D game simulation, and open-world navigation. Results show that the transferred models not only achieve high-fidelity video predictions but also support downstream tasks in decision-making showcased by real-to-sim policy evaluation.
5.1 VID2WORLD FOR ROBOT MANIPULATION
Robot manipulation an ideal test for world models, demanding conditoond predictions that are both visually realistic and causally faithful under real-world physical constraints.
Setup. We utilize the RT-1 dataset (Brohan et al. 2023 a collection real-world robotic experi- ences spanning diverse manipulation tasks such picking, placing, and drawer operation. Our base model under extrapolative weight transfer is post-trained for 100k gradient steps (~ 7 days on 4x A100 GPUs), with two inference variants: (1) Vid2 World-NAR, which follows conventional video diffusion models and baseline methods by denoising all frames simultancously in a non-autoregressive manner, under homogeneous noise levels; and (2) Vid2Worle World, which denoises frames autoregressively with proposed action guidance. Evaluation uses standard video generation metrics, including FVD (Unterthiner al. 2018). FID (Heusel al. 2017). SSIM, PSNR, LPIPS (Zhang et al. 2018). and DreamSim Fu et al. 2023). Implementation details can be found in Appendix C.3.1
Baselines. We compare against a variety baselines introduced by Rigter et al. (2024) that build upon the same base model but utilize different transfer approaches, including Action- ArtionCoonditioned Fine-tuning, Language-Conua Conditioned Fine-tuning, ControlNet (Zhang et al. 2023). Classifier Guidance, and AVID (Rigter al. 2024). Details shown in Appendix C.3.2
Results. As shown in Table | Vid2World demonstrates strong quantitative performance across both non-autoregressive and autoregressive settings, outperforming or matching other transfer methods. In the non-autoregressive setting, it delivers superior or comparable results compared to all prior methods. Even the autoregressive generation setup, where other baselines are not capable of doing so, Worl still attains superior performance in FVD and FID, as well as par performance to previous best methods in other metrics, showcasing its strong capabilities in world modeling.
Application: Real2Sim Policy Evaluation. We further conduct Real2Sim Policy Evaluation to demonstrate our method's potential to aid downstream decision- makiing... Following SIMPLER (Li et al. 2025). goal is to estimate the performance of given policy by interacting with the world
7

---

Published as a conference paper at ICLR 2026
Table 1: World modeling performance across various domains. Best performances are in bold, second best are underlined. Dash (-) indicates the metric was not originally evaluated for that dataset. "Autoregressive prediction. -on-autoreeressivee prediction. =One-step prediction.
Model
FVD
FID
SSIM f
LPIPS
PSNR 1
DreamSim
Robot Manipulation: RT-1
Pre-trained Base Model
237.6
5.432
0.712
0.228
20.6
Classifier Guidance
213.1
6.005
0.683
0.250
19.8
ControlNet!
27.1
3.248
0.836
0.148
24.5
Action-Conditionn Conditioned
24.2
2.965
0.852
0.134
25.6
Language-Condittoneel Conditioned!
33.7
3.511
0.812
0.177
22.1
AVID
39.3
3,436
0,842
0.142
25.3
Vid2 Wo2WWold-NARR
18.7
5.871
0.856
0.140
25.8
0.048
Vid2World
18.5
5.806
0.842
0.152
24.6
0.054
3D Game Simulation: CS:GO
DIAMOND
577.1
115.6
0.449
0.547
18.2
0.2817
DIAMOND-HQ"
368.5
87.2
0.447
0,510
18.3
0.2416
Vid2World
106.6
17.5
0.481
0.404
18.7
0.135
Open- World Navigation: RECON
NWM (1B)*
31.2
34.1
0.389
0.295 + 0.002
15.343
0.060
0.091
0.001
NWM Ego4D (IB)²
41.0
34.9
0.361
0.368 + 0.003
14.072 + 0.075
0.138 + 0.002
Vid2World
59.4
42.9
0.481
0.3236
16.10
0.108
model, rather than the real world. This requires the model to perform autoregressive rollouts and faithfully predict the diverse outcomes induced from different policies. The procedure is summarized in Algorithm] 3 We evaluate on the task of closing drawers and consider three policy checkpoints from RT-1 (Brohan et al., 2023): RT-1 (Begin). RT-1 (15%), and RT-1 (Converged), representing different stages of training. Human evaluation is used as the verifier to annotate trajectory success. As shown in Figure 5 Vid2World reliably reflects the performance gap among policies, closely tracking their real-world success trends. Further details can be found in Appendix C.4
5.2
VID2WORLD FOR 3D GAME SIMULATION
Game simulation is key application for world models and neural game engines (Bamford & Lucas
2020 Bruce et al., 2024 Decart et al. 2024) have attracted growing attention. It is particularly challenging due to complex temporal dynamics and strong action dependence, involving rapid viewpoint shifts, contact-rich interactions, and fine-grained motion patterns that demand reasoning causally entangled visual-temporal cues.
Setup. We evaluate Vir2World the celebrated 3D video game Counter-Strike: Global Offensive (CS:GO) using the online gameplay dataset from Pearce & Zhu (2022) (5,5M frames, 95 hours), with the exact 0.5M-frame holdout set from DIAMOND (Alonso et 2024 for testing. DIA- MOND (Alonso et al. 2024), state-of-the-art autoregressive world model, generates the next frame conditioned on a fixed number of previous observations and actions. Following its setup with conditioning frames, we initialize with four history frames, and autoregressively generate frames until sequence length of 16. Evaluation metrics are the same as Section 5.1 computed on predicted frames excluding conditioning frames. More details are listed in Appendix
Results. As shown in Table I Vid2World outperforms both configurations of DIAMOND across all evaluation metries with significant margin, including 79.9% relative performance improvement
In the publicly released code of AVID Rigter al. 2024 the FID scores computed without setting the Inception model to evaluation mode, making it artificial lower. These results shown gray accordingly.
8

---

Published as a conference paper at ICLR 2026
Real World
Vid2World
LPIPS
FID
1.0
1,0
0.000
0.5
Sucpes Ratee
Succoss Rate
0.5
CA
014
0.260
0.3
6a
1000
0g
0.2
RT-1
RT-1 (199)
RT.1 (Coneeges)
RT-1 (lggin)
RT-1 (399)
RT-1
(Pegin)
(Canvergel)
Seconds
Seconds
NWM
1pps 4.ps DIAMOND + 1ppo - Vid2World
0ps
Figure 5: Wid2World for real2sim policy evalua- tion, validated by real-world evaluation.
Figure 6: Comparison autoregressive rollout for open-world navigation.
Ablation study two components of our proposed method: the choice of Weight Transfer (WT) mechanisms and the use of Action Guidance (AG).
Table 2:
Model
WT
AG
FVD
FID
SSIM
LPIPS 4
PSNR
Vid2World
Shift
29.9
7.85
0.799
0.185
21.5
Vid2World
Masked
29.4
7.07
0.824
0.169
22.9
Vid2World
Extrapolative
28.6
7.52
0.832
0.162
23.4
Vid2World Masked
25.8
6.84
0.840
0.159
23.9
Vid2World
Extrapolative
22.4
6.16
0.839
0.159
23.9
in FID and 71.1% performance gain in FVD compared the best baseline configuration. These results demonstrate the superior visual fidelity and semantic consistency of our method, showcasing potential for leveraging video diffusion models to interactive neural game engines.
5.3 VID2WORLD FOR OPEN-WORLD NAVIGATION
Open-world navigation is a fundamental capability for autonomous agents, with broad applications to autonomous driving Yuaan al. 2025) and roboties (Shah al. 2022).
Setup. We evaluate on the RECON dataset (Shah et al. 2022), which uses 3D (z, y. yaw) action space. Comparisons are made against two leading baselines, Navigation World Model (NWM) (Bar et al. (025) and DIAMOND (Alonso et al.) (222)). We also include NWM (+Ego4D), variant of NWM co-trained with action-free videos, aiming for oft-oandomain generalization. Unlike our model, which is restricted to sequential, autoregressive generation, NWM explicitly conditions on the prediction timestep f, allowing single-step prediction distant future frame. Accordingly, we evaluate against both baseline setups: single-step prediction and autoregressive rollout, at the dataset's native 4 fps rate. Especially in the autoregressive setup, by comparing our model to well-established baselines such NWM and Diamond at different timesteps, this offers clear insight into model's robustness against error accumulation. More details are listed in Appendix C.6
Results. In the single-step prediction setting (Table D. Vid2World achieves performance on par with NWM and surpasses NWM (+Ego4D) for out of 6 evaluated metrics, even under error accumulation from autoregressive rollouts. This is a particularly surprising result since NWM is state of the art model, and trained with significantly more computation and bypassing eттor accumulation via single- step prediction this setup. It's also worth noting that our model's prediction horizon 16 frames combined with history length of 4 results in total context length of 20 (00eexeeding the training horizon of 16 demonstttrainng strong temporal generalization. In the autoregressive rollout setup (Figure 6). model consistently produces predictions that are superior or comparable NWM, while significantly outperforming DIAMOND baselines. Taken together, these results highlight the effectiveness of Vid2World in leveraging rich priors from action-free video data, obviating the prohibitive data requirements and training costs associated with pre- pre-training on cross-domane action-labeled datasets.

---

Published as a conference paper at ICLR 2026
Robot Manipulation
1-2
1-12
Ground
truth
(context)
Predicted
Game Simulation
t
6-
1-
15
12
15
Ground
(context)
(context)
Predicted
Open-World Navigation
h=9
1-12
1515
t
Ground
truth
(context)
(context)
Predicted
Figure 7: Qualitative evaluation of Vid2World across various domains. Zoom in for details. Extended examples be found in Appendix D
380
11.0
3250
11:0
3.50
3.50
0155
0.40
0.60
130
0.50
3300
613
Figure 8: Video Prediction Metrics as a function of Causal Action Guidance Scale (A) in the CS:GO environment. While increasing À initially improves performance by enforcing action alignment, excessive guidance leads to degradation due to over-sharpening artifacts.
5.4 ABLATION STUDY
To verify the effectiveness of components in our method, we focus on two questions: (I) How critical is action guidance? (2) Which weight transfer mechanisms do best transfer? Due to limited computational budgets, all models are trained for 30k gradient steps. Regarding the first question, as shown in Table we observe that for both Extrapolative Weight Transfer and Masked Weight Transfer, enforcing action guidance yields better performance compared to their counterpart, which have never dropped out action training and inference. We further validate the impact of guidance scale on generation quality in Figure 8 detailed in Appendix D.5 Regarding the second question, both Masked and Extrapolative Weight Transfer yield better performance than Shift Weight Transfer, and utilizing Extrapolative Weight Transfer yields slightly better outcomes compared to Masked Weight Transfer. Hence, both techniques play a dominant role the superior performance of Vid2World.
CONCLUSION
In this work, we transform passive video diffusion models into interactive world models. We propose Vid2World, introducing two key mechanisms--y -video diffusion causalization and causal action guidance- -t support autoregressive, action- action-conditioneed generation. Extensive experiments demonstrate that Vid2World achieves state-of-the-ar performance in world modeling tasks and also effectively supports downstream decision-making. While this work marks a successful first attempt, it leaves plentiful space for further exploration. First, due to computational resource constraints, we are limited to employing a relatively lightweight video diffusion model as the base model. We envision that exploring larger-scale models (NVIDIA et al. (2255] Peng al. (025) may lead to better performance. Second, the training process remains relatively time-consumen coneemnn.umnng.. We look forward to future methods that achieve comparable or even superior performance with fewer training steps.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work complies with the ICLR Code Ethics. Human evaluations were conducted with informed consent from volunteers. All datasets and pretrained models are publicly available and used under their respective licenses. We have carefully designed our experiments to minimize risks of bias or discrimination. We uphold the principles of transparency and integrity throughout the work.
REPRODUCIBILITY STATEMENT
We have taken concrete steps to ensure that our results are reproducible. All datasets and models used in this work are publicly available and do not require special access. We have included detailed descriptions of our experimental setup, including detailed algorithmic descriptions (Appendix [) training and inference procedures (Appendix [B), hyperparameter configurations (Table B) and evaluation protocols (Appendix C). We have released the full source code, configuration files, and trained model checkpoints https: / knightt github vo/vi22wrrdd/ ensure full reproducibility and future extensions.
ACKNOWLEDGMENTS
The authors would like to thank Chaoyi Deng, Ningya Feng, Xingzhuo Guo and Yuchen Zhang for helpful discussions and instructive suggestions. This work was supported by the Natural Science Foundation of China (U2342217), the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (JYB202XXXXM803) the Beijing Scholar Program, and the National Engineering Research Center for Big Data Software
REFERENCES
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-588 -57791, 2024.
Chris Bamford and Simon M Lucas. Neural game engine: Accurate learning of generalizable forward
models from pixels. In 2020 IEEE Conference on Games (CoG), pp. 81-88. IEEE, 2020.
Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15791-15801, 2025.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall--3 pdf. 2(3):8, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv: 2311. 2023.
Nicholas M. Boffi, Michael S. Albergo, and Eric Vanden- How to build a consistency model: Learning flow maps via self-distillation. arXiv preprint arXiv: 2505. 18825, 2025.
Rishi Bommasani al. On the opportunities and risks of foundation models. arXiv preprint arXiv: 2008.07258, 2021.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. Robotics: Science and Systems XIX, 2023.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. Forty-first International Conference on Machine Learning, 2024.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets sequence diffusion. Advances in Neural Information Processing Systemx, 37:24081-24111 2024.

---

Published as a conference paper at ICLR 2026
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXhv:2310.19512 2023.
Anna Dawid and Yann LeCun. Introduction to latent variable energy-based models: A path towards autonomous machine intelligence. Journal of Statistical Mechanics: Theory and Experiment, 2023. doi: 10. 10.1088/1742-54466 2229////22//////222...
Decart, Julian Quevedo, Quinn McIntyre, Spruce Campbell, Xinlei Chen, and Robert Wachen. Oasis: universe transformer, 2024. URL httpttt://sssssmmooose gitpueb..gttubooo...
Google DeepMind. Veo 2: Our state-of-the-art video generation model. httpss://eeepd pooog/ee/tachnnoooggiess////
2/| 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
Stephanic Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. Advances in Neural Information Processing Systems, 36:50742-5017 2023.
Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. Adaworld: Learning adaptable world models with latent actions. In Forty-second International Conference on Machine Learning, 2025.
Zhengyang Geng, Mingyang Deng, Xingjian Bai, J Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505.13444 2025.
John Gkountouras, Matthias Lindemann, Phillip Lippe, Efstratios Gavves, and Ivan Titov. Language agents meet causality- bridging llms and causal world models. In International Conference on Learning Representations, 2025.
Google. Genie A new frontier for world models, 2025. URL httpatts////eepdddddd..gg// blog/genie 3 a new frontier for world models/
Daya Guo, Yang. et al. DeepSeek-rl incentivizes reasoning in LLMs through reinforcement learning. Nature, (14500081))33333 638, 2025. ISSN 1476-4687. doi: 10.
URL
https::/dol.orgg/ 11t.://0......1111110-00-----030
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In International Conference on Learning Representations, 2024.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. Ltx-video: Realtime video latent diffusion. arXiv preprint arviv: 2501.00103 2024.
Danijar Hafner. Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations. OpenReview.net, 2020. URL URLhtttps: /ttp///eeneeeeeeeeeeee/eeeeeeeee fort/formm? d= 1esss1111c44DD
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Empirical Methods in Natural Language Processing, 2023.
Simon Haykin. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1994.
Haoran He, Yang Zhang, Liang Lin, Zhongwen Xu, and Ling Pan. Pre-trained video generative models as world simulators. arXiv preprint arXiv: 2502.07825, 2025.
12

---

Published as a conference paper at ICLR 2026
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 15:6840-6851 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633-8646, 2022.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. 2010 20th international conference on pattern recognition, pp. 2366-2369, IEEE, 2010.
Mengkang Hu, Tianxing Chen, Yude Zou, Yuheng Lei, Qiguang Chen, Ming Li, Qiwei Liang, Yao Mu, Hongyuan Zhang. Wenqi Shao, et al. Text2world: Benchmarking large language models for symbolic world model generation. In ICLR 2025 Workshop on World Models: Understanding. Modelling and Scaling, 2025.
Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-fest gap in autoregressive video diffusion. ArXiv, abs/2506.0800 08009, 2025.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21 21807-21818 2024.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony Cassandra. Planning and acting in partially observable stochastic domains. Artif. Intell., (0(112))99-134, May 1998. ISSN 0004- 3702.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, 2012.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Oier Mees, Karl Pertsch, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, al. Evaluating real-world robot manipulation policies in simulation. In Conference on Robot Learning, pp. 3705-3728. PMLR, 2025.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5399. IEEE Computer Society, 2024.
Meta. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09990 2025.
NVIDIA al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv: 2501.03575, 2025.
OpenAl. Video generation models world simulators. htttps://ppenai..... index/ video ceneeration-modee as aaswworld simulators 2024.
OpenAl et al. Gpt-4 technical report. arXiv preprint arXiv: 0003008774,002.
PAN-Team. Pan: world model for general, interactable, and long-horizon world simulation. arXiv
preprint arXiv: 25/ /.09057, 2025.
Tim Pearce and Jun Zhu. Counter-strike deathmatch with large-scale behavioural cloning. In 2022 IEEE Conference on Games (CoG), pp. 104-111. IEEE, 2022.
Judea Pearl. Causality. Cambridge university press, 2009.
13

---

Published as a conference paper at ICLR 2026
William Peebles and Saining Xie. Scalable diffusion models with transformers. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4182. IEEE, 2023.
Xiangyu Peng, Zangwei Zheng. Chenhui Shen, Tom Young, Xinying Guo, Binluo Wang, Hang Xu, Hongxin Liu, Mingyan Jiang, Wenjun Li, Yuhui Wang, Anbang Ye, Gang Ren, Qianran Ma, Wanying Liang, Xiang Lian, Xiwen Wu, Yuting Zhong, Zhuangyan Li, Chaoyu Gong, Guojun Lei, Leijun Cheng, Limin Zhang, Minghao Li, Ruijie Zhang, Silan Hu, Shijie Huang, Xiaokang Wang, Yuanheng Zhao, Yuqi Wang. Ziang Wei, and Yang You. Open-sora 2.0: Training a commercial-level video generation model in $ 200k. arXiv preprint arXiv: 2503. 096429942 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PmLR, 2021.
7748-87658
Marc Rigter, Tarun Gupta, Agrin Hilmkil, and Chao Ma. Avid: Adapting video diffusion models to world models. Reinforcement Learning Conference, 2024.
Riot Games. Valorant. https://playyaloraatt com/ 2020. Video game.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Dhruv Shah, Benjamin Eysenbach, Nicholas Rhinehart, and Sergey Levine. Rapid exploration for open-world navigation with latent goal models. In Conference Robot Learning, pp. 674-684. PMLR, 2022.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pp. 32211 1-3252.. PMLR, 2023.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv preprint 17210718/2.01717, 2018
Ruoyao Wang, Graham Todd, Ziang Xiao, Xingdi Yuan, Marc-Alexandre Côté, Peter Clark, and Peter Jansen. Can language models serve text-based world simulators? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),
pp. 1-17, 2024a.
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In CVPR, 2024b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing. 13(4):600-612, 2004.
Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. In Advances in Neural Information Processing Systems, 2024.
Jialong Wu, Shaofeng Yin, Ningya Feng, and Mingsheng Long. Rlvr-world: Training world models with reinforcement learning. In Advances in Neural Information Processing Systems, 2025.
14

---

Published as a conference paper at ICLR 2026
Jinbo Xing, Menghan Xia, Yong Zhang, Hao Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien- Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In ECCV (46), 2024.
Sherry Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Leslie Kaelbling, Dale Schuur- mans, and Pieter Abbeel. Learning interactive real-world simulators. In International Conference on Learning Representations, 2024.
Shuai Yang, Wei Huang, Ruihang Chu, Yicheng Xiao, Yuyang Zhao, Xianbang Wang, Muyang Li, Enze Xie, Yingcong Chen, Yao Lu, and Song Hanand Yukang Chen. Longlive: Real-time interactive long video generation. arXiv preprint arXiv:2509. 22622,, 2025.
Shaofeng Yin, Jialong Wu, Siqiao Huang, Xingjian Su, Xu He, Jianye Hao, and Mingsheng Long. Trajectory world models for heterogeneous environments, In International Conference on Machine Learning, 2025.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Frédo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 229744, 2024. URLLtttps://apie semant icscholar
Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos. arXiv preprint arXiv: 2501 2025.
Mingfeng Yuan, Letian Wang, and Steven I.. Waslander. Opennav: Open-world navigation with multimodal large language models. arXiv preprint arXiv: 2507.18033 2025.
Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. In The Twelfth International Conference Learning Representations, International Conference on Learning Representations 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL net wwtp:/eeeeeeoeeeeeeeee...
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. In International Conference Machine Learning, pp. 61885-61896. PMLR, 2024.
Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. In Proceedings of Robotics: Science and Systems (RSS), 2025.
15

---

Published as a conference paper at ICLR 2026
A THEORETICAL JUSTIFICATIONS AND EXTENDED DISCUSSION
A.1 CONSTRUCTION OF COUNTER- COUNTER-EXAMPLE FOR SHIFT WEIGHT TRANSFER
We start by showing that even when the input sequence 2; _ ft) is L-smooth, Shift Weight Transfer (SWT) can still yield outputs with arbitrarily large error. Denote
y.
WiZt+imm m b.
Counter-example. Consider the one-dimensional input f(t) = at, where a > 0 is a scaling parameter. Clearly f is L-smooth for any finite L (since f (t) = 0). The original convolution output at time should be
Yr
WiZ++i + b.
The error term now becomes:
Iy SWT
Yell =
Σ
Wi (zttimm - 22+:)ll
wall
Implication. Even though f' is perfectly smooth (indeed linear), the approximation error of SWT grows endlessly as a + oo. Hence, the error not controlled by the smoothness constant alone. This shows that Shift Weight Transfer may catastrophically fail, motivating the more principled extrapolative construction.
A.2 DETAILED DERIVATION FOR EXTRAPOLATIVE WEIGHT TRANSFER
From first principles, we posit that Z can be linearly approximated over a window of past timesteps. Specifically, perform linear regression on {(2,, ()) and predict 2t44 based the regression result (2,42, t k))
Ze+k 2== Tk. Zt-j + Ba-
where Yk.., .hg are determined by linear extrapolation (OLS) from the past p features. Concretely, let 7j & 1 and define the empirical mean and variance of the timestamps:
p-1
S =
(T 22)
2
j=0
1m
Then the regression prediction admits closed form:
#-1
(1 k u)(T) - 14)
Σ
24th =
+
zz-j.
P
S
5=0
Thus, the coefficients are explicitly
(t+ k m)(TT - 11)
Kk.j
G =
S
Note that Σ, Fk. = 1, hence the intercept By vanishes automatically, and the extrapolation is expressed weighted combination of past features Zt-j _dd
Keeping in mind the design principle of maximally preserving the output representation of the original convolution, such that the new causal computation produces similar result the original non-causal one:
Σ
WiZ++i + b Σ jm w/ze4s + b'.
16

---

Published as a conference paper at ICLR 2026
We rewrite the left-hand side
Wiz++i + b WiZ +
i= Wiz++i + b
m
Wi Σ |=
p-1
WiZi+i
Vii Ze-j + Bi +b
>/ Yi WilWiZz-s)) )
Em=
WiZiii
i=1
Rearranging the terms with respect to z gives us:
w// = 1b2-ml Wy + + 1-p+1ss50 E. whh b'= Σ,- WrB,.
In the specialized case of m 1,p 2, Ze+k satisfies:
2 (k 1)z, kz,
Since 1, explicitly write out the three terms:
w/J 162- Wy
Σ m
i=1 W
wo 2wi
j =
w-1 W1, 0,
j j= j -
Also, b' = b. Since all temporal convolution layers in DynamiCrafter (Xing et al. 2024 have kernel size of 3, we are restricted to using the formulation for m = 1,p = 2. However, we anticipate that extrapolating with higher terms may lead to better performance well more complicated technical designs, which we leave for future work.
A.3 EXTRAPOLATIVE WEIGHT TRANSFER ERROR BOUND
Proposition 1. Assuming the input sequence 2, & f(t) is generated by a tvice-differentiabbe L- smooth function ft), the approximation error of the Extrapolative Weight Transfer (EWT) can be bounded by:
L m
6p p+
(p- 1)(p 2)
y
BWT
S
wel
2
Proof. The total error the weighted sum of the per-term extrapolation errors:
I
EWT
Σ w ||ze+i -Z4+ l|2:
(1)
We derive a complete bound for the per-term error ||z++ 2etr/22- Let (xx = f(t) + (x-
be the true tangent line at f, and ((x) be the OLS fitted line. The error is [] i) - + +(t+ We use the triangle inequality to decompose this error into three distinct sources:
(1 + 1) 1(t i)ll2 [ + a) - I(t + 1) S [I + (- 122 + (f(t) - (())l2
(tff(tt)) S)||2
(A) Taylor Error
(B) Intercept Error at
(C) Propagated Slope Ertor
We now bound each of these three terms.
(A) Bounding the Taylor Error By Taylor's theorem, there exists some (t, such that f(t + i) (t + & (E).. Given [Il2()))22 S L, this term is bounded by:
L 32 2
Ilf(t + 1) - (t + 1) S
17

---

Published as a conference paper at ICLR 2026
(B) Bounding the Intercept Error This term, [[f(t) - 1(t)ll,, represents the error of the OLS
prediction at time t. The prediction is ((t) = 2, = Li=0 Zc. We analyze the error component-
wise using Taylor expansion of f(t j) around t:
p
p-
((t) = Σ fo.sf(t - -j) = Σ
((t) (f)) (()
Yo.s
1=0
For the coefficient sums for Yo. = 1/2 + (t-p)(T, m
we have:
20,j = 1,
(2)
1=0
Σ joooj
p-1
1 Σ
j
1-1 11
- 11)
p-1
(p (p-1)/2
(-5) O.
(3)
S
2
5
/=0
Thus, the error simplifies to:
j2 (")(s)
1 p-1
(()-((()) = f(t)
(10)
Σ
70.s
-
Σ
j
%0.j (E5)
f=0
1=0
Taking the norm and the bound [))2 S L:
p-1
- (10() S 2
Σ
1=0
The sum be calculated in closed form:
p 1
Σ
j vo.s
_ t カー 1-> ET(T - -1)
1 p
S
0=0
(p- 1)(2p -1)
(p (p)/)/2/ ( p(p 1p + 1)
5
12
(p - 1) 1) (p 12/
(p 1)(p - 2)
S(p 1) S
Therefore, the intercept error is bounded by:
(p 1)(p 2)
llf(t) - [(t)ll - 2
6
(C) Bounding the Propagated Slope Error This can be achieved by bounding the slope-estimation еттог:
A a [l) $||. for the OLS fit on uniformly spaced timestamps T3 = =t- - j.
The OLS slope estimator is given by
-T)=)(11-1)111 $ 2T - 11)2 8
1 p-1
Σ 1=0
[ (
Using the Taylor expansion f(T)) = s(t) - jjf(()) + ("())), we can distribute the sums into:
s = S [(t)))) (t) 5it) 1)) 2 -G F
Using the properties in Equation 3 giives us
s 0 ())) 2 E(T 108 ( (())) =
1 Σ (T3- --) (()) 25
18

---

Published as a conference paper at ICLR 2026
Taking vector norm and using
(-)l L get the uniform bound
A = ||8 - f'(1)l2 -
1
Σ
p-1
-
Σ
p-1
25
25 1=
2=
As
Σ
(Tg 13)5 xp² Σ
[T3 1+) S pd. p-
j=0
therefore
L 6p
A S
2
1)p²
28
2 p+1
Combining All Terms
Combining the three terms, we get the complete per-term error bound:
L 1
L
6p
L
1)
|1141 2etill2
+
i
2
2
p+1
2
Substituting this back into Equation|] D
L 2
6p
L Lp
1)(p - 2)
I| EWT
+
i
2
2 +
2
6
==1
T. m
6p
(p
1)(p 2)
=
|w
8
i
2
p+1
p
This completes the full proof.
A.4 CAUSAL ACTION GUIDANCE AS PROBABILITY STEERING
formally demonstrating that our Causal Action Guidance mechanism is mathematically equivalent to sampling from a sharpened posterior distribution, effectively steering the generation process.
In this section, we provide the proof for Theorem 4.1
Theorem 4.1 (Causal Action Guidance as Probability Steering). Let H. == ([x,|rets (arlr<t-1) denote the history context excluding the current action. Under standard score-based formulation, the proposed score composition is mathematically equivalent to sampling from the following steered posterior distribution, where w (1 + X) is a fixed constant:
p((x= pxx+ He)
ar-1, He)
p(xe 1 He) x
p(x:
Hol)
x p(xe He) pp(a-1 Xx, He)
Eistory-Consisterr Prior
Action Alignment
Proof. We begin by establishing the relationship between the score functions and the probability densities. In the context of diffusion models (Ho al. 2020 Song al. (021b), the trained noise prediction network (x,,, estimates the score of the data distribution -v. log p(x: (up to scaling constant).
Let H4 == ([x)lr<e, (a-)r<(-1) denote the history context excluding the current action. We identify the two score components used in our method as:
Econd IX Vx, log (ug at-1, He),
Eucond
Vx, log p(x, Ht).
Applying Bayes" rule to the conditional density p(x; a He):
log,(xx a He) log p(x, (He)) log p(a-( x He) log logp(a.-1 He).
19

---

Published as a conference paper at ICLR 2026
Since the term log p(aa-1 H4) is independent of x,, taking the gradient with respect x yields the following decomposition of the conditional score:
V Xx, log p(x: a H4) Vx, log p(x;
M4) + Xx, logp(a--1 x1, H4).
Grand
Cucond
V log p(a--1 x,, He).
where y is a proportionality constant absorbed into the guidance scale. During sampling, the update rule for Causal Action Guidance can now be rewritten
Eguided
(1 + Econnd Xuucond
(1 + A) (Encomd - - Vx, log p(r-pa Xx, H.)) - Aumond
Gccnd - 7 (1 + X)V x. (ogp(a, X,, H4).
Recalling that Eguided X Xx, log p(xx a1-1, He), we equate the gradients:
Vx, log f(x; a,-1; H4) = Vx, log (4) + r (1 + X) logp(ar-1 Xt; He).
Denoting 7. (1 )) integrating both sides with respect x, recovers the log-density, up to constant C::
log p(x, a H4) = log p(x: | H4) + logp(a;-1 xx, H₂)a + C.
Exponentiation both sides yields the steered posterior distribution:
p(xe a He) x p(xe He) p(aa-1 X H,)
This completes the proof.
A.5 CLARIFICATION ON CAUSAL TERMINOLOGY AND CONNECTIONS TO INTERVENTIONAL. WORLD MODELS
In this work, we bridge concepts from video generation and world modeling, domains where the term "causal" carries distinct nuances. Throughout the paper, our usage of "causal" primarily refers to Temporal Causality, focusing on the sequential evolution of enviromment dynamics. However, specifically within our proposed Causal Action Guidance, the mechanism also implicitly embodies Interventional Causality, i.e. p(8++1 St, do a.)). In this section, we clarify these two interpretations and discuss their connections to the broader causal inference literature.
Temporal Causality. In the deep learning community, particularly in autoregressive sequence modeling, "causal" typically refers to the temporal structure or non-anticipative dependency. A model is temporally causal if the prediction at time t, x,, depends only the past context x<( and is independent of future information Xyt. Our Video Diffusion Causalization (Section [.1) strictly follows this definition. Standard video diffusion models utilize bidirectional attention, violating the arrow of time required for online interaction. Our method repurposes these architectures to strictly enforce temporal causality, establishing the structural foundation for autoregressive rollout.
Interventional Causality. While our framework is built on temporal causality, our Causal Action Guidance (Section (.2) naturally extends into the realm of interventional causality (Pearl 2009). In causal inference, this refers to reasoning about the effect of an intervention, i.c., estimating p(S(+1 st, ()) By injecting the action signal a, and utilizing classifier-fre free guidance amplify its likelihood, we are effectively performing an intervention (a.a.. ). This forces the model to generate the specific counterfactual future resulting from that action, rather than generic future based solely on observational correlations.
In Vid2World, we establish temporal causality as the necessary architectural prerequisite to enable on- line rollout, while leveraging action guidance to implicitly enforce interventional causality. Together, these components enable controllable and action-conditioned world simulation.
A.6 EXTENDED DISCUSSION ON DATA DISTRIBUTION MISMATCH DURING VIDEO GENERATION AND WORLD MODELING
While Vid2World is motivated by leveraging the broad visual experience and rich physical priors encoded in the pretrained video diffusion models, it is natural that the distribution of scaee pretraining video data differs from the agent-centric interaction data used in world modeling. To this end, we highlight two key dimensions of this data distribution gap:
20

---

Published as a conference paper at ICLR 2026
1. Scene Composition and Object Distribution. Internet videos span wide variety of environments, objects, and scene layouts, whereas world-model training datasets generally contain more controlled, task-centric settings with narrower range of objects and inter- actions. Although this introduces a distribution mismatch, the underlying compositional regularities (e.g- object permanence, contact dynamies, spatial relations), are shared across all sources and serve as transferable priors.
2. Motion Scale and Interaction Granularity. Pretraining videos typically capture large- scale, coarse global motions, while world models focus on fine-grained, local interactions between agents and objects. Despite this difference in motion granularity, low-level physical regularities, such as temporal continuity, acceleration smoothness and occlusion dynamics, remain consistent, supporting transfer of motion priors into action-conditioned rollouts.
Even with these mismatches, we argue that by large-scale pretraining on diverse visual experience, the video diffusion model captures strong and broadly applicable visual and physical priors. This is especially true in our Vid2World approach. Building upon DynamiCrafee (Xing et al. 2024), an image animation model fine-tuned from the general-purpose video generation model VideoCrafter (Chen et al.) [023), Vid2World inherits both the broad visual experiences from internet-scale video pretraining and the structured motion priors essential for producing physically coherent dynamics, properties that are directly beneficial for action-conditioncs generation.
A.7 EXTENDED DISCUSSION ON UTILIZING VID2WORLD FOR DOWNSTREAM TASKS
While Vid2World demonstrates a successful first step at transferring video diffusion models to world models, due to the large parameter size of the pretrained model and the iterative process in diffusion, the model does not enjoy fast inference speed, compared counterparts trained using teacher forcing, see Appendix[E] for further details. However, we fully acknowledge that the world model's performance in downstream tasks is a critical factor for evaluating the world model's applicability, especially for applications in domains such as embodied artificial intelligence.
We conduct the downstream task of Real2Sim Policy Evaluation in the RT-1 environment (Section 5.1, a challenging task validating the model's effectiveness in discriminating the performance of different policies as well as serving as reference of the policy's real world success rate. Due to limited computation resources, we did not explore training agents via reinforcement learning with our world models, as reinforcement learning is notoriously known as being sample inefficient.
As the model scale grows increasingly larger, the computation cost grows higher accordingly. Even at industry-level world models, such as Genie 3 (Google 2025), V-JEPA 2 (Meta) 2025) and PAN (PAN-Team) 2025). downstream tasks especially reinforcement learning are not conducted. We believe this is a collective effort where multiple domain may require breakthroughs:
1. Novel world modeling methods with faster inference speed: Current high-fidelity world models are dominated by diffusion models. With recent advances in one-step and few-step generative models showing feasibility in domains such as text to image generation (Song) al. 2023] Bofh et al.) [025] Geng et al.) (025). we anticipate world modeling methods that generate high-fidelity future predictions with improved inference latency.
2. Sample-Efficient Methods for Model-Based Reinforcement Learning: Reinforcement Learning is notoriously known for being sample inefficient. We strongly believe that designing model-based reinforcement learning algorithms with improved sample efficiency (both for offline and online reinforcement learning) may be crucial fully unlock the potential of training policy model inside world models with reinforcement learning.
3. Hardware Acceleration Methods for Inference Speedup: Aside from algorithmic in- novations, strongly believe hardware speedup methods are necessary to unleash the full capabilities of world models in downstream tasks, especially planning, control and reinforcement learning. Such improvement may include novel methods for implementing KV Cache in the context of world modeling (Yin et al., 2024), Huang et al.] 2025 Yang et al. 2025), better utilization of GPU locality and developing hardware-friendly software packages that are easier to use.
21

---

Published as a conference paper at ICLR 2026
B VID2WORLD IMPLEMENTATION DETAILS
B.1 ALGORITHM PSEUDO-CODE
In this subsection, we provide the pseudo-code for training and autoregressive inference of Vid2World.
Algorithm 1 Vid2World Training
Algorithm 2 Auto-Regressive Sampling
Input: Model 0, Trajectory dataset D.
Input: Model 0, Initial observation zo, Action quence aa-oorr- Action guidance scale A.
2:
loop
3: Sample trajectory [x[', af Jo:r from D
2: Initialize x; ~ (0, o I); Vt E I,...?? T.
4: for = O,.., T do
3: for f = 1,..., T.. do
5:
x;
{(- x kz), le ~ 210, K]
4: for l: K ..... 0 do
5: è cs ca(((x [xx,, [a. [O,...,, 0,)
6:
å,
Ø
w.p.
then
a o/w.
7:
Ex = ca c ([x+] I (arct-1, Ø). [0. (0...... .... k))
x
7:
E
/1 V1- (1x*
8:
é (1 + A)ē Xeu
8:
é,
Is [ar (k))
9:
end
10:
w ~ N)(1.)
9:
end for
10:
C -MSELoss((en ēn], [et..... En])
11:
Ie Jan (xi
1-mg 1-mg é) dew
11:
Backprop with L and update
12:
end for
12:
end loop
13:
end for
13:
Return Model
14:
Return X0:T.
B.2 MODEL DETAILS
Base Model Details. The pre-trained model DynamiCrafter (Xing et al. 2024 is a state-of-the-art latent video diffusion model conditioned on text and image, with its full-sized version ranking high on the VBench leaderboard (Huang et al. 2024). It builds on the Stable Diffusion variational autoencoder (Rombach et al. 2022) and trains 3D U-Net for video generation using web-scale video data. Specifically, starting from the pre-trained VideoCrafter T2V model (Chen et al. 2023). DynamiCrafter introduces dual-stream conditional image injection paradigm: in one stream, CLIP (Radford et al.) 2021) image encoder embeddings are fed into the U-Net via cross-attention in the other, images are encoded into VAE latents, which are then replicated along the channel dimension for the full video length and concatenated with the initial noise latents. This mechanism simultaneously injects text-aligned semantic representations and fine-graine visual details, improving video quality. For the noise level k the model injects such information into the diffusion network by firstly using sinusoidal embedding to transform it into a vector, which is subsequently fed into a two-layer MLP, obtaining learned embedding. The embedding is then added to the convolutional features to provide the noise level condition. Since the base model only contains temporal convolution layers with kernel size 3, our Extrapolative Weight Transfer Method is applied using hyperparameters
1,p
Image Preprocessing. For all of our experiments, we use the publicly released DynamiCrafter model at 320 x 512 resolution, which has I.IB trainable parameters. During data preprocessing, we resize the shorter side to 320 px while preserving the aspect ratio. After resizing, if the longer side remains below 512 px, we pad with black borders up 512 px; otherwise, we take the other approach: resizing the longer edge to 512 px, and pad with black borders on the height dimension. This setup is used in both training and inference. For evaluation metrics calculation, we resize the model output to the baseline method's 's resolution. For instance, in CS:GO, we calculate the metrics by firstly cropping out the black paddings the model output, followed by resizing to 150 x 280 resolution.
Noise-level Conditioning. The structure of noise level embedding layers naturally supports the transformation different noise scales at different frames. Specifically, instead of broadcasting the identical noise level sinusoidal embedding along the temporal axis, we use the independently sampled noise level at each frame, stacking it in the temporal dimension.
22

---

Published as a conference paper at ICLR 2026
Action Conditioning. For action conditioning, inject frame-level action conditions into the base model, similar to the injection of noise levels. For cases where actions are discrete, we alter the first layer of the noise conditioning network into learned embedding layer. For cases where the action space is continuous, we simply switch the first layer to a linear projection. The embedding obtained through action conditioning is later integrated with the noise conditioning through element-wise addition.
B.3 TRAINING DETAILS
We use the 320 x 512 version of DynamiCrafter (Xing et al.) 2024) as the base model for all experiments. For robot manipulation, game simulation as well as open-world navigation tasks, train for 100k gradient steps; for ablation studies, all models are trained for 30k steps. The training is conducted using 4 x 40GB NVIDIA A100 GPUs.
B.4 INFERENCE DETAILS
During autoregressive rollout, we denoise the current frame by fixing noise levels at the history frames to be zero, whereas denoising the current frame using DDIM (Song et al., [021a), In practice, following diffusion forcing (Chen et al., 2024), we add a small noise Ksmall uniformly to history frames. Under all settings in this paper, concerning action guidance, we apply guidance scale of 2.5 for our experiments, optimal values of these hyperparameters are related to domains, and an extensive hyperparameter search can lead to even better performance. A detailed list of hyperparameters regarding the model architecture, training, and inference process shown in Table
well as a guidance rescale factor (Lin et al. 2024) of 0.7. We believe that the
23

---

Published as a conference paper at ICLR 2026
Table 3: Hyperparameters for Vid2World
Hyperparameter
Value
Architecture
Base Model:
Resolution
320 512 Truc
Latent Diffusion
Downsample Ratio
2-shape
32 x 32 4 320
U-Net Chancels
Noise level Conditioning: Emhedding dimension
1024
Action Conditioning: Embedding dimension
1024
Other Conditioning:
Language condition
Empty Sequence
FPS condition
3
Image condition
First frame
Training
I Learning rate
1.0 10
# training steps Batch size per GPU
100k
# GPUs
Accumulate gradient batches GPU-type
2
A100-40GB
Diffission Setup:
Diffusion steps K
1000
Noise schedule
Linear
80
0.00085
Ax
0.0120
Noise level along Temporal Axis
iid. samples
Data Processing:
Input video length
16
Normalize
[-1,1]
Input resize
Resize, Center-Crop
Brightness
[0.9,1.1]
Contrast
[0.9,1.1]
Saturation
[0.9.1.1]
Hue
[-0.05,0.05]
Causalization:
Mixed weight transfer Causal Mask for Temporal attention
True
True
Action Conditioning:
Dropout rate p
0.2
Sampling along Temporal Axis
iid. samples
Sampling
Sampler
DDIM
Steps
Timestep spacing
Uniform trailing
Action Guidance scale
2.5
Guidance rescale
0.7
Kymall
24

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS
C.I DATASET DETAILS
RT-1 Details. RT-1 (Brohan et al. 2023) a widely used dataset consisting of real- world robot experiences, spanning multiple robot manipulation tasks, including opening drawers, closing drawers, picking and placing. Each episode is sampled at an fps of 3, with the embodiment, a robot arm, performing certain tasks. In addition to video frames, it also records action sequences as well as annotated language prompts. In our setup, we use the observations obtained by RGB cameras, as well as the action sequence.
CS:GO Details. We use the publicly released dataset collected by Pearce & Zhu (022). ). It contains different subsets of human players interacting with the CS:GO maps, spanning from expert-level to novice players. Here, use the largest subset in their dataset, which contains 5.5M frames (95 hours) of online human gameplay from the map Dust The dataset is created by scraping user behaviors online servers, offering a diverse set of interactions from policies of all sorts. For each timestep, the actions are represented as an array of discrete values.
well-known open-world navigation dataset. consists of 40 hours across 9 open-world environments, collected using a Clearpath Jackal UGV platform. The dataset is collected fps of 4, and The action space defined as a 3D vector a (zr, y. yaw), where (æc, y) RR2 denotes translation along the forward/backward and left/right axes, and yaw E R. denotes the change rotation angle. Formally, each action is given by the proprioceptive state difference between timesteps, i.e.,0, = Sy 8,, where is the agent's s proprioceptive state and denotes either the next timestep or a future timestep of interest (as in NWM).
RECON Details. RECON (Shah al. 2022)
C.2 METRICS FOR VIDEO PREDICTION
For Robot Manipulation, Game Simulation and Open- World Navigation tasks, we adopt commonly used video prediction metrics for image or video generation tasks. These metrics measure either the pixel-level or the semantic-level similarity between the generated videos and the ground truth videos. For metrics calculated on each image, the values are obtained by extracting all frames and treating them as independent images for feature extraction and statistical estimation.
Next, we provide a description for each metric:
We compute the Fréchet Inception Distance (FID) introduced by Heusel et al. (2017). FID measures the Fréchet distance between multivariate Gaussians fitted Inception-v3 activations of real and generated frames. Specifically, let μr, E, and My. Mg denote the empirical means and covariances of these activations for real and generated frames, respectively. FID is defined as:
FID.
FID P₂) ||,,, 11113 + Tr (E, Eg - 2EEEE,)))
FVD. Fréchet Video Distance (FVD), introduced by Unterthiner et al. 2018), generalizes FID by embedding entire video clips via pre-trained Inflated 3D ConvNet (I3D) and computing the Fréchet distance between the resulting feature distributions real and generated videos. Concretely, let P. and Py be the distributions of I3D activations for real and generated videos, respectively, with empirical means pr, Hy and covariances E,, Eg. FVD is then defined as:
FVD(P, Pa) = ||, Mo 112 + E, + Eg - 2 ((E,,)))
SSIM. Structural Similarity Index Measure (SSIM) (Wang et al. 2004 quantifies perceptual similarity by jointly comparing the luminance, contrast, and structural information between two image patches. Given a pair of patches x and y, let Pas By be their mean intensities, o2, oy their variances, and any their covariance. The SSIM index is calculated using:
(2prrly C1) (20., C2)
SSIM(H, 9)
(nz + F C1) + C2)
25

---

Published as a conference paper at ICLR 2026
where 11 222 and C2 (K2L)² are stability constants with L the pixel dynamic range. For our purpose, we compute SSIM over an 11 x 11 Gaussian Gaussian-weighted sliding window and average the local SSIM values to obtain mean SSIM (MSSIM) per frame; the final video-level SSIM score is the average MSSIM across all sampled frames.
LPIPS. Learned Perceptual Image Patch Similarity (LPIPS) (Zhang al. 2018 measures percep- tual similarity by comparing deep feature activations of real and generated frames across multiple layers of a pre-trained network. Specifically, let (x) and (v) be the mnit-normalized activations layer for inputs x and y, and Wa the learned channel-wise weights. LPIPS is computed via:
H. LPIPS(x. 9) Σ EEWLE
wy (Phmola) Ph P
HWW,
It is worth noting that for evaluation in RT-1 and CS:GO, VGG (Simonyan & Zisserman 2015) as the feature extraction network, whereas in RECON, we use AlexNet (Krizhevsky et al. 2012 as the network, following baselines.
PSNR. Peak Signal-to-Noue Noisee Ratio (PSNR) (Hore & Ziou 2010) quantifies pixel-Ievel fidelity by comparing the maximum possible pixel intensity the mean squared error (MSE) between two frames. PSNR is defined as:
L
PSNR( v) 10 log10 MSE(2, (x,
where L is the maximum pixel value (e.g. 255 for 8-bit images).
2023) is relatively new metric for measuring perceptual image DreamSim. DreamSim (Fu et al. similarity, which aims to evaluate perceptual similarity. This is accomplished by comparing deep features from neural network. The resulting metric is better aligned with human perception.
C.3 DETAILS VID2WORLD FOR ROBOT MANIPULATION.
C.3.1 IMPLEMENTATION
We make use of the RT-1 (Brohan et al. 2023) dataset. To align with the baseline evaluation methods, we randomly split 4361 episodes as the holdout set, using the remaining 82851 episodes as the training set. In this case, since the action space is continuous, use linear layer the first layer to add the action condition, as described in Appendix B
Following baseline (Rigter et al., 2024), train the model for up 100k gradient steps on 4 x A A100, which takes less time (6.4 days) than the seven days reported for training baseline methods. During training. the model inputs are video and action sequence segments of length 16. At test time, we randomly sample 1024 episodes from the evaluation set, and sample a segment of 16 frames for each episode. The model provided with the first frame of the segment as well as the action sequence, and the metric is calculated on all 16 frames, the same as baseline methods.
C.3.2 BASELINES
We compare Vid2World with several baselines, all utilizing the same base model (Dynamicrafter (Xing al. 2024), resolution 320 512), while differing in their transfer methods. worth noting that for all baseline methods in this setting, the model is transferred without enforcing causality, neglecting the need for interactiveness: i.e.. the models are still trained and sampled with homogeneous noise levels in all frames and the model is still architecturally non-causal. Therefore, the transferred models are unable perform autoregressive rollout. During testing, the models generate videos in non-autoregressive manner. Next, we provide brief introduction to each bascline method:
Action-Condinn Coliin-Codditooeed Fine-tuning. In this approach, all parameters of the pre-trained model fine- tuned on the oction-conditionn dataset. For each timestep of the noisy video x, the corresponding action a' is embedded compute the action embedding e using embedding table for discrete
26

---

Published as a conference paper at ICLR 2026
actions linear layer for continuous actions. For RT', action embeddings are both concatenated with and added to the corresponding timestep embeddings.
Language-Condee Conditioned Fine-tuning. Iinguage-Conditionel Fine-tuning fine-tunes the pre-trained model using a textual description of each video. Each description is embedded via CLIP (Radford al. 2021 and incorporated through cross-attention following the approach of the original model.
ControlNet (Zhang et al., 2023). ControlNet freezes the parameters of the pre-trained model and creates trainable copy of its UNet encoder. The trainable branch is conditioned on the action signal and connected to the original decoder via zero-initialized convolutions. In this work, ControlNet is employed with the aim of incorporating action-conditioning into the diffusion process.
Classifier Guidance (Dhariwal & Nichol 2021 A classifier fa xi)) is trained on noisy images I; predict actions. With weight u, this classifier steers the diffusion sampling process toward samples that are consistent with the specified actions. The resulting noise prediction is
Emnal (24, a, i, 2o) = Epre (Zz, i, 20) v1 a V. log fa(())))
C.4 DETAILS OF REAL2SIM POLICY EVALUATION
Real2Sim Policy Evaluation (Li et al. 2025) aims evaluate policies using simulation as a surrogate for the real world, serving an indicator of the performance of different policies. This interaction between the policy and the simulation environment requires world models to generate images in an interactive manner. A well-performing model should be capable of distinguishing successful trajectories from failure cases by autoregressively simulating the outcomes of different policy actions.
We employ Vid2World as the world model to evaluate three policies; RT-1 (Begin), RT-1 (15%), RT-1 (Converged), taken for different stages of RT-1 (Brohan et al.] 2023) training. Specifically, we sample N trajectories from the RT-1 dataset for the given task and extract their initial frames. These frames are provided to each RT-1 policy to generate actions, which are then fed into the world model to simulate the next frame. The policy continues
these imagined frames in iterative manner.
For the first frames, new frames are generated autoregressively based on all previously observed frames. Beyond this point, each subsequent frame is generated based sliding window of the most recent L frames. This process continues until a sequence of length H is produced. We then employ verifier to determine whether each trajectory is successful, and compute the overall success rate accordingly. In our experiments, we sample trajectories from the "close drawer" task in the RT-1 dataset. For each policy, we use sample number = 50, sliding window length 10, and rollout horizon H = 40. For simplicity, we use human evaluation as the verifier v).
The complete procedure described in Algorithm
Algorithm 3 Real2Sim Policy Evaluation
Require: World model P(o44 (gr,agt) policy r(ar)o)). task x, initial frame D., trajectory success verifier H)( > 1).
1: Init success count +
2: for N de
3: Sample initial frame ou from Dx
4: for = 0,... ..... H do
5: Sample a. T(- o:)
6: i < then
7:
Or+1 P(or+1/og,,00 og.)
8: else
O
10: end if
11:
end for
12:
success_comme +- success_couuee (000.1))
13:
end for
Return success_raate rate success_ccom
We provide the instructions for human verification below:
27

---

Published as a conference paper at ICLR 2026
Instruction for Human Verification
Watch each clip of the robot attempting to close drawer and decide if the attempt succeeds or fails: label Success when, by the final frame, the drawer face sits flush with the cabinet frame (no visible gap and no rebound); label Failure when any gap remains, the drawer re-opens after contact, the robot jams or stops short, or the view prevents you from confirming full closure.
C.5 DETAILS OF VID2WORLD FOR GAME SIMULATION
C.5.1 IMPLEMENTATION
We utilize the largest subset the CS:GO dataset (Pearce & Zhu 2022). Following DIAMOND
(Alonso et al.] 2024), we use exactly the same holdout set of 0.5M frames (corresponding to 500 episodes, or 8 hours) for testing. As actions are discrete values in this domain, the first layer in the action is injected via a learned embedding layer. For training and evaluation purposes, we use segments of 16 frames. For evaluation, since DIAMOND (Alonso al. 2024) requires frames as history context, autoregressively generate frames from four consecutive history frames, until sequence length of 16 is reached. In this experiment, the metrics are calculated only on the predicted frames, excluding frames used for conditioning. Since the output of the baseline method, DIAMOND, in resolution of 150 x 280, we downsampled our generated image to match this corresponding resolution. For our model, we train for 100k steps.
C.5.2 BASELINES
We use DIAMOND (Alonso et al. 2024), state-of-the-are autoregressive world model the baseline. It treats the world modeling task as an image generation problem, which learns an image diffusion model based on the previous four observations and actions. In practice, the input image diffusion model is downsampled, and a separate upsampler is learned to upsample the diffusion model's output to higher resolutions. Here, we use the publicly released checkpoints of DIA.MOND, which contain both the diffusion model and the upsampler. We evaluated both sampling configurations provided by the authors, namely:
1. DIAMOND-Fram Under this configuration, the model generates images with lower fidelity in exchange for faster inference speed, necessary for interactive gaming.
2. DIAMOND-HQ: This is the configuration where the generated images have higher fidelity, coming at the cost of slower inference speed.
We test our model's performance with baseline performances using exactly the same test set. Addi- tional generation results can be viewed in Appendix D.2
C.6 DETAILSS OF VID2WORLD FOR OPEN- WORLD NAVIGATION
C.6.1 IMPLEMENTATION
We utilize the RECON dataset (Shah et al. 2022), well-celebrated dataset for open-world navigation. Following baseline implementations, we split the data into two parts: 9,468 videos for training and 2367 videos for evaluation, using exactly the same data split. Since the action space is continuous, we use linear projection as the first layer for injecting actions. During training, we preprocess the image into 320 x 512 resolutions by padding 320 x 320 with black borders. During evaluation, we cut out the black borders and downsample the image to 224 x 224, making comparable with baselines. During training, dataset is collected at 4 fps, for our evaluations into 48 into the future, the model is provided with history of 4 frames (following baselines) and predicts sequence of 16 frames, creating total context length of 20 frames. which is longer than the training horizon. In our experiments, we are focused on two setups: Single-Step Prediction and autoregressive Prediction.
context length of 16, with no downsampling in the temporal fps. Since the
Single-Step Prediction. The Single-Step Prediction set contains 500 video segments. Since NWM capable of single-step prediction a future timestep within its training horizon, the model is evaluated given 4 frames of history context and asked to single-shotedly predict the observations
28

---

Published as a conference paper at ICLR 2026
4s into the future. Our model, however, must generate predictions in sequential manner; hence, evaluations are done using autoregressive inference. It is worth noting that this makes the problem significantly harder, as the model's prediction will degrade with respect to the rollout horizon due to error accumulation. The results are shown Table [ For image generation metrics (i.e., all metrics except FVD), we report the results the predicted frame, different from RT-1 and CS:GO evaluations, where we report means across all predicted frames. For FVD, we report the metric acquired by evaluating the video sequence of all 16 predicted frames. For LPIPS, PSNR and Dreamsim, we take the results from of NWM directly from the reported numbers in their paper, whereas for FVD, FID and SSIM, we report our evaluated numbers, which are obtained by building on the official implementation of NWM.
autoregressive Predictions. In this setup, the evaluation set consists of 150 video segments. Here, baseline methods well Vid2World conduct inference via autoregressive rollout. For baseline methods, except for the normal version of predicting future frames at fps, there is also a downsampled version, which predicts future frames at 1 fps. This results in fewer autoregressive rollout steps, potentially leading to less error accumulation. For results in this domain, following baselines, evaluate our model for 5 parallel runs using different random seeds, and report the mean and std.
C.6.2 BASELINES
Here we consider two state-of tfate-of-the-ar art baselines: Navigation World Models (NWM) Bar et al. 2025 and DIAMOND (Alonso al. 2024).
NWM. Navigation World Model (Bar et al. 2025) is a state-of-the-an model, built on novel architecture CDiT. At its core, the model takes in action as well as the predicted timestep as conditions, and the backbone, follows the architecture of DiT (Peebles & Xie 2023) used in image generation. This equips the model with the ability to single-step predict timestep in the future. Here we use CDiT-XL, IB model trained on various action-labeled cross-domain data, leveraging 4 history frames as context. The original autoregressive setup is fps, and fps denotes the autoregressive rollout by predicting the future 1 second from the current time. We also consider model variant: NWM+Ego4D, which was co-trained with action-fred video data to improve out-of-distribution generalization. It is worth noting that the NWM model is trained on 8 nodes, each with 8 Nvidia H100 GPUs, trained for 100k (NWM) 200k (NWM+Fgo4D) gradient steps using a batch size of 1024. This is significantly more computationally expensive than Vid2World's setup, with trained on 4 Nvidia A100 40GB GPUs for 100k gradient steps using a batch size of 32.
DIAMOND. DIAMOND (Alonso et al. 2024) is also the baseline we used for CS:GO. In this setup, following NWM, the model is trained from various cross-domain data, and inference is done using autoregressive generation. Additionally, we include DIAMOND (Ifps), which a model trained using observations and actions intervals of second.
We use exactly the same training and test split, and the same evaluation samples; showcases of generation results are included in Appendix D.3
29

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS AND VISUALIZATION RESULTS
In this section, we provide additional visualization results for our proposed Vid2World model. Generated results from our model are obtained by autoregressive rollout. Section D.]
include
visual results for the RT-I dataset in the video prediction task. In Section D.2] we provide generated
results in the CS:GO environment under the video prediction task. In Section D.3
we include
showcases of Vid2World generation in RECON environment. In Section D.4 we provide some generated examples for the Real2Sim Policy Evaluation experiments.
D.1 GENERATION RESULTS OF RT-1
We provide additional visualization results for Vid2World on the RT-1 Dataset in Figure 10 As shown in the figure, our model makes video predictions that accurately represent the environment dynamics. Our world model generates physically plausible frame sequences with high fidelity, offering great potential in video prediction tasks.
However, limitations still exist. We provide two examples of such limitations in Figure 9 into two categories:
These fall
Failing to predict fine-grained control: In the upper case of Figure 9
the model predicts
the moving directions of the robot arm successfully, but fails to capture the gripper's 's control over the green bag.
2. Regressing more familiar scenes: In the lower case of Figure although the robot movement is mostly correct, the grasped object changes to more often seen object.
10
14
GT
Ous
Figure 9: Failure cases for RT-1 dataset
30

---

Published as a conference paper at ICLR 2026
GT
Oun
GT
GT
Owns
LE
GT
Ous
GT
Ours
GT
GT
Oun
GT
Ous
Figure 10: Comparison between ground truth and generated videos by Vid2World in RT-1 Environ- ment. The first frame is provided as context.
31

---

Published as a conference paper at ICLR 2026
D.2 GENERATION RESULTS OF CS:GO
We provide generation results of Vid2World compared baseline methods (DIAMOND (Alonso et al. 2024) the CS:GO environment. We observe several interesting phenomenon, demonstrating the characteristics, both in strength and in limitations, of our model. We provide the discussion below.
Error Accumulation. A common challenge for autoregressive models (for example, the baseline model DIAMOND) in multi-step prediction is performance degradation due to error accumulation, which is especially pronounced when consecutive frames exhibit large variation. In Figure [1] compare the qualitative predictions of Vid2World and DIAMOND under rapid viewpoint changes. By contrast, DIAMOND's frames become progressively blurred; Vid2World maintains sharpness and closely follows the ground truth trajectory.
1-6
|-8
(-10
H-
[
Ours
DIAMOND-HQ
DIAMOND-FaL
Figure 11: Error Accumulation in CS:GO. While DIAMOND's fidelity degrades significantly during rollout, Vid2World maintains high-quality generation with strong physical accuracy.
Action Alignment. The reliability of a world model, to a large extent, depends on how well its predictions align with the input actions. As shown in Figure T2] Vid2World accurately reflects the ain-down-sightt action in its predicted video, whereas DIAMOND fails to manifest this action.
H
(-2
(
1-6
1-8
I-10
(-12
[-14
Ours
DIAMOND-HQ
DIAMOND-Fast
Figure 12: Action alignment in CS:GO. Vid2World truthfully reflects the aim-down-sights action in its predicted video, while DIAMOND fails follow the action.
Failure Cases. Despite substantially reducing the accumulated error and preserving action alignment, Vid2World still encounters failure cases, as demonstrated in Figure [3] In this figure, neither Vid2World DIAMOND matches the ground truth. Although the model's capability is one important factor leading to failure, the environment's randomness, in this case, the place for the player's respawn, also adds the difficulty.
32

---

Published as a conference paper at ICLR 2026
I-0
1-5
1-8
L-10
[-14
Qurs
DIAMOND-HQ
DIAMOND Fast
Figure 13: Failure Cases in CS:GO environments.
Action influence generated sequence. For world models, it is important to do so-called counterfactual reasoning with the current action, instead of predicting trends based solely on past observations. In Figure 14, showease the capability of our model to perform generation based on action sequences. All trajectories start from the same observation, but lead to completely different generated frame sequences due to different action sequences.
6
1-10
1-2
1-14
Forward
Lelt
Backward
Right
Look left
Look right
Look
Look
Figure 14: Effect of different actions on generated videos in CS:GO for Vid2World. Trajectories start with the same initial observation, diverging drastically result of different action sequences.
33

---

Published as a conference paper at ICLR 2026
D.3 GENERATION RESULTS FOR OPEN-WORLD WORLD NAVIGATION
We provide generation results of Vid2World in the open-world navigation video prediction task, as shown in Figure 15
t
14
6
1=8
10
t-12
14
16
t 18
GT
Predicted
GT
Predicted
GT
Predicted
GT
Predicted
GT
Predicted
GT
Predicted
Figure 15: Comparison between ground truth and generated videos by Vid2World in RECON Environment. The first four frames are provided context.
34

---

Published as a conference paper at ICLR 2026
D.4 GENERATION RESULTS OF REAL2SIM POLICY EVALUATION
We provide generation results of Vid2World in the Real2Sim Policy Evaluation task, as shown in Figure T6
5
F0
F-
20
2-
F30
1
t=0
RT-1-Begin
RT-1-15%
RT-I-Converged
1=10
15
1=
[=25
[=30
t=35
t=40
RT-I-Begin
RT-1-15%
RT-I-Converged
1
5
-
1-
20
25
t030
1=35
-00
RT-1-Begin
RT-1-15%
RT-I-Converged
Figure 16: Generation Results for Vid2World in Real2Sim Policy Evaluation experiments.
35

---

Published as a conference paper at ICLR 2026
D.5 IMPACT OF CAUSAL ACTION GUIDANCE SCALE ON GENERATION QUALITY
To further validate the efficacy of our proposed Causal Action Guidance mechanism, we analyze the impact of the guidance scale À on generation quality using 3D Game Simulation.
Setup. We randomly subsample a subset of 50 trajectories from the validation set in the CS:GO environment, and evaluate Vid2World's performance across a range of scales À e [1.0, 1.5, 2.0, 2.5, 3.5, 4.0), reporting the following four metrics: PSNR, SSIM, LPIPS, and DreamSim. Following the setup in Section 5.2 the video prediction metrics are calculated on the predicted frames, conditioned on four initial history frames given ground truth.
Results. As shown in Figure X we observe a consistent trend of improvement followed by degrada- tion with respect increasing guidance scale À in the generation quality across all evaluated metrics. This is aligned with our intuition of probability steering: While absence or insufficiency of guidance scale results in suboptimal metric scores due to poor action adherence, overshoot of guidance scale leads to over-sharpened distributions and visual artifacts. Through varying causal action guidance scale X, our mechanism offer the test-time flexibility of trading off action aligument with visual fidelity, leading to improved world modeling capabilities. These results, taken together with ablation studies in Section 5.4 (which confirm the necessity of guidance) and theoretical justifications in Appendix A..4 provide holistic validation of our method, spanning theoretical arguments, intuitive explanations and empirical validations.
D.6 LONG-HORIZON ROLLOUT AND ZERO-SHOV GENERALIZATION
To validate Vid2World's robustness temporal extrapolation, conduct a long-horizon rollout experiment in the CS:GO environment, Initialized with 9 ground truth frames, the model performs autoregressive generation conditioned on sliding window of the most recent observations. The action sequence is sampled uniformly from {w, A, S, D}, with each action held constant for 10 consecutive frames induce significant movement. We extend the rollout a total of 100 generated frames, significantly exceeding the training horizon of 16. As illustrated in Figure 17 while artifacts such as softened textures and drift spatial layout occur as the rollout horizon increases, generation maintains relatively high fidelity and physical realism, demonstrating strong robustness against error accumulation.
In addition to in-domain temporal extrapolation, we further evaluate the model's zero-shot general- ization on a completely out-of-distribution (OOD) dataset: the tactical shooter game Valorant (Riot Games 2020). Crucially, during the Vid2World training process, the model was exposed exclusively to GO:GO data. Therefore, any cross-domain generalization observed in this setting stems entirely from the robust visual priors preserved from the pre-trained video diffusion backbone during our transformation process. We perform autoregressive rollouts up to 50 frames. As shown in Figure 18 although the visual quality degrades much faster compared to the in-domain setting, the model surprisingly retains rudimentary capabilities of temporal consistency and action responsiveness. This zero-shot generalization experiment further validates that Vid2World enables interactivity while preserving and channeling the video diffusion model's 's inherent generalization capabilities.
36

---

Published as a conference paper at ICLR 2026
AAA TAEA
NAA PANMS
HAN WARD
HARE
WAR
HAH RORA
HAH SON
HAR CA
FARE
HATDEN
HAM
JAH WAY
HARER CAR
HAT
FAL
HARIE
TARDE WAYC
TAMEN
TATKER
TARKE
ANEY
PACAO
PARIE VPY
BA
AN
PACLE WIR
AAPLE
AAR PRT
BARENA
TARI
(a) Rollout Sample
Stal
(b) Rollout Sample
Figure 17: Temporal Extrapolation via Horizon Rollout (Part I). Initialized with 9 history frames, Vid2World autoregressively generates 100 frames conditioned on random action sequences in the CS:GO environment (over 6x the training horizon). Frame indices and corresponding actions are annotated at the start of each row. Zoom in for details. Continued on next page.
37

---

Published as a conference paper at ICLR 2026
(c) Rollout Sample
(d) Rollout Sample
Figure 17: Temporal Extrapolation via Long-Horizon Rollout (Part II). Initialized with 9 history frames, Vid2World autoregressively generates 100 frames conditioned on random action sequences in the CO:GO environment (over 6x the training horizon). Frame indices and corresponding actions are annotated at the start of each row. Zoom in for details. Continued on next page.
38

---

Published as a conference paper at ICLR 2026
(c) Rollout Sample
(f) Rollout Sample
Figure 17: Temporal Extrapolation via Horizon Rollout (Part III). Initialized with 9 history frames, Vid2World autoregressively generates 100 frames conditioned on random action sequences in the CS:GO environment (over 6x the training horizon). Frame indices and corresponding actions are annotated at the start of each row. Zoom in for details.
39

---

Published as a conference paper at ICLR 2026
(a) Cero-Shot Rollout Sample 1
(b) Zero-Shot Rollout Sample 2. Notice the movement the railing from Frame 21 to 30, moving right due the character moving left, showcosing action responsiveness.
Figure 18: Zero-Shot Generalization on Valorant (OOD). Despite being trained exclusively on CS:GO, Vid2World autoregressively generates 50-frame rollouts in the unseen game Valorant while suprisingly retaining rudimentary temporal consistency and action responsiveness, demonstrating the robust visual priors preserved from the pre-trained backbone. Zoom for details.
History Context
Action: Weapon
1-6
1-8
t-10
Figure 19: Capturing Environment Stochasticity. Conditioned on 4 history frames, World predict the future auto-regressive rollout. On timestep t=8, given the "switch weapon" action, the model generates the previously unseen weapon. Since the specific weapon is unobserved in history, the model produces diverse plausible outcomes with various shapes and colors across samples, effectively capturing the stochastic nature of the environment. Zoom in for details.
D.7 CAPTURING ENVIRONMENT STOCHASTICITY
To further investigate whether Vid2World captures the complex stochastic nature of the underlying dynamics, rather than merely predicting a single-modal distribution, we conduct a qualitative analysis in the domain of CS:GO. Specifically, fix the provided history (4 frames) and the future action
40

---

Published as a conference paper at ICLR 2026
H
[-
(-6
[-8
1-10
1-12
1-14
Move Forward (x+)
Move Backward (x-)
Move Left (y+)
Move Right (y-)
Lift Up (z+)
Press Down (x-)
Figure 20: Following Action Semantics. Conditioned on the same initial frame, Vid2World predicts diverging futures under different action conditions, faithfully following action semantics.
sequence (triggering "switch weapon" at 1 = 8), while sampling multiple trajectories with our Vid2World model. As illustrated in Figure 19 the model produces diverse plausible outcomes of the potential weapon, varying in shapes and colors. Since the secondary weapon is not visible in the conditioning context, the model is destined draw the weapon from multi-modal distribution. This diversity confirms that our model effectively learns the joint distribution of the data, capturing the inherent stochasticity of the environment rather than collapsing to single mode.
D.8 FOLLOWING ACTION SEMANTICS
To validate Vid2World genuinely follows action semanties, rather than predicting observatory trends based solely on history frames, we start from the same initial frame, sampling different action sequences for video prediction. addition validating this capability in the CS:GO environment (as shown in Figure 14), we conduct the experiment with similar spirits using the RT-1 environment.
Setup. We randomly sample an initial frame from the validation set, using it as the history context. Since in the RT-1 environment, the action space represents the end effector position of the robot arm, we therefore condition generation on six canonical action sequences: Move Forward, Move Backward, Move Left, Move Right, Lift Up, and Press Down, corresponding translations of the end effector along the x, y and z axes. Using the same configurations as Section 5.1 rollout trajectories until total number of 16 frames is reached.
we autoregressively
Results. As shown in Figure 20 Vid2World's predictions exhibit clear, directionally coherent end-effector motion that aligns with the semantics of the conditioned action sequence. Notably, the model correctly anticipates challenging phenomena such as end effector exiting the camera view (for Move Backward and Move Right) and physical contact with the table (for Press Down). Taken together with the action influence generated trajectory experiment in Figure [4 and the Real2Sim Policy Evaluation Experiment in Section 5.]] these results validate Vid2World's 's capability of generating future predictions that faithfully reflect the semantics of actions.
41

---

Published as a conference paper at ICLR 2026
Table Quantitative Results of Interactive Metries on CS:GO environment. After generating video predictions based on ground truth actions as well as random actions, we calculate the normalized delta metrics Av
M.. Best results are shown in bold.
Model
AN-FVD
Av-FID
Ax-SSIM
AN-LPIPS
AN-PSNR
Ax-Dreamsim J
Diamond- Fast
.48.43
-5.01
44.84
-20.95
52.94
14.16
Diamond-HO
-55.85
-14.76
47.04
-25.55
48.78
-53.5
Vid2World
-77.06
-29.44
22.40
-20.32
17.60
-41.11
D.9 QUANTITATIVE RESULTS ON INTERACTIVE METRICS
To quantitatively validate Vid2World's capability of performing counterfactural reasoning based on the action sequence, following Bruce et al. (2024), we evaluate this "action semantics following" capability by comparing the generation quality conditioned ground truth and conditioned random action input.
Metrics. We generalize the A,-PSN proposed by Bruce et al. (2024) to video generation met- rics, introducing normalized delta metrics v- for corresponding video generation metric M. Specifically, given the ground truth video sequence Xo:7, generated sequence conditioned on ground truth action Xo:T and generated sequence conditioned on random actions Ko.r, as well as a video generation metric M Rxxx x I + I. the metric defined as:
AN-M(XO.T. No.T, X = M(Xo.T: Xo.T) M(X X
M(xo.T, %
Intuitively, this metric measures how much the video generations differ when conditioned on ground truth action sequences compared to actions sampled from random distribution, normalized by the scale of the metric M. For metric M such that higher is better, AN-M also satisfies higher is better, as vice versa.
Setup. We evaluate our model's action following capability under the CS:GO environment. Fol- lowing Section 5.2 we initialize with history frames and corresponding ground truth actions, and predict on the validation set (500 trajectories) autoregressively until a total frame number of 16 is reached. For randomly sampled actions, we uniformly sample from all valid keyboard inputs. We report the calculated Delta Normalized Metrics A M. are shown in Table
Results and Discussion. As shown in Table Vid2World outperforms baseline methods in AN-FVD and AN-FID, showing Vid2World's capability of following action semantics. However, we notice clear trends generation quality degradation when sampling DIAMOND with random actions, whereas our model does not. Hence, we raise the concern of "hacking" the metric, where a model can acquire high quantitative performance on these delta normalized metrics by generating low-quality videos when conditioned random action distribution. This inability to distinguish action following capabilities with generation quality degrude under ood action distributions, contributes at least partially to Vid2World's 's limited performance these metrics. We anticipate futher work come up with better evaluation metrics for measuring interactivity capabilities.
D.10 THE ROLE OF LARGE-SCALLE PRETRAINING
In this section, focus a key question, highlighting the role of large-scale pretraining on this transfer method:
RQ Is the model's s success attributed to channeling visual priors from the pre-trained video diffusion model merely the novel architectural changes?
To ablate the role of large-scale video pretraining, randomly initialize the model parameters, following the exact architectural choice as Vid2World, and train the entire model from scratch based solely the RT-1 dataset, using 30k gradient steps for fair comparison with other ablated models.
42

---

Published as a conference paper at ICLR 2026
Table 5: Ablation study on the role of video pretraining: To validate Vid2World truly transfers priors from the pretrained video diffusion model, we train an additional model from scratch in the RT-1 environment, maintaining the exact architecture as Vid2World but randomly initializing the parameters. Best results in bold, worst in italics.
Model
WT
AG
FVD 4
FID
SSIM
LPIPS 4
PSNR
Vid2World Shift
29.9
7.85
0.799
0.185
21.5
Vid2World
Masked
29.4
7.07
0.824
0.169
22.9
Vid2World
Extrapolative
28.6
7.52
0.832
0.162
23.4
Vid2World
Masked
25.8
6.84
0.840
0.159
23,9
Vid2World
Extrapolative
22.4
6.16
0.839
0.159
23.9
From Scratch
Does not Apply
1768.8
469.4
0.065
0.8177
8.3
As shown in Table 3 even with the identical architectural and algorithmic designs, training the world model from scratch leads to significant performance drop compared to all other Vid2World models that utilize the pre-existing video diffusion model across all metries. This indicates that during Vid2World, the generation priors encapsulated in video diffusion model due to large-scale video pretraining truly transfers the transformed interactive world model.
D.11 EMPIRICAL VALIDATION OF CAUSAL WEIGHT TRANSFER
To validate the functioning causal weight transfer mechanism, conduct additional experiment, investigating whether representation similarity perserved during transformation. Specifically, we focus on the following research question:
RQ: Does Weight Transfer preserve representations close to the original video diffusion model?
Setup. To measure the similarity of the transformed feature our transformed world model and the original pretrained video diffusion model, utilize the metric of Cosine Similarity. Cosine Similarity measures the similarity between two non-zero vectors calculating the cosine of the angle between them, resulting in values in [-1, 1]. The higher the score, the stronger the correlation. We randomly sample 50 trajectories in the validation set of RT-1, passing the clean video sequence as input. Actions dropped for world models, and noise scale is set to zero across of frames. We measure the mean cosine similarity between latent representations extracted after the first convolutional layer of our transformed world model and the original pretrained video diffusion model. For all models, we employ training on 30k gradient steps, following ablation setup (Section 5.4). The results are shown Table
multi-dimennional space by
Table 6: Cosine similarity between features of transformed world models and original video diffusion model. Taken the same video sequence as input, measure the mean cosine similar of features taken after the first convolution layer both models. Values are scaled by 100 for better visualization. Best results shown in bold.
Model
WT
AG
Cosine Similarity 1
Vid2World
Shift
71.04
Vid2World Masked
71.09
Vid2World
Extrapolative
71.13
Results. As shown in Table all three weight transfer methods of Vid2World achieve high cosine similarity scores (>0.7), demonstrating that Vid2World preserves the representational structure the original video diffusion model. The extrapolative variant achieves the highest similarity, providing empirical evidence of functioning linear extrapolation, effectively repurposing generative video priors for interactive world modeling.
43

---

Published as a conference paper at ICLR 2026
E COMPUTATION COSTS
In this section, we provide a detailed breakdown of the computational costs required for Vid2World. As first step toward repurposing large-scale video diffusion models into interactive world models, our primary focus has been on establishing visual fidelity and physical realism rather than optimizing for inference latency.
Training Computation Costs. We conduct training on 4 Nvidia 40GB A100 GPUs, using the hyperparameters provided in T The configured training completes 100k gradient steps in 7 days, utilizing memory footprint of 37GB of GPU VRAM.
Inference Computation Costs. Since our model is capable of autoregressive generation conditioned flexible number of history frames and ddim sample steps, inference speed depends the configuration of such hyperparameters. We provide a realistic setup below: On single NVIDIA A100 (40GB) GPU, the model is configured with context window of 10 history frames and utilizes 50-step DDIM sampling with causal action guidance scale greater than 1. Under these settings, with batch size of 2, the model's inference latency is approximately 20.0 seconds per generated frame, with memory footprint of approximately 16 GB of GPU VRAM.
Potential Speedup Methods. While the current inference latency reflects the heavy computational cost of high-fidelity diffusion sampling, identify several promising avenues of speeding up inference: (a.) Decreasing the number of sampling steps; (b.) Conditioning on shorter history; (c.) Software optimizations for instance JIT compiling the model; (d.) Implementing (rolling) KV-Cache for autoregressive generation (Yin et al.) 2024 Huang et al., 2025). We anticipate further work that builds upon these insights, achieving real time action-conditioned video generation.
F
THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large Language Models (LLMs) were employed with the single purpose of polishing the writing of this manuscript. Their use was strictly limited to enhancing language quality, improving readability. and ensuring clarity throughout the paper. In particular, LLM assisted in rephrasing sentences, checking grammar, and identifying typos.
is important to note that LLMs were not involved the conception and realization of research ideas, the design of methodologies, the execution of experiments. All research conceptual development, methodological choices, and data analysis- were carried out independently by the authors. L.Mss role was strictly confined to linguistic refinement, without influencing the scientific substance the work.
The authors take full responsibility for the entire content of the manuscript, including text that was modified using LLM-based assistance. We have taken great care to ensure our use of LLMs adheres to ethical guidelines and does give rise to plagiarism or scientific misconduct.
44


Reviews:
[{"review_id": "n0emo9Uv2N", "reviewer": "Reviewer_ak6V", "summary": "This paper presents Vid2World, an approach to convert pre-trained video diffusion models into iteractive, action-conditioned world models. The core idea is to utilize the key physical knowledge already encoded in video generation model, and then converting bidirectional layers into casual/temporal layers.", "questions": "This work is build on top of bidirectional video diffusion models. This underlines the core methods around causal mask adaptation. Would it be possible to start from strong autoregressive video models without the need to repurpose from bidirectional ones to casual ones?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes an interesting solution to solve the problem of converting a non-causal video diffusion model into an autoregressive, interactive world model by manipulating the layers and further finetune the model.\n- By transferring priors from a pre-trained video diffusion model, Vid2World produces high-fidelity predictions that significantly outperform other world models on metrics like FVD and FID.\n- The approach is demonstrated across multiple, diverse domains.", "weaknesses": "- The base model is pre-trained on passive, action-free videos, which has strong prior that may be mismatching the goal of world models. This potential mismatch and its effect should have been discussed by the paper. \n- The converted autoregressive model may suffer from error accumulation. Converting a diffusion model to such an AR model may make this problem even more pronounced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWXruTRiBY", "reviewer": "Reviewer_mVqf", "summary": "The paper focuses on dealing with temporal causality in adapting pretrained video models to interactive world models. To enable cost-efficient adaptation from foundation video models, the paper explores video diffusion causalization by transferring the pretrained weights, and causal action guidance that injects frame-level action conditions independently. The experiments show some improvements in simulation quality compared to previous baselines on three difference benchmarks.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 1, "strengths": "S1) The proposed method is simple and effective. The writing is easy to follow.\n\nS2) I like the scope of the paper, which is trying to establish general world models for various domains through adapting from foundation video models.", "weaknesses": "W1) I believe the core contribution of this paper is how to causalizing temporal convolution layers. Unfortunately, the most recent video models (e.g., NVIDIA Cosmos and WAN, also mentioned by the authors) are typically using pure DiT architectures without temporal convolutions. Therefore, the main story of this paper is not applicable to those models, which somewhat diminishes the significance of the contribution.\n\nW2) Frame-level action conditioning is not new, and the paper misrepresents its difference from previous works (Line 259). Specifically, existing approaches such as IRASim and Cosmos-Predict-ActionCond already employ frame-level injection, offering the same functionality as the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iPRnzolhd2", "reviewer": "Reviewer_zQ4H", "summary": "The paper proposes a method to build world models from video diffusion models. It argues that this requires two key steps: (1) ‘causalization’ of the model (i.e., the current state should only depend on the past), and (2) ‘action conditioning’ (i.e., the possibility for the future to be influenced by a present action). Especially point (2) is nontrivial since most training data does not contain actions.", "questions": "The description of the method did not really give me an intuition why the action conditioning works. If the authors have a good intuition on this, then it would be helpful to share this.\n\nOne thing that confused me was the use of the world ‘causal’. In some places, this is restricted to time direction; however, the model also talks about counterfactuals and (at least implicitly) interventions, i.e., notions from a community where ‘causal’ has a much broader meaning. I think this should be clarified; otherwise, statements such as \"non-causal generative modeling could be inherently easier than its causal counterpart” are prone to misunderstanding. The best would be to add some explanation and discussing connections to “interventional/causal world models” in the causality community (e.g. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9363924, maybe also https://arxiv.org/pdf/2110.06562 ). \n\nGeneralization stress test (video game): Having a video of the following kind would be compelling: \nFor a video game, condition on a start frame, and show the time evolution for sequences of random actions (and mark the actions under the video). I’d like to see a long video, to understand where it starts breaking. Also, I would like to see this both (a) in the case where that game was included in the action-labelled part of the training set, or (b) it was not included (but potentially other similar games). I don’t expect it to work great - but I’d like to understand where it breaks.\n\nFinal question, Error accumulation (l1410): the authors argue that other models (DIAMOND, Fig 10) get blurry over time and Vid2World does not. I would like to understand this better. If there is inherent stochasticity in the dynamics (and I believe this is the case at least for some of the considered tasks), and you predict the mean, then things necessarily get blurry. If you don’t predict the mean (and I realize Vid2world does not), then things should sometimes go in the wrong direction. Of course the respawn example Fig 12 shows this, but it should also happen in less obvious cases?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The problem is relevant and interesting.\nThe paper is linguistically well written (in places, it sounds like LLM lingo).\nThe authors tackle the problem by applying a clever combination of methods, mostly from the literature. \nThe execution seems competent. \nThe empirical results look strong. \nI enjoyed reading the paper.", "weaknesses": "•\tThe method description does not provide clear intuition for why the action conditioning works; the paper could better convey the functioning of the model (beyond the mechanics).\n\t•\tThe use of the word “causal” is ambiguous—sometimes merely time-directional, while elsewhere invoking counterfactuals/interventions in a broader sense. This limits the potential audience of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5XrtxttCgf", "reviewer": "Reviewer_XNTH", "summary": "The paper proposes Vid2World, a framework that turns a single video into a 4D (3D + time) dynamic scene representation. Unlike typical video diffusion models that just generate future frames, this method tries to reconstruct a consistent, camera-aware world that can be rendered from novel viewpoints over time. Technically, the authors build on top of DynamiCrafter and modify it for causal, autoregressive video generation with action conditioning. They introduce weight-transfer tricks and causal attention masking so that the model can generate frames one by one while maintaining spatial-temporal coherence.", "questions": "see weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of turning raw videos into a dynamic 4D world is compelling. Most video models only hallucinate future frames from a fixed camera, so explicitly reconstructing a consistent world is an interesting step forward.\n\nI appreciate that they go beyond synthetic datasets and test on RT-1 real robot data and a large-scale human gameplay dataset (CS:GO). It shows they’re not only targeting clean, toy data.\n\nThe architectural changes to turn an offline video diffusion model into a causal and action-aware generator are reasonable. Especially, I think the “extrapolative weight transfer” idea for converting bidirectional attention layers into causal ones is practical.\n\nThe results (especially novel-view video rollout) are visually impressive and clearly better than standard video diffusion models that ignore camera motion.", "weaknesses": "No true 3D ground-truth evaluation.\nThe method claims to build a 4D world, but there’s no quantitative evaluation comparing it to 3D scene reconstruction methods (like NeRF, BANMo, DynamicNeRF, etc.). Most results are still evaluated only on video quality metrics or visuals. It's hard to tell if the “world” is actually geometrically consistent or just looks plausible.\n\nMostly relies on pre-trained models.\nA large part of the pipeline inherits DynamiCrafter. The core novelty is in causalization and conditioning, but I sometimes felt like the paper is more of a clever adaptation rather than a fundamentally new representation of dynamic scenes.\n\nAction conditioning is not strongly validated.\nThey claim the model can take actions and simulate their effects in the generated world, but I didn’t see strong evidence that the generated world actually follows the action semantics accurately, especially in RT-1 tasks. It feels closer to action-guided video synthesis than true action-driven world modeling.\n\nNo comparison to video-to-3D methods.\nWorks like Nerfies, Vid2NeRF, DynIBaR also reconstruct dynamic scenes from monocular videos. It would help to position Vid2World against those rather than only comparing to video diffusion baselines.\n\nComputation is heavy and unclear.\nTraining takes 4×A100 for 100k steps with 3 FPS video clips. The paper doesn’t report inference speed or memory usage. If the goal is a usable world model, some efficiency discussion would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rWI5oAFcjH", "reviewer": "Reviewer_Yc6M", "summary": "This paper proposes a straightforward yet effective method for adapting a pre-trained video diffusion model into an action-conditioned world model. The core contributions are two-fold: 1) \"Video Diffusion Causalization,\" which enforces temporal causality through a causal masked attention layer and a temporal convolution layer with a novel \"extrapolative weight transfer\" technique, and 2) \"Causal Action Injection,\" which integrates action-conditioning via causal guidance with dropouts. The authors conduct extensive experiments across diverse domains, including robotic manipulation (RT-1), game simulation (CS:GO), and real-world navigation (NWM), demonstrating that their proposed model achieves higher fidelity in video prediction compared to existing world models.", "questions": "- Figure 6: This figure presents a crucial comparison of auto-regressive generation capabilities. However, the motivation for why this specific comparison is necessary and what insights it provides is detailed in Appendix C.6 but absent from the main text. Briefly explaining this context in the main paper would significantly improve the figure's clarity and impact.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Simplicity and Effectiveness: The primary strength of this work lies in its simple and elegant approach. The proposed modifications to transform a standard video diffusion model into a world model are minimal, yet they prove to be highly effective across multiple challenging domains.\n- Broad Applicability: The method's effectiveness is convincingly demonstrated on a variety of environments, from simulated robotics and games to real-world navigation. This suggests the proposed techniques are general and widely applicable.\n- High-Fidelity Generation: The paper provides strong quantitative and qualitative results, showing that the model can generate high-fidelity, fine-grained, and action-conditioned video sequences that are visually superior to prior work.", "weaknesses": "- Insufficient Motivation for Extrapolative Weight Transfer: While the paper introduces \"extrapolative weight transfer\" as a key component, its theoretical and empirical motivation remains underdeveloped. The appendix (A.1) presents a simple counter-example involving a linearly increasing kernel, but this theoretical argument is not supported by empirical evidence. To strengthen this claim, the authors should provide experiments that verify the existence and prevalence of such temporal patterns in the learned kernels of baseline models, thereby justifying the need for their proposed solution.\n\n- Lack of Experiments Validating Utility as a World Model: The paper claims to build a \"world model,\" but the experiments primarily focus on video prediction fidelity (e.g., FVD, FID, SSIM). A true world model should be useful for downstream tasks like planning and control. The evaluation is missing crucial experiments, such as: (1) Reinforcement Learning Tasks: Can an RL agent be successfully trained using the learned model as the environment simulator? Demonstrating successful policy learning would provide compelling evidence of the model's utility and temporal consistency. If this is not feasible, a discussion of the model's limitations for such tasks is warranted. (2) Quantitative Interactivity Metrics: The current evaluation lacks metrics that specifically measure the model's fine-grained controllability and causal soundness in response to actions.\n\n- Ambiguity in the Role of Large-Scale Pre-training: The experimental setup creates a potential ambiguity. The proposed model leverages an internet-scale pre-trained video model, whereas the robotics and gaming experiments are conducted on domain-specific datasets. It is unclear which components of the model's success are attributable to the novel architectural changes versus the powerful, pre-trained backbone. An ablation study clarifying the benefits of internet-scale pre-training for domain-specific tasks like RT-1 or CS:GO would be highly valuable.\n\n- Incomplete Reporting of Metrics: For the real-world navigation (NWM) experiments, key fidelity metrics such as FVD, FID, and SSIM appear to be missing from the results tables. Including these would allow for a more direct and fair comparison with other models and experiments within the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)