Learning World Models for Interactive Video Generation
Authors: Taiye Chen
Xun Hu
Zihan Ding
Chi Jin
Venue: NeurIPS 2025 poster ()
Keywords: world model
diffusion model
video generation
Primary area: deep_learning

Abstract: Foundational world models must be both interactive and preserve spatialtemporal coherence to enable effective future planning with different action choices. However, present models for long video generation have limited inherent world modeling capabilities due to two main challenges: compounding errors and insufficient memory mechanisms. 
We enhance image-to-video models with interactive capabilities through additional action conditioning and autoregressive framework, and reveal that compounding error is inherently irreducible in autoregressive video generation, while insufficient memory mechanism leads to incoherence of world models. We propose video retrieval augmented generation (VRAG) with explicit global state conditioning, which significantly reduces long-term compounding errors and increases spatialtemporal consistency of video world models. 
In contrast, naive autoregressive generation with extended context windows and retrieval-augmented generation prove less effective for video generation, primarily due to the limited in-context learning capabilities of current video models. Our work illuminates the fundamental challenges in video world models and establishes a comprehensive benchmark for improving video generation models with internal world modeling capabilities.

Learning World Models for Interactive Video Generation
Taiye Chen' Xun Hu
Zihan Ding Chi Jin
Peking University
University of Oxford Princeton University yeyutaihan@stu. pku. edu. en {xh4421 zihand, chij//@princeton. edu
Abstract
Foundational world models must be both interactive and preserve spatiotemporal coherence for effective future planning with action choices. However, present mod- els for long video generation have limited inherent world modeling capabilitics due to main challenges: compounding errors and insufficient memory mechanisms. We enhance image-to-video models with interactive capabilities through additional action conditioning and autoregressive framework, and reveal that compounding error is inherently irreducible in autoregressive video generation, while insufficient memory mechanism leads to incoherence of world models. We propose video retrieval augmented generation (VRAG) with explicit global state conditioning, which significantly reduces long-term compounding errors and increases spatiotem- poral consistency of world models. In contrast, naive autoregressive generation with extended context windows and rerieval-augmenter generation prove less effective for video generation, primarily due to the limited in-context learning capa- bilities of current video models. Our work illuminates the fundamental challenges in video world models and establishes comprehensive benchmark for improving video generation models with internal world modeling capabilities.
Project page: https://sites www.:/e.s.goolll..c///ie/eeee cow/viww/vrag
Introduction
Foundational world models capable of simulat- ing future outcomes based on different actions are crucial for effective planning and decision- making [T] 2 3.. To achieve this, these mod- els must exhibit both interactivity, allowing for action conditioning, and spatiotemporal consis- tency over long horizons. While recent advance- ments in video generation, particularly diffu- sion models 4
77 have shown promise, extending them to generate long, interactive, and consistent videos remains a significant chal-
lenge 8 9 10].
Figure I: A world model possesses memory capa- bilities and enables faithful long-term future predic- tion by maintaining awareness of its environment and generating predictions based on the current state and actions. Example is in Minecraft game.
Autoregressive approaches which generate videos frame by frame chunk by chunk conditioned on previous outputs, are
Equal contribution.
Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

natural fit for modeling long temporal dependencies and incorporating interactivity. However, these methods face significant challenges stemming from two fundamental, often coupled. limitations: compounding errors and insufficient memory mechanisms. Compounding emror arise as small inaccuracies early predictions accumulate over time, leading to significant divergence from plau- sible future states. Our analysis suggests this may be inherent to current autoregressive paradigma gms. Insufficient memory mechanisms hinder the models' ability to maintain consistent object identities, spatial layouts, and world states over extended durations, resulting in inconsistent world models. These two issues often exacerbate one another, making long-term consistent generation difficult.
Inspired by the success of large language models (LLMs) [13 [6] in handling long sequences, we investigate analogous techniques for video generation. Extending the context window, while potentially alleviating compounding errors to some degree, introduces substantial computational and memory overhead. More critically, we find that unlike L.LMs, current video generation models exhibit weaker in-context learning capabilities, making longer context less effective in resolving fundamental consistency issues. Similarly, retrieval-augmented generation (RAG) [7 [8 a powerful technique for incorporating external knowledge in LLMs, shows limited benefits in our experiments with video models. Neither static retrieval with heuristic sampling nor dynamic retrieval based on similarity search significantly improved world model consistency.
These findings suggest that implicitly learning world consistency solely from autoregressive prediction on pixel or latent representations is insufficient. We argue that explicit global state conditioning necessary. Incorporating explicit representations like world maps, object states, or coordinate systems as conditioning information could provide the necessary grounding for generating consistent long-term interactive simulations.
Furthermore, evaluating the specific failure modes of long video generation demands appropriate metrics. Existing metrics often conflate the distinct issues of compounding errors and long-term consistency (memory faithfulness), providing a coupled assessment that obscures the underlying problems. To enable clearer analysis, we advocate for and introduce decoupled evaluation strategy by separately quantify the severity of compounding errors and the faithfulness of memory retrieval in long interactive video generation.
Our main contributions are: (1). We systematically decouple and analyze the challenges of com- pounding errors and insufficient memory in autoregressive video generation for interactive world modeling. (2). We propose video retrieval augmented generation (VRAG) with explicit global state conditioning, which significantly improves long-term spatiotemporal coherence and reduces compounding errors for interactive video generation. (3). We conduct a comprehensive comparison with various long-context methods adapted from LLM techniques, including position interpolation, neural memory augmentation, and historical frame retrieval, demonstrating their limited effectiveness due to the inherent weak in-context learning capabilities of video diffusion models. This work sheds light on the fundamental obstacles in building consistent, interactive video world models and provides a benchmark and evaluation framework for future research in this direction.
Related Works
Video Diffusion Models Diffusion generative modeling has significantly advanced the fields of image and video generation [19 12 20 21 22
27 28 29]. Latent video diffusion models [2 operate on video tokens within a latent space derived from a variational auto-encoder
23]
24
25
26
(VAE) 300, building upon prior work in latent image diffusion models BTI. The Diffusion Trans-
former (DiT) 322 introduced the Transformer
333 backbone as an alternative to the previously 23 21 22 in diffusion models.
prevalent U-Net architecture
Long Video Generation Autoregressive video generation 12 13 14
36
38
39
represents a natural approach for long video synthesis by conditioning on preceding frames, drawing inspiration from successes in large language models. This can be implemented using techniques such as masked conditional video diffusion 40 34 or Diffusion Forcing [1). Diffusion Foreing
introduces varying levels of random noise per frame to facilitate autoregressive generation conditioned
on frames at inference time. Furthermore, the autoregressive framework naturally supports interactive world simulation by allowing action inputs at each step to influence future predictions. Nevertheless,

---

compounding errors remain a significant challenge in long video generation, particularly within the autoregressive paradigm, as will be discussed subsequently.
2 B are simulation systems designed to predict future trajectories based on the current state and chosen actions. Diffusion-bases world models 42 43 [0] facilitate the modeling of high-dimennional distributions, enabling high-fidelity prediction of diverse trajectories, even directly in pixel space. The Sora model [ introduced the concept of leveraging video generation models as world simulators. Extending video generation models with interactive capabilities has led to promising applications diverse domains, including game simulation like Genie [9] GameNGen [TO]. Oasis [44), Gamegen-x 45]. The Matrix [37],
Interactive Video World Models World models 1
Mineworld [46], GameFactory [47] and so on [43] autonomous driving (48) robotic manipula-
tion [35 49 and navigation [50]. While existing work interactive video world models has
made significant engineering advances, there remains a notable gap in systematically analyzing and addressing the fundamental challenges underlying long-term consistency and compounding errors.
lack of spatiotemporal consistency a primary bottleneck for developing internal world models using current video generation techniques. One line of research addressing this involves predicting the underlying 3D world structure like Genie2 [51], Aether [52), Gen3C (53] and others [54 55 56];
however, these approaches often suffer from lower resolution compared to direct video generation due to the complexity of 3D representations, exhibit limited interaction capabilities, and typically operate only within localized regions. Consequently,
work focuses enhancing the consistency of video-based world models [10] 34 57]. SlowFast- (34] employs dual-speed learning system to progressively trained LoRA modules for memory recall, utilizing semantic actions but offering limited interactivity. Concurrent work [57] explores interactive world simulation through the
integration of supplementary memory blocks.
3 Methodology
3.1 Preliminary: Latent Video Diffusion Model
Video diffusion models have emerged powerful framework for video generation. We adopt latent video diffusion model [21] that operates in a compressed latent space rather than pixel space for computational efficiency. Specifically, given an input video sequence R we first encode it into a latent representation = = ((x) using a pretrained variational autoencoder (VAE). The forward process gradually adds Gaussian noise the latent according variance schedule { T
g<z1|z=11) Nzzi VI B, 3,1)
The model learns to reverse this process by predicting the noise at each step:
L E,,,,=lle - e0 (z., 11)
(2)
where 2, = varzo + /1 ä
a,e with e ~ (0,
At inference time, we can sample new videos by starting from random noise zy ~ NN(0,I and iteratively denoising:
1 Vas
3e co( + orE
h6-1
(z:
VI a4
where 02 = - 3g and ā₂ = II ag. The final latent sequence zu is decoded back to pixel space using the decoder D obtain the generated video.
3.2 Interactive Long Video Generation
To enable interactive long video generation conditioned on action sequences, we augment the base diffusion model with two techniques: (I) additional action condition with adaptive layer normalization (AdaLN), and (2) random frame noise for autoregressive modeling, as shown diagram Fig.
Action Conditioning To enable interactive video generation conditioned on action sequences, we augment the base diffusion model with adaptive layer normalization (AdaI.N). Given an action sequence e 4xA where A is the action dimension, we first embed it into latent space using
3

---

Diffusion Loss
Hietorlool Context
Buffer (FIFO)
AdaLN
Frome
Action
DiT Block
State
Causal Mask
Similarity Search
AdaLN
Retrieval
DIT Block
Autoregreasive Inference
Historical Context
Current Chunk
Add Frams
Retrieved Frames
Current Chunk
Nose
Law Noise
Random Maise
Video Retrieval Augmented Generation
Figure 2: Overview of our VRAG framework for interactive video generation. The framework incorporates global state conditioning and memory retrieval mechanisms to ensure spatiotemporal consistency and mitigate eтror accumulation. During both training and inference, retrieved memory serves as context for joint self-attention spatiotemporal DiT blocks. The model employs per- cer-frame noise injection during training to facilitate autoregressive sampling inference time.
learnable embedding layer: e, Embed(a) RLxD. where D. is the embedding dimension. For each normalization layer in the diffusion model, we learn action-dependent scale and shift parameters through linear projections: 7h e,W, b, E RL ,3a e, Wa by RLxD, where D₁ matches the hidden dimension of the feature maps. We have (h) = C LayerNorm(h) + Pa. where h E IR 1x
represents the intermediate feature maps and @ denotes dot production.
Autoregressive Video Generation To enable long video generation, we adopt an autoregressive approach where generate frames sequentially. At each step, condition fixed-length context window Le of previously generated frames. However, naive autoregressive generation with teacher forcing can suffer from large compounding errors where mistakes accumulate over time. We apply the Diffusion Forcing [1 technique during training.
Specifically, during training, we randomly add noise to each frame in the entire input video sequence according to the diffusion schedule: z V VI- Oytt , (O, I), where z represents noised latent of the i-th frame. This forces the model to be robust noise in the conditioning frames and prevents it from relying too heavily on the context. With above two techniques, the training objective for action-conditioned autoregressive video models become:
DFF = [t),,c,z.allle
ca (z( [t) 2))
E(c] = (() Z = (2
(4)
where [t] is vector of T. timesteps with different / E [7] for each frame. The noise prediction model Eg conditioned on both the action sequence and noised frames 24-
Architecture We apply diffusion transformer (DiT) for video generation modeling. We adopt spatiotemporal DiT block with separate spatial and temporal attention modules. Rotary Position Embedding (RoPE) (58) is applied for both attention modules, and temporal attention is implemented with causal masking.
3.3 Retrieval Augmented Video World Model with Global State
While the vanilla model in Sec. 3.2 provides a foundation for interactive video generation, it lacks robust mechanisms for maintaining long-term consistency and world model coherence. To address these limitations, we integrate memory retrieval and context enhancement with inspiration from L.Mss, and incorporate video-specific approaches such as historical frame buffer and global state con- ditioning. These enhancements enable more consistent and coherent autoregressive video generation by providing the model with better access to historical context and spatial awareness.

---

Global State Conditioning To enhance spatial consistency in video generation, we incorporate global state informationn sppeifically the character's current coordinates and pose-as as an additional conditioning signal. The global state vector E R consists of two key components: Spoc representing 3D position coordinates and Soii capturing orientation angles. Given an action sequence a E RL and the global state sequence E TRL2S both are transformed by a learnable embedding layer, e, Embed, (a, 8), produce conditioning features. These features are then fed into AdaLN layers within the diffusion model. This mechanism allows the model to modulate its generation process, adapting to both the input actions and the character's spatial context, thereby improving overall coherence.
Video Retrieval Augmented Generation (VRAG) Beyond global state conditioning, we propose memory retrieval augmented generation to enhance the model's 's ability to leverage historical context while maintaining temporal coherence, namely video retrieval augmented generation (VRAG). For VRAG, we combine the concatenated historical and current frames with their corresponding action sequences RLXA and global state sequences model. The historical frames are retrieved from fixed-length buffer B, which stores previously generated frames. The per-frame retrieval process is based on heuristic sampling strategy, where we select the most relevant historical frames based on similarity search to concatenate with the current context. The similarity score based on global state is defined as:
|Shist; s] E RLLSS as conditional inputs to the
r ($) = faim (3 O w, SL-1 O & B
(5)
where fsim is a distance metric (e.g.. Euclidean distance) between the history frame and the last frame is weight vector that modulates the importance of different state
to be predicted $1-1, and w $ components. The top Lt, most similar historical states and frames are selected and sorted to form the retrieved context. Unlike RAG in L.LMs which leverages strong in-context learning capabilities, video diffusion models exhibit weak in-context learning abilities, making direct inference with historical frames as context ineffective, demonstrated later in our experiments. To address this limitation, we propose VRAG training with key modifications to the standard RAG approach, enabling effective memory-auummnnted video generation.
During training, we retrieve historical frames Zuist E RRA xD and concatenate them with the current context window E x to form the extended context à [Zuist z). For effective VRAG, we make several key modifications: (1). To distinguish retrieved frames from normal context frames, modify the RoPE embeddings by adding temporal offset AL to the retrieved frames' position indices. (2). Additionally, we apply lower noise levels 3r < Be to the retrieved frames Zhis to simulate partially denoised historical frames during inference. This enhances the robustness of the model with imperfect historical frames generated previously during the autoregressive process. The model is trained denoise for the entire context 2 including both retrieved and current frames. (3). To ensure the model focuses on denoising the current context while leveraging historical information, (4). Furthermore, for retrieved frames, we only condition on their global states Suist E IR [ S masking out action conditions Anist E R x to avoid temporal discontinuity in action sequences. This selective conditioning approach helps maintain spatial consistency while preventing action-related artifacts from propagating through the generation process. Overall, the training objective of VRAG on diffusion models is defined as:
we mask the diffusion loss LDF for retrieved frames.
LVRAG
- EB (ā i, ā, 8) 3) O m),
(6)
zi = (Ziisi (t)) [1]],, à =I
$ = [Smit: s).
m Oz., 1
where f is concatenation of [t'] and [t], with t' < and E
3.4 Long-context Extension Baselines
To investigate whether established long-context extension techniques from LLMs can effectively enhance video generation models, we design three complementary approaches that leverage either explicit frame context or neural memory hidden states, based on vanilla models in Sec. 33.2 These methods serve as baseline comparisons to our main approach, specifically targeting the model's ability to maintain spatial coherence and temporal consistency in long video generation. Through these baselines, we aim to verify the in-context learning capabilities of video diffusion models and assess their effectiveness in handling extended sequences.
5

---

Long-context Enhancement We extend the temporal context window using YaRN [59 modifica- tion for RoPE in temporal attention. RoPE encodes relative positions via complex-valued rotations. where the inner product between query 9m and key k,, depends on relative distance (m n). YaRN extends the context window by applying a frequency transformation to the rotary position embed- dings. This transformation scales the rotation angles in way that preserves the relative positioning information while allowing the model to handle longer video sequences, after small-scale fine-tuning on longer video clips.
Frame Retrieval from History Buffer We implement fixed-length buffer B storing historical latent frames with a heuristic sampling strategy. The buffer is partitioned into Ns = 5 exponentially decreasing segments Gj, where Lj = L1 a From each segment G5, we sample is frames to form subset F). The retrieved memory Zmem = [F...... FNs] is concatenated with current frame window as additional context:
Zzmem z]. which then passed into the spatiotemporal DiT blocks. This design ensures higher sampling density for recent frames, emphasizing recent visual information while maintaining access to historical context for temporal consistency.
Neural Memory Augmented Attention Instead of using explicit frames as context in above two methods, explore a neural memory mechanism to store and retrieve hidden states. This approach is inspired by the success of Infini-attention [60] in LLMs, which utilizes a compressed memory representation to enhance attention mechanisms. The model processes video in overlapping segments to maintain temporal continuity. For each video segment 2g. we compute query qs, key k, and value V, matrices. The model retrieves hidden state Amem from compressive memory M,-1'
Amem = aqq. MM Memory M,-1 and normalization vector n,-1 are then updated. The final a(q, mm
attention output combines retrieved hidden state Amem and standard attention using learnable gating maintain visual consistency across the long video sequence.
As another baseline, we follow the Frame Pack [6] to compress historical frames as context. Three input compression kernels with different kernel sizes-(2, 4, 4), (4, 8, 8), and (8. 16, 16)- are employed to condense the historical frames into a fixed-length context. This approach essentially achieves frame compression through importance sampling with recency bias, which enables larger field of view while maintaining lower computational costs. However, the prioritization of most recent frames can be suboptimal in many cases for long video generation especially when considering the memory issue. Our VRAG based on frame relevance provides theoretically better historical information retrieval. Moreover, our method is actually orthogonal to the frame compression technique in Frame Pack. We leave the combined methods as future work.
Frame Pack
More details of the above methods can be found in the supplementary material.
4 Experiments
4.1 Datasets and Evaluation Protocol
For training, we collected 1000 long Minecraft gameplay videos (17 hours total) using MineRL 62 All videos have fixed resolution of 640x360 pixels. Each sequence spans 1200 frames, annotated with action vectors (forward/backkwrrd movement, jumping, camera rotation) and world coordinates (x, y, z positions and yaw angle).
For evaluation, we assembled two distinct test sets: (1) for compounding error evaluation, we use 20 long videos of 1200 frames with randomized actions and locations, and (2) for world coherence, we use 60 carefully curated 300-frame video sequences designed to systematically assess spatiotemporal consistency. These curated sequences feature controlled motion patterns including in-place rotation, direction reversal, and circular trajectory following. The first 100 frames cach sequence serve as initialization buffer for methods requiring buffer frames or are excluded from evaluation for others. Each model autoregressively generates next single frame with stride until the desired length.
We evaluate the models against ground-truth test sets using several metrics: Structural Similarity Index (SSIM) (63) to measure spatial consistency, Peak Signal-to-Noise Ratio (PSNR) for pixel- level reconstruction quality, Learned Perceptual Image Patch Similarity (LPIPS) 64 to assess perceptual similarity. For the compounding error evaluation, we find SSIM more accurately reflect the faithfulness of frames over long sequences.

---

VRAG
GT
VRAG
GT
VRAG
GT
Figure Visual comparison of VRAG with ground truth videos on world coherence evaluation. With
100 initial frames as history buffer, VRAG predicts 200 subsequent frames.
4.2 Training Details
A consistent window size of 20 frames is applied for both model training and evaluation for fair compar- ison. For vanilla Diffu- sion Forcing, we addition- ally train a variant with window sizes of 10 frame for context length evalu- ation. For our VRAG method, we combine 10 re- trieved frames with 10 cur-
DF20
10RN
Homory Btrner Memorr
rent frames for both train-
ing and inference. We rep- resent the agent's s state us- ing global state vector (r. y, 2, yaw] during train- ing, which can be extended to incorporate
full 3D pose representation when needed. To facilitate train- ing convergence, these val- ues are normalized relative to the initial state, thereby reducing the complexity of the diffusion process. The YaRN implementation extends the vanilla model (window size 20) by replacing position embeddings with YaRN and stretching factor 4, fol- lowed by fine-tuning for 10 steps on 80- 80-frame sequences. During evaluation of Yarn, we use 40-frame window. The Infini-attention with neural memory employs sliding window size 20 and stride 10, using the first 10 frames for memory state updates and the last 10 for local attention com- putation. The History Buffer method maintains a 124-frame buffer partitioned into 5 exponentially decreasing segments (L, = 2, 2), sampling 2 frames per segment to form 10 historical frames that are concatenated with the current frames. All models are trained for 3 epochs on the dataset, with batch size of 32 across 8 1.00 GPUs.
Figure 4: Visual comparison of different methods, evaluated for world coherence analysis.
7

---

4.3 World Coherence Results
We investigate the spatiotemporal consistency of internal world models by evaluating the predicted videos given initial frames and action sequences. As visualized in Fig. our VRAG provides an effective approach to enhance the model's ability to leverage historical context for improving world coherence. Fig- shows more visual comparison of VRAG with ground truth videos. We evaluate the world coherence of different methods using multiple metrics. Fig. 5 shows the SSIM scores over time, while Tab. [ presents comprchensive comparison across all metrics. Our VRAG method achieves the best performance across all metrics, demonstrating its superior ability to maintain world coherence in generated videos. Our experimental results demonstrate that expanding the window size from 10 to 20 frames in the baseline DF model improves world coherence, indicating that longer context windows enhance consistency. However, further context extension using YaRN shows improvement over the vanilla DF model. This suggests that YaRN's ' context extension capabilitiess while effective in language models, do not transfer effectively video generation for maintaining world coherence. Similarly, the History Buffer method fails to effectively utilize historical frames for spatiotemporal consistency without explicit in-context training. These findings from both YaRN and History Buffer approaches reveal that video diffusion models at the current scale possess limited in-context learning capabilities, preventing them from effectively leveraging historical frames for maintaining long-term consistency. The Neural Memory method performs poorly due to its instability in model training.
DFtwindow Offeindow
Nerwyy
Method
SSIM 1
PSNR 1
LPIPS !
0.55
DF (window 10)
0.455
16.161
0.509
0.50
DF (window 20)
0.466
16.643
0.538
0.45
60
0.40
YaRN
0.462
16.567
0.532
0.35
History Buffer
0.459
16.922
0.543
0.30
Frame Pack
0.421
16.372
0.574
0.25
VRAG
0.506
17.097
0,506
125
150
175
200
225
250
275
300
Frame Index
Table 1: Quantitative comparison of world co-
Figure 5: SSIM scores over time for different meth- herence across different methods, evaluated on ods on world coherence evaluation.
videos with 300 frames.
4.4 Compounding Error Results
We evaluate the compound- ing error in long video generation across different methods using the SSIM metric. As shown in Fig. 7 and Tab. 2 our VRAG method achieves superior performance with an SSIM score of 0.349, demonstrat- ing better structural similar- ity preservation compared to baseline methods. In- creasing the window size in DF from 10 to 20 frames improves SSIM, indicating that longer context helps mitigate compounding er- rors. However, this im- provement still inferior
Hastory Itee
Hame 200
Rame 500
Fame 101
Home L100
Figure 6: Visual comparison of long-term video prediction (1200 frames) across different methods, evaluated for compounding error analysis.
VRAG's performance, suggesting that our retrieval- augmented approach provides more effective long-term consistency. As visualized in Fig. [ our VRAG method generates more coherent and consistent frames over long sequences, while other methods exhibit noticeable artifacts and incon- sistencies. The History Buffer method performs poorly, with an SSIM score of 0.188, indicating that naive historical frame retrieval without effective in-context training fails to maintain fong-term

---

consistency. Given its limited performance in the world coherence experiments (Sec. the Neural Memory method from visualization this longer video prediction visualization.
4.3 we exclude
0.5
Method
SSIM
0.3
DF (window 10)
0.297
SSIM
0.2
DF (window 20)
0.321
0.1
YaRN
0.316
VRGG
History Buffer
0.188
-0.1
Hittyry Bu
Neural Memory
0.283
Naural Demmryy
VRAG
0.349
1100
Frame Index
Table 2: Average SSIM scores across all frames in compounding emoor evaluation
Figure 7: SSIM scores over time for compound-
ing error evaluation
4.5 VBench Evaluation
We evaluate the long video generation with five Video Quality metrics in VBench [65] for generated videos in Sec. 4.4 The evaluation results on VBench (higher better) are shown in the Tab As demonstrated in the results, temporal quality and video frame quality. The Neural Memory baseline has Aesthetic Quality 0.343 and Imaging Quality score 0.3597 respectively, which significantly lower than other baselines, therefore not listed here. Our VRAG shows better temporal consistency compared with all baseline methods, and the high video frame quality indicates the results are not over-smoothed.
method outperforms all other baselines across all metrics in both
4.6 Extension: Real World Setting
Background Consistency
Temporal Flickering
Motion
Aesthetic Quality
Imaging
Method
Metric
DFoT
VRAG
Smoothness
Quality
SSIM
0.4436
0.9116
DF20
0.9668
0.9485
0,9582
0.5272
0.6058
PSNR
13.03
32.21
YaRN
0.9686
0.9401
0,9523
0.5252
0.6323
LPIPS +
0.4469
0.1146
History Buffer
0.9664
0.9475
0.9579
0.5167
0.6253
FVD 4
337.5
221
VRAG
0.9686
0.9511
0.9603
0.5295
0.6444
Table 4: Quantitative com-
Table 3: Evaluation results on five Video Quality metrics in VBench.
parison on RealEstatel dataset.
We conduct additional experiments in real-world setting beyond Minecraft simulation to show generalization of our approach. Specifically, following the experimental setup of Diffusion Foreing Transformer (DFoT) 666. our VRAG model is initialized from pre-
DFoT
trained DFoT and finetuned on the
VRAG
RealEsttae 10K dataset [67] with ad-
ditionally retrieved historical context,
as described in Sec. 3.3
After fine-tuning for just 2 epochs (10% of the original training steps), our method significantly outperforms the DFoT baseline in terms of mem- orization capability. The visualized frames are presented in Fig. 8 and quantitative results are summarized in Tab. 4 The results effectively demon- strate the generalization of our ap- proach beyond Minecraft for solving the memory issue in long video prediction.
Figure 8: Visualized video frames on RealEstate 10K dataset. Red blocks indicate the ground-truth frames.

---

4.7
Ablation: Memory and Training of VRAG
Method
SSIM 1
PSNR
LPIPS
VRAG
0.55
0.50
VRAG (no training!
VRAG
0.506
17.097
0.506
3 0.45
VRAG (no memory!
VRAG (no training) 0,455
16.670
0.528
9 0.440
VRAG (no memory) 0.436
16.372
0.547
0.35
0.30
0.25
Table 5: Ablation study of VRAG components. We compare the full model with variants that remove ei- ther the memory component (additional global state conditioning only) or training component (in-context learning only).
100
125
150
175
200
225
275
Frame Index
Figure 9: Comparison of SSIM scores over time for VRAG variants.
We ablate the key designs for VRAG methods, including the memory and training components. The ablation results shown in Fig. 2 and Tab. [] We compare the full VRAG model with two variants: (1) VRAG without the memory component, which only uses additional global state conditioning, and (2) VRAG without the training component, i.e., vanilla model with retrieval augmented generation for in-context learning at inference. The ablation study is conducted on the world coherence evaluation dataset.
The ablation results reveal several key insights about VRAG components. First, removing the mem- ory component leads to the largest performance drop across all metries, with SSIM decreasing by 13.8% and LPIPS increasing by 8.1 1% This demonstrates that the memory mechanism is crucial for maintaining spatiotemporal consistency and quality. Second, removing the training component also causes significant degradation, with SSIM dropping by 10.1% and LPIPS increasing by 4.3%, high- lighting the weak capabilities of in-context learning for current video models. The full VRAG model achieves the best performance across all metrics, showing that both components work synergistically improve video generation quality.
Conclusions and Discussions
In conclusion, VRAG tackles the fundamental challenge of maintaining long-term consistency in interactive video world models through an innovative combination of memory eetrieval-augmente generation and global state conditioning. By maintaining a buffer of past frames associated with spatial information, VRAG effectively recalls relevant context and preserves coherent dynamics across se extended sequences. Its memory mechanism with explicit in-context training process substantially mitigates compounding errors and improves spatiotemporal consistency. Extensive experiments long-horizon interactive tasks demonstrate the superior performance of VRAG over both long-context and cemory-based basclines, establishing scalable framework for faithful video-based world modeling. Notably, we discovered that context enhancement techniques from LLMs fail to transfer effectively the video generation domain, even with shared transformer backbones, due to the inherent limitations of in-context learning capabilities for video models. This finding underscores the critical importance of VRAG's in-context training approach. We hope our work will inspire further exploration into memory retrieval mechanisms for long video generation and interactive simulation.
Limitaitons. We acknowledge the current computational limitations preventing effective scaling to longer sequences or larger architectures. GPU memory constraints severely restricted memory buffer size and training sequence length, potentially impacting long- morizon consistency and model performance. The higher computational cost of memory eetrival--uugmente generation may further limit deployment in susource-constraiing settings such edge devices. Future work could explore more efficient memory mechanisms, adaptive optimization strategies, and hardware-awar algorithms.
Broader Impacts. We acknowledge serious ethical concerns regarding the potential misuse of such technology for creating highly convincing misleading or manipulated video content in games or simulation systems. We strongly encourage responsible development and deployment of video generation technologies, with appropriate technical and ethical safeguards, clear accountability frameworks, and transparency measures in place to mitigate risks.
10

---

References
[1] Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
[2] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Ad- vances in neural information processing systems, 31, 2018.
[3] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02199. 2020.
[4] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning pages 2256 2256-2265, 2015.
[5] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[6] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[7] Yang Song. Jascha Dock-Diccsttin, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[8] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024.
[9] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
[10] Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models real-time game engines. arXiv preprint arXiv:2408. /4837, 2024.
[11] Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprint 1906v 2019.
[12] William Harvey, Søren Nørskov, Niklas Kölch, and George Vogiatzis. Flexible diffusion modeling of long videos. arXiv preprint arXiv:2205. //495, 2022.
[13] Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng. Xun Guo, Jinyu Li, Hefei Ling, and Furu Wei. Arlon: Boosting diffusion transformers with autoregressive models for long video generation. arXiv preprint arXiv:2410.20502 2024.
[14] Desai Xie, Zhan Xu, Yicong Hong, Hao Tan, Difan Liu, Feng Liu, Arie Kaufman, and Yang Zhou. Progressive autoregressive video diffusion models. arXiv preprint arXiv:2410.0815,, 2024.
[15] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad. Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[16] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[17] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-auge -aggmeetee generation for large language models: A survey, arXiv preprint 4xX1v:2312.10997 2:1, 2023.

---

[18] Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui. sorieval-augmente generation for ai-generated content: A survey. arXiv preprint arXiv:2402. 19473, 2024.
[19] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22553-2557, 2023.
[20] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7312-7322, 2023.
[21] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311. 2023,
[22] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2310-7320, 2024.
[23] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03455 2022.
[24] Uriel Singer, Adam Polyak, Eliya Nachmani, Guy Dahan, Eli Shechtman, and Haggai Ha- cohen. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.1477 14792, 2022.
[25] Yu Hong, Jing Wei, Xing Liu, Xiaodi Wang, Yutong Bai, Haitao Li, Ming Zhang, and Hao Xu. Cogvideo: scale pretraining for text-to-video generation with transformers. arXiv preprint arXiv:2205. 15868 2022.
[26] Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408.06677 06072, 2024.
[27] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to- texteoo-video technical report. arXiv preprint 2rxiv:2308.0657/ 2023.
[28) Zihan Ding, Chi Jin, Difan Liu, Haitian Zheng, Krishna Kumar Singh, Qiang Zhang, Yan Kang, Zhe Lin, and Yuchen Liu. Dollar: Few-step video generation via distillation and latent reward optimization. arXiv preprint arXiv:2412.. 5689, 2024.
[29] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient vidco production for all. arXiv preprint arXiv:2412.20404 2024.
[30] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695. 2022.
[32] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[34] Yining Hong, Beide Liu, Maxine WI, Yuanhao Zhai, Kai-Wei Chang, Linjie Li, Kevin Lin, Chung-Ching Lin, Jianfeng Wang, Zhengyuan Yang, et al. Slowfast-vgen: Slow-fast learning for action-driven long video generation. arXiv preprint
12

---

[35] Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. Advances in Neural Information Processing Systems, 37:68082-681 19, 2024.
[36] Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han. Fifo-diffusion: Generating infinite videos from text without training. arXiv preprint arXiv:2405. 1473, 2024.
[37] Ruili Feng, Han Zhang, Zhantao Yang. Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The matrix: Infinite- world generation with real-time moving control. arXiv preprint arXiv:2412.03568, 2024.
[38] Sand-AL. Magi-1: Autoregressive video generation at scale, 2025.
[39] Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tade- vosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403.14773, 2024.
[40] Vikram Voleti, Alexia Jolicoeur-Mariinn Martineau, and Chris Pal. Mevd-masked conditional video dif- fusion for prediction, generation, and interpolation. Advances in neural information processing systems, 35:23371-23385 2022.
[41] Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081-24125,, 2024.
[42] Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model: Fu- ture modeling beyond step-by-step rollout for offline reinforcement learning. arXiv preprint arXiv:2402.03550, 2024.
[43] Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-587911 2024.
[44] Decart, Etched, Julian Quevedo, Quinn McIntyre. Spruce Campbell, Xinlei Chen, and Robert Wachen. Oasis: A universe transformer. 2024.
[45] Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation. arXiv preprint ArXiv:2411.00700 2024.
[46] Junliang Guo, Yang Ye, Tianyu He, Haoyu Wu, Yushu Jiang, Tim Pearce, and Jiang Bian. Mineworld: a real-time and open-source interactive world model on minecraft, arXiv preprint arXiv:2504.08388 2025.
[47] Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos. arXiv preprint arXiv:2501.08325 2025.
[48] Authony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309. 17000, 2023.
[49] Alisson Azzolini, Hannah Brandon, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, Francesco Ferroni, Rama Govindaraju, al. Cosmos-reason From physical common sense to embodied reasoning. arXiv preprint 2rXiv22503.15556 2025.
[50] Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. arXiv preprint arXiv:2412.03577 2024.
[51] Jack Hooker, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Christos Kaplanis. Alexandre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung, Michael Dennis, Sultan Kenjeyev, Shangbang Long, Vlad Mnih, Harris Chan, Maxime Gazeau, Bonnie Li, Fabio Pardo, Luyu Wang, Lei Zhang, Frederic Besse, Tim Harley, Anna Mitenkova, Jane Wang, Jeff Clune, Demis Hassabis, Raia Hadsell, Adrian Bolton, Satinder Singh, and Tim Rocktäschel. Genie 2: A large-scale foundation world model. 2024.
13

---

[52] Aether Team, Haoyi Zhu, Yifan Wang. Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, et al. Aether: Geometric-aware unified world modeling. arXiv preprint arXiv:2503. 18945, 2025.
[53] Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3c: 3d-informed world- consistent video generation with precise camera control. arXiv preprint arXiv:2503.033: 2025.
[54] Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan. Reconx: Reconstruct any scene from sparse views with video diffusion model. arXiv preprini arXiv:2408. 16767, 2024.
[55] Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Bartin-Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint 0rXiv22405.13314 2024.
[56] Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, and Chuang Gan. Tesseract: Learning 4d embodied world models. arXiv preprint arXiv:2504.20995, 2025.
[57] Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xin- gang Pan. Worldmem: Long-term consistent world simulation with memory. arXiv preprint arXiv:2504. /2369, 2025.
[58] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568:127063, 2024.
[59] Bowen Peng. Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint urXiv:2309.0MM/ 2023.
[60] Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave context behind: Efficient infinite context transformers with infini-attention, 2024.
[61] Lvmin Zhang and Maneesh Agrawala. Packing input frame context in next-frame prediction models for video generation, 2025.
[62] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1900 13440, 2019.
[63] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, (34)6000 612, 2004.
[64] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
[65] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang. Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models. In 2024 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21807-21818, 2024.
[66] Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilum Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion, 2025.
[67] Google, Realestate 10k. Accessed: 2025-07-27.
html 2018
[68] Jianwen Jiang, Chao Liang, Jiaqi Yang, Gaojie Lin, Tianyun Zhong, and Yanbo Zheng. Loopy: Taming audio-driven portrait avatar with long-term motion dependency. arXiv preprint 24K.:220090263344 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract clearly identifies spatiotemporal consistency and compounding enror key challenges, and the paper directly addresses them through the proposed VRAG model, with improvements demonstrated using a comprehensive set evaluation metrics.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss in See the computational limitations of our work that the higher computational cost of memory retrieval-auugmente generation may limit deployment in essurce-constraing settings.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results are provided in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided relevant details on the methodology and experiment setups as seen in Section an3 Secion
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Code will be available later on Github with sufficient instructions reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httppp.ccppubcccguuuceeee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have described all relevant training and evaluation details Section More details will be provided in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The error bars are not reported in paper due heavy computation for each model training. However, we report the average scores across whole test sets for each method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The detailed computational resources reported in Sec. 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our paper does not involve human or animal subjects, and all data are collected from openly available sources.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader social impacts are discussed in Section 5 and acknowledge the
potential misuse of the video generation technology for creating mislcading or manipulated video content in games simulation systems.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our video generation models are trained from scratch, with self-collected datasets in Minecraft simulation games, therefore does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All code and data are cited explicitly and used within the scope the license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release our code on Github with a detailed documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our core method does not involve LLMs, and the entire content of this paper is written by the authors.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) (ttrge///eerrpppccc///fofeeeeeee /ctp:///e/eeeeeeee////////.... for what should or should not be described.
21

---

Appendix
A
Baseline Method Details
For the baseline methods in Sec. 3.4 we implemented the following techniques to enhance the temporal context window of our video generation model.
Long-context Enhancement To extend the temporal context window of our video generation model, we apply the YaRN [59] modification for ROPE in temporal attention module for improved extrapolation. RoPE encodes relative position via complex-valued rotations, such that the inner product between the m-th query qm and n-th key k,, depends only on the relative distance (m n):
(Am, kn) (fw, (2mm m), fw. (Zn, (1)R
(8)
Re ((Wazm)e"m) (W (Zn
(9)
Re ((W,z))Wwcce
emmm-)))
(10)
g(zms Zn, m n)
(11)
where Rel-] is real part of complex values and (-)* represents conjugate of complex numbers, Zm Zy E RDD are input vectors, W g. We are learned projections, and E IRD encodes rotation
frequencies per dimension: 0g = 2-2d/D with b = 10000.
YaRN modifies modifies the rotated input vector (w m, Oa) by applying frequency transforma- tion:
fw m, 0a) fw (00mmmgmmm).... gmm), k(0.))
(12)
with g(m) = m and frequency warping function:
h(aa) = (1 - x(ra)) 0a ((r)) -0
(13)
Here, 8 is a stretching factor and rd = Le/Xat is the context-to-waveleng ratio with Aa = 2n 2 = 2x (b))d//D and R' bs Daz The ramp function 7(-) interpolates low- www-frquency dimensions to improve extrapolation while preserving high-frequency components.
Frame Retrieval from History Buffer We also experimented with fixed-length buffer B that stores a history of previously generated latent frames, employing a heuristic sampling strategy for retrieval. Following (68), this strategy involves partitioning B into Ns 5 segments G3 for j E
Ns ordered from oldest (G1) to most recent (Gng). The total number of frames in the buffer is NB = د |Gyl. The lengths of these segments, Lj- |Gyl. decrease exponentially (e.g.. L, =1 r for a base a and L1 being the length of the oldest segment G1), ensuring that more recent segments are shorter. From each segment Gj. l frames are randomly sampled to form subset F, S G, (where [F = k). The retrieved memory 7mem is constructed as the concatenation of these sampled frames, 7mnem [Fi, F2..... FNy). totaling Nş k frames. This design with recency bias implies that the sampling density k/Lj is higher for more recent segments, thereby placing greater emphasis on recent information. This retrieved information Zmem is concatenated with current frame window z along temporal dimension as additional context: [imem z). which is then passed as input to the spatiotemporal DiT blocks, enabling the model to jointly attend to both recent and historical frames.
Neural Memory Augmentation To extend video generation capabilities to longer sequences be- yond a fixed attention window while retaining memory of past scenes, we adapt Infini-attention [60] as neural memory mechanism for our video diffusion model. Infini-attention is recurrent mechanism that augments standard dot- pro-prrd attention (local context) with a compressed summary of past context (global context) stored in evolving memory.
The model processes the video in segments using sliding window. To maintain the high degree of temporal continuity crucial for video generation, we employ overlapping segments. This is modification from the original Infini-attention, which typically processes non-overlapping segments. The input latent video segment 2, R (s is segment index) is processed to derive query qa
22

---

key k, and value matrices using standard attention mechanisms. Key-value pairs from processed segments are incrementally summarized and stored in a compressive memory M, which can be efficiently queried by subsequent segments using their query vectors. After each slide, the model first retrieves a hidden state A mem by querying the compressive memory M,-1:
(g.)M,
(14)
mem
(44)) D
where o(-) is element-wise nonlinear activation function (e.g., ELU()) + 1) and n,-1 is normalization vector (accumulated up to segment 8
1).
Next, the compressive memory M, and normalization vector n, are updated using the KV entries of the current segment s:
ak,,)
M, = M,-1 + (k,))
o(k,)m,-1 m
(15)
n, n + (k,,)= 1N
Here, N is the length of the current segment s. a.)) is applied element-wise, and 1x is an x 1 vector of ones.
The final attention output for segment s, denoted A,, combines the standard dot dot-product attention output local (local context from the current segment) with the retrieved memory state A (global context from past segments) using a learnable gating scalar 33 R::
A, sigmoid(3) Ø Amm + (1 - sigmoid( (3)) A
(16)
As in standard multi-head attention, a final linear projection is applied to A, to produce the output of the Infini-attention layer.
B Implementation Details
The VAE compresses each input frame of size 3 640 x 360 into a latent representation of size 16 32 18 before processing by the diffusion model. All diffusion models employ hidden size of 1024 and depth of 16, with one temporal and one spatial attention modules in each spatialtemporal DIT block. We use uniform learning rate of 10 during training. For Infini-Attntionn we apply a learning rate of 3 x 10-3 specifically to the global weight parameter to effectively balance global and local attention contributions while maintaining stable convergence. In VRAG, we set the weights as [10.0, 10.0, 3.00 across the global state dimentions ([x,, V, 2, yawl) in the similarity function, to accommodate the wider range yaw values. To differentiate retrieved historical frames from current context frames along the temporal dimension, we incorporate a temporal offset of 100 in the rotary position embeddings of temporal attentional for retrieved frames.
C Additional Experiments
C.1 Analysis of Compounding Error Evaluation Metries
Traditional metrics like SSIM, PSNR, and LPIPS measure pixel-level or feature-level differences between original and generated images. However, these metrics lose effectiveness when the generated video sample deviates significantly from the original video sample, especially for the compounding error evaluation, even if they falls in the same distribution and are visually reasonable. As shown in Figure I0, we normalize all metrics to a 0-1 scale where higher values indicate better generation quality (with SSIM score flipped). While all metrics perform well on the initial frame (index 0), assigning high scores to ground truth, their values begin to deteriorate after frame 100.
To address this limitation, we developed a discriminator-base based evaluation metric. We train a discrimi- nator using 1000 videos from the vanilla DF model (window size 20), with each video containing 1000 frames. This yielded dataset of 10 ground truth frames and 10 generated frames as fake ones. We implemented the discriminator as a binary classifier using lightweight architecture with ResNet blocks. Too large discriminator architecture will lead to less meaningful discriminative signals. Each block contains two convolutional layers with batch normalization and activation functions. This design provides discriminative outputs while maintains computational efficiency.
23

---

3.0
2.8
Metric 3.6
200
2000
Sample
Figure Comparison of SSIM, PSNR, LPIPS, and discriminator metrics. All metrics are normalized to the [0,1] range, where higher values indicate better performance for all scores. The discriminator score can accurately capture variations in generated image quality, while the other metrics are affected by distribution shift and fail to properly reflect compounding errors.
As shown in Fig. 10 the decrease of discriminator value faithfully reflects the distortions in generated images, while other metries decline for two reasons: image quality degradation and distributional shift from the original video. This shift prevents traditional metrics from accurately assessing generation performance in terms of the compounding error. For instance, while the 270th frame shows significantly better generation quality than the 1080th frame, SSIM, PSNR, and LPIPS assign similar scores both. This indicates that the distribution shift has become the dominant factor in lowering the metric scores, making these metrics unreliable for evaluating compounding error in long-range video generation.
Unlike traditional metrics, the discriminator's evaluation remains robust to distribution shifts since doesn't depend on the original image, but rather depending only on the distortion of the generated images. This makes the discriminator score a more reliable metric for evaluating compounding errors in this case. However, the discriminator approach has several limitations. First, training requires sampling from pre-trained diffusion model, which incurs computational overhead. Second, the training of the discriminator heavily depends on human judgment. We find that even shallow ResNet architecture can effectively distinguish between ground truth and generated images. This suggests that an overly complex model might assign uniformly low scores all generated content, making the discriminator metric less meaningful to look at. Finally, the discriminator shows limited generalization capability. When evaluating videos generated by new methods datasets, the discriminator may be deceived into assigning inappropriately high scores. Therefore we do not report the discriminator score the main paper, and advocate more investigation into faithful evaluation of compounding error in future work.
C.2 Vanilla Long-context Extension YaRN
To ensure a fair comparison, we evaluate YaRN against a baseline that directly extrapolates the vanilla model's window size from 20 during training to 40 at inference, to match the inference window length YaRN in experiments Sec. Evaluation of quantitative metrics LPIPS, SSIM and PSNR shown in Figures IT] 12 and 13 indicates that, YaRN maintains lower compounding error for long video generation (I 100 frames). This demonstrates YaRN's effectiveness in extending the context window of diffusion video models to 40 frames after minimal fine-tuning. Vanilla extension of context length DF models performs poorly due to nat-of-distribution window size at inference.
While YaRN effectively extends the context window, performance improvements are constrained by the inherent limitations of diffusion models in in-context learning. As demonstrated in Figure [1] the model exhibits difficulties effectively leveraging long-range dependencies, leading to subopti- mal spatialtemporal consistency against the ground truth. In addition, YaRN also requires greater

---

computational overheads during inference as it has a larger window size compared with other methods in our experiments Sec. making less suitable for real-time gameplay applications.
0.75
0.70
LPIPS
0.65
0.60
0.55
YaRN
DF(window 40)
100
300
500
700
900
1100
Frame Index
Figure 11: Comparison of vanilla long-context extension for DF model and YaRN with window length of 40 frames inferences. Lower better for LPIPS score.
0,45
YaRN
0.40
DF(window 40)
0.35
SSIM
0.30
0.25
0.20
0.15
0.10
100
300
500
700
900
1100
Frame Index
Figure 12: Comparison of vanilla long-context extension for DF model and YaRN with window length of 40 frames at inferences. Higher is better for SSIM score.
YaRN
16
(wiindww 40)
14
PSNR
12
mwm
10
8
100
300
500
700
900
1100
Frame Index
Figure 13: Comparison of vanilla long-context extension for DF model and YaRN with window length of 40 frames at inferences. Higher is better for PSNR score.
C.3 More Discussions on Main Results
For the main results in Sec. 4.3 and Sec. 4.4 we provide more discussions here. The Infini-attention model faces significant training challenges due its global attention mechanism. As evidenced in Figure [5] the model struggles converge during training. For VRAG without memory component, incorporated global state conditioning (specifically [x, v, 2, yaw]) into the input. However,
25

---

Prieeeee 40
Frame 100
Frome 400
Frame 700
Frame 1000
Figure 14:
Visual comparison of vanilla long-context extension for DF model and YaRN. Both models are inferred with 40 frames window.
compared the vanilla diffusion model, the training process becomes significantly more difficult. This may be due to the higher dimensionality and larger ranges of the spatial condition, whereas the action condition mostly consists of binary states ((O, 1]), making harder for the model learn and increasing perplexity.
Loss Curve
DFlwindow 20)
LO55
0.8
(na memory)
Neural Memory
0.6
0.4
0.2
200
400
600
800
1000
1200
Training Step
Figure 15: Training Loss Curves
C.4 Predicted Global State
In the paper, main experiments conducted with the access the ground-truth global state conditions during training and inference. However, the practical usage may require the global state to be also predicted based on historical states and actions. To ablate this effect, we trained pose (global state) prediction model that takes the current frame and action as inputs and outputs the predicted pose change. The next post can be derived by adding the predicted pose change the current pose. Its architecture consists of only a few convolutional layers and fully connected layers, with very small inference time overhead. state at next step, and generate videos based on the predicted global state. Following the same setting as in Sec. 4.3 the experimental results (for 300 frames prediction) are summarized in Table
evaluation, we apply this trained predictor to predict the global
Method
SSIM1
PSNR
LPIPS
DF20
0.466
16.643
0.538
VRAG (predicted pose)
0.500
17.116
0.506
VRAG
0.506
17.097
0.506
Table 6: Ablation study replacing the ground-truth global state with predicted ones by a trained
pose predictor.
26

---

As shown in the table, the evaluation results are nearly identical with or without the pose prediction, since the pose prediction is relatively simple task compared with the video generation. This proves the feasibility of using the predicted global state without significant video performance degradation.
C.5 Memory and Time Overhead
We also compare the memory usage and inference time of VRAG against several baselines: diffusion forcing with 10 and 20 context frames, and YaRN with 40 context frames.
Method
DFTO
DF20
YaRN
VRAG
Context length (Frame)
10
20
40
Memory usage (MB)
4420
4448
4543
4452
Inference Time (min)
12
23
12
Table 7: Memory usage and inference-time of different methods
As demonstrated in the Table 7 VRAG's GPU memory usage and inference time overhead are nearly identical to DF20. The inference time is derived for autoregressive generation over 600 frames. Meanwhile, the computation for the retrieval operations can be entirely performed on the CPU, with its memory footprint being only num frame x action dim x 4 Bytes = 9,4 KB in our experiments, which is almost negligible.
In summary, VRAG incurs almost no additional inference-time overhead compared to standard diffusion forcing. The memory and computational cost introduced by the retrieval mechanism negligible, as it only involves similarity calculations between set of vectors.
D
More Results
Method
SSIM f
PSNR 1
LPIPS |
VRAG
0.349
12.039
0.654
VRAG (no training)
0.218
11.588
0.712
VRAG (no memory) 0.205
11.367
0.746
Table 8: Ablation study of VRAG components for compounding error on long video generation. We compare full model with variants that remove either memory component (additional global state conditioning only) or training component (in-context learning only).
DF(window 10)
DF(window 20)
YaRN
0.60
PSNR
History Buffer
0.68
DFtwncow 10)
DF(wnnow VRAG
0.40
YaRN
History Bufter
0.35
100
125
150
225
250
275
300
100
125
150
175
200
225
250
275
300
Frame Index
Frame Index
Figure 16: World coherence evaluation all methods for PSNR (left) and LPIPS (right).
27

---

0.75
0.70
PSNR
0.65
0.60
DROWindw
DRtsindow 201
8
VRAG
0.55
VRAG
YORN
Hictrry Buller
Neural Memory
0.60
Neural Memory
Hatory
100
300
500
700
900
1100
100
300
500
700
900
1100
Frame Index
Frame Index
Figure 17: Compounding error evaluation all methods for PSNR (left) and LPIPS (right).
22
VRAG
18
VRAG
training)
training)
20
VRAG (no memory)
16
VRAG (no memory!
PSNR
2.4
69
16
14
10
100
125
150
175
200
225
250
275
300
100
300
500
700
900
1100
Frame Index
Frame Index
Figure 18: Ablation study of VRAG components for world coherence (left) and compounding error (right), with PSNR metric. We compare the full model with variants that remove either the memory component (additional global state conditioning only) or training component (in-context learning only).
0.75
0.80
0.65
0.75
0.70
0.55
0.65
0.60
0.45
VRAG
VRAG
training)
0.55
treining)
VRAG (no memory)
VRAG Ino memory)
0.35
0.50
100
125
150
175
200
225
250
275
300
100
300
500
700
900
1100
Frame Index
Frame Index
Figure 19: Ablation study of VRAG components for world coherence (left) and compounding error (right), with LPIPS metric. We compare the full model with variants that remove either the memory component (additional global state conditioning only) or training component (in-context learning only).
0.5
VRAG
VRAG training)
0.4
VRAG (no memory!
0.60
VRAG
SSIM
0.3
training)
3 0.45
VRAG (no memory)
8 0.40
0.2
0.30
0.25
11.00
Frame Index
Frame Index
Figure 20: Ablation study of VRAG components for world coherence (left) and compounding (right), with SSIM metric. We compare the full model with variants that remove either the memory component (additional global state conditioning only) training component (in-context learning only).
28


Reviews:
[{"review_id": "rUoXDQ69Tb", "reviewer": "Reviewer_udTT", "summary": "This paper focuses on world models (action conditioned interactive video generation), training models in the minecraft domain. The authors propose VRAG, comprising of two approaches to improve memory - a RAG based system and some game state conditioning. The model is based off external baselines like diffusion forcing. The ideas are neat, and seem to help in this setting, although questions remain about their general applicability given the extremely limited dataset used for the results.", "questions": "Do you have any fair comparisons w.r.t memory usage for your method vs. baselines? I assume yours uses more, for the same model size, so it is not a fair comparison.\n\nWhat if you scale up the baseline?", "limitations": "Compute/dataset size is so small that this might not be relevant for the frontier of the field.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper is in a relevant area, as world models and interactive video generation are becoming increasingly important.\n- The model ideas seem novel, borrowing concepts from LLMs (e.g. RAG) and showing it can be effectively applied to this new setting.\n- Experiments show clear ablations which means actionable insights can be taken and applied to the more general domain. Further, this work may help folks train better world models for single gaming environments, which is a current popular paradigm.\n\nWeaknesses\n- The dataset size is trivially small, which makes all the analysis potentially limited to the specific small scale setting. The methods proposed might be very effective for memorizing the data, rather than providing generalization benefits at scale. To be concrete, the dataset is 17 hours of minecraft, vs. Genie 1 which used 30,000 hours of internet videos from hundreds of games. Genie 1 is also 18 months old and not SoTA. \n- It is definitely a negative that this work focuses entirely on Minecraft. We cannot be sure if ideas generalize to a truly “foundation” world model if it is only trained on a single game.\n- The citations are totally off in this paper, which makes the positioning very strange. In the opening statement, the authors say “Foundational world models capable of simulating future outcomes based on different actions 21 are crucial for effective planning and decision making [1, 2, 3].” and then cite [1] a NeurIPS 2015 paper which is not a foundation world model, [2] David Ha World Models paper, which is iconic, but also not a foundation world model, and then [3] Dreamer, which is a single environment, not foundation world model. A foundation world model is one where it can be prompted to generate new worlds, which none of these can. The only foundation world models produced so far are Genie1, Genie2 and Cosmos, as far as I am aware. \n- The authors incorrectly reference the Genie 2 model “One line of research addressing this involves predicting the underlying 3D world structure like Genie2”. This is incorrect, Genie 2 was a latent diffusion model, with no inductive bias. If anything, it is very close related work.", "was_revised": false, "final_justification": "", "rebuttal": "> W1: The dataset size is trivially small, which makes all the analysis potentially limited to the specific small scale setting. The methods proposed might be very effective for memorizing the data, rather than providing generalization benefits at scale. To be concrete, the dataset is 17 hours of minecraft, vs. Genie 1 which used 30,000 hours of internet videos from hundreds of games. Genie 1 is also 18 months old and not SoTA.\n> \n\n**Reply to W1:** We appreciate the reviewer’s feedback regarding dataset size and scalability. As an academic research group, our computational resources are inherently limited compared to industry-scale projects like Genie, which leverages Google’s infrastructure for 30K hours of training data. Our goal is not to achieve state-of-the-art generalization across diverse domains but to validate our method’s efficacy specifically for compounding errors and memory issues in modeling Minecraft—a well-defined, structured environment where 17 hours of gameplay (though modest) suffices to demonstrate meaningful world modeling capabilities. The results show our model captures key dynamics without overfitting, suggesting the approach is viable for targeted domains even with smaller-scale data. We believe our method can show more significant performance gain at a larger scale. Scaling to internet-level datasets remains valuable future work, but our focus here is on foundational insights for domain-specific applications.\n\n> W2: It is definitely a negative that this work focuses entirely on Minecraft. We cannot be sure if ideas generalize to a truly “foundation” world model if it is only trained on a single game.\n> \n\n**Reply to W2:** We sincerely appreciate your concern regarding the generalizability of our approach. To address this, we have conducted additional experiments in a real-world setting. Specifically, following the experimental setup of **Diffusion Forcing Transformer (DFoT)** [1], we trained our diffusion models on the RealEstate10K dataset and used the pretrained model provided in [1] as the baseline. After fine-tuning for just 2 epochs (10% of the original training steps), our method significantly outperforms the baseline in terms of memorization capability. The experimental results are summarized in the table below:\n\n| Method\\Metric | SSIM↑ | PSNR↑ | LPIPS↓ | FVD↓ |\n| --- | --- | --- | --- | --- |\n| DFoT | 0.4436 | 13.03 | 0.4469 | 337.5 |\n| VRAG | **0.9116** | **32.21** | **0.1146** | **221** |\n\nWe believe these results effectively demonstrate the generalizability of our approach beyond Minecraft.\n\n> W3: The citations are totally off in this paper, which makes the positioning very strange. In the opening statement, the authors say “Foundational world models capable of simulating future outcomes based on different actions 21 are crucial for effective planning and decision making [1, 2, 3].” and then cite [1] a NeurIPS 2015 paper which is not a foundation world model, [2] David Ha World Models paper, which is iconic, but also not a foundation world model, and then [3] Dreamer, which is a single environment, not foundation world model. A foundation world model is one where it can be prompted to generate new worlds, which none of these can. The only foundation world models produced so far are Genie1, Genie2 and Cosmos, as far as I am aware.\n> \n\n**Reply to W3:** We appreciate the reviewer for pointing out misleading references for “foundational world models” and will modify accordingly. \n\n> W4: The authors incorrectly reference the Genie 2 model “One line of research addressing this involves predicting the underlying 3D world structure like Genie2”. This is incorrect, Genie 2 was a latent diffusion model, with no inductive bias. If anything, it is very close related work.\n> \n\n**Reply to W4:** Here we aim the emphasize the distinguished 3D world modeling capability of Genie2 compared with Genie, which only models 2D visual features with much less spatiotemporal consistency. We didn’t indicate the inductive bias in the method, but will modify this sentence to avoid any confusion.\n\n> Q1: Do you have any fair comparisons w.r.t memory usage for your method vs. baselines? I assume yours uses more, for the same model size, so it is not a fair comparison.\n> \n\n**Reply to Q1:** All models are trained on the same dataset. The baselines DF20 and History Buffer exhibit comparable memory usage and inference-time overhead to our method during both inference and training phases. The inference time is derived for autoregressive generation over 600 frames. DF10 demonstrates lower inference-time overhead due to shorter context, YaRN incurs higher overhead. We think that this constitutes a fair and balanced comparison.\n\n| Method | DF10 | DF20 | YaRN | VRAG |\n| --- | --- | --- | --- | --- |\n| Context length (Frame) | 10 | 20 | 40 | 20 |\n| Memory usage (MB) | 4420 | 4448 | 4543 | 4452 |\n| Inference Time (min) | 9 | 12 | 23 | 12 |\n\n> Q2: What if you scale up the baseline?\n> \n\n**Reply to Q2:** We sincerely would like the reviewer to clarify which aspect of \"scale up\" that is referred to. To alleviate the concerns, we show results with longer context and more training data.  First, we increased the window size to 40 (DF40) on the same dataset. Second, we trained DF20 on a significantly larger dataset—collecting 6x more data (100 hours of video), which required 4 days of training on 8 A100 GPUs. The experimental results are as follows:\n\n| Method\\Metric | SSIM↑ | PSNR↑ | LPIPS↓ |\n| --- | --- | --- | --- |\n| DF20 | 0.466 | 16.643 | 0.538 |\n| DF40 | 0.484 | 17.048 | 0.500 |\n| DF20 (Larger dataset) | 0.482 | 16.908 | **0.496** |\n| VRAG | **0.506** | **17.097** | 0.506 |\n\nAs shown in the table, increasing the dataset size or window size does help improve the model's performance, but it still cannot fully resolve the issue of compounding errors and insufficient memory. VRAG still outperforms in both SSIM and PSNR metrics against the baselines with larger scales, demonstrating the effectiveness of our method.\n\n**Reference**\n\n[1] History-Guided Video Diffusion, Arxiv 2025"}, {"review_id": "ax9sqXVuui", "reviewer": "Reviewer_UWPa", "summary": "The paper observes that existing video generation models, when used as interactive world models, exhibit pronounced spatiotemporal inconsistencies. It traces these issues to two shortcomings of autoregressive generators: compounding prediction errors and an inadequate memory mechanism. To overcome these limitations, the authors propose Video Retrieval-Augmented Generation (VRAG), which enriches the video world model with global state conditioning and memory retrieval mechanisms, thereby achieving long-term consistency.", "questions": "(1)\tAll reported metrics—SSIM, PSNR, and LPIPS—are frame-level measures and therefore do not adequately capture improvements in long-term consistency. It would be better to provide a justification for why these metrics are suitable for evaluating long-term consistency, or introduce and employ improved metrics that better capture long-term consistency. \n\n(2) The experiments are conducted on only a single dataset, which may restrict the generalizability of the results. To strengthen the claims, the authors may consider validate their approach on additional datasets.\n\n(3) The range of comparison methods is too limited—experiments compare only against a few variants of the baseline (the vanilla model in Sec. 3.2).  For a more comprehensive evaluation, it would be better to include comparisons with a broader range of state-of-the-art methods.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n(1) The authors seamlessly transfer techniques from the LLM to the autoregressive-generation setting and refine them so they operate effectively in that regime. \n(2) The paper conducts a thorough ablation study, demonstrating the effectiveness of the proposed method.\n\nWeaknesses:\n(1) All reported metrics—SSIM, PSNR, and LPIPS—are frame-level measures and therefore do not adequately capture improvements in long-term consistency. Maybe an analysis between the metrics and\n(2) Experiments are restricted to a single dataset, which limits the generalizability of the findings.\n(3) The range of comparison methods is too limited—experiments compare only against a few variants of the baseline (the vanilla model in Sec. 3.2).", "was_revised": true, "final_justification": "All my concerns about metrics and comprison on more datasets are well answered.", "rebuttal": "> Q1: All reported metrics—SSIM, PSNR, and LPIPS—are frame-level measures and therefore do not adequately capture improvements in long-term consistency. It would be better to provide a justification for why these metrics are suitable for evaluating long-term consistency, or introduce and employ improved metrics that better capture long-term consistency.\n> \n\n**Reply to Q1:** We sincerely appreciate the reviewer's valuable concerns and suggestions regarding the evaluation metrics. These metrics are commonly applied in existing research of video world modeling [4,5,6,7]. Introducing improved metrics for long-term consistency is indeed important but it could be an independent research topic. The frame level distance against ground truth indicates the fidelity of prediction regardless of the video length. However, the temporal consistency is indeed lack in our measure. To address this concern, we have conducted additional evaluations using metrics from **VBench** [1] to assess our generated videos, including temporal quality metrics like Background Consistency, Temporal Flickering and Motion Smoothness. Since our work does not focus on text-to-video generation, we did not employ the Video-Condition Consistency metrics from VBench, but instead focused on the Video Quality metrics. The evaluation results on VBench (higher is better) are shown in the following table:\n\n| Method\\Metric | Background Consistency | Temporal Flickering | Motion Smoothness | **Aesthetic Quality** | **Imaging Quality** |\n| --- | --- | --- | --- | --- | --- |\n| DF20 | 0.9668 | 0.9485 | 0.9582 | 0.5272 | 0.6058 |\n| YaRN | **0.9686** | 0.9401 | 0.9523 | 0.5252 | 0.6323 |\n| Historical Buffer | 0.9664 | 0.9475 | 0.9579 | 0.5167 | 0.6253 |\n| **VRAG** | **0.9686** | **0.9511** | **0.9603** | **0.5295** | **0.6444** |\n\nAs demonstrated in the results, our method outperforms all other baselines across all metrics in both temporal quality and video frame quality. The Neural Memory baseline has Aesthetic Quality 0.343 and Imaging Quality score 0.3597 respectively, which are significantly lower than other baselines, therefore not listed here. Our VRAG shows better temporal consistency compared with all baseline methods, and the high video frame quality indicates the results are not over-smoothed.\n\n> Q2: The experiments are conducted on only a single dataset, which may restrict the generalizability of the results. To strengthen the claims, the authors may consider validate their approach on additional datasets.\n> \n\n**Reply to Q2:** We sincerely appreciate your concern regarding the generalizability of our approach. To address this, we have conducted additional experiments in a real-world setting. Specifically, following the experimental setup of **Diffusion Forcing Transformer (DFoT)** [2], we trained our diffusion model on the RealEstate10K dataset and used the pretrained model provided in [2] as the baseline.\n\nAfter fine-tuning for just 2 epochs (10% of the original training steps), our method significantly outperforms the baseline in terms of memorization capability. The experimental results are summarized in the table below:\n\n| Method\\Metric | SSIM↑ | PSNR↑ | LPIPS↓ | FVD↓ |\n| --- | --- | --- | --- | --- |\n| DFoT | 0.4436 | 13.03 | 0.4469 | 337.5 |\n| VRAG | **0.9116** | **32.21** | **0.1146** | **221** |\n\nWe believe these results effectively demonstrate the generalizability of our approach beyond Minecraft.\n\n> Q3: The range of comparison methods is too limited—experiments compare only against a few variants of the baseline (the vanilla model in Sec. 3.2). For a more comprehensive evaluation, it would be better to include comparisons with a broader range of state-of-the-art methods.\n> \n\n**Reply to Q3:** We sincerely appreciate the reviewer's concern regarding the comprehensiveness of our baseline methods. To address this, we have expanded our experiments by incorporating two additional baselines to strengthen our paper. Specifically, we have supplemented experiments on RealEstate and adopted the Diffusion Forcing Transformer [2] as one of our baselines (please refer to our response to Q2 for details). For the experiments in Minecraft, following Reviewer M9fd's suggestion, we have included Frame Pack [3] as another baseline to further demonstrate the superiority of our approach. The detailed experimental results are presented in the table below, and our method is superior to the Frame Pack baseline.\n\n| Method\\Metric | SSIM↑ | PSNR↑ | LPIPS↓ |\n| --- | --- | --- | --- |\n| Frame Pack | 0.421 | 16.372 | 0.574 |\n| DF20 | 0.466 | 16.643 | 0.538 |\n| VRAG | **0.506** | **17.097** | **0.506** |\n\n**Reference**\n\n[1] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024\n\n[2] History-Guided Video Diffusion, Arxiv 2025\n\n[3] Packing Input Frame Context in Next-Frame Prediction Models for Video Generation, Arxiv 2025\n\n[4] Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, Arxiv 2023\n\n[5] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer, ICLR 2025\n\n[6] SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation, Arxiv 2024\n\n[7] GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control, Arxiv 2025"}, {"review_id": "vicvk6rUdU", "reviewer": "Reviewer_M9fd", "summary": "This paper proposes several methods to address the challenges of compounding errors and insufficient memory in autoregressive, action-conditioned video generation. These methods include action and state conditioning, history state retrieval, and so on. Experiments are conducted on the Minecraft datasets, and the performance is evaluated based on several video-quality metrics.", "questions": "- How's the performance of the proposed retrieval method compared to FramePack [1]? This baseline should be included in the next version of this paper.\n- Can the authors provide more qualitative demos for assessing the performance, such as the generated videos?\n- Is the model trained from scratch or adapted from a pre-trained model? If this model is trained from scratch, why don't the authors choose a pre-trained model for fine-tuning?\n\n[1] Packing Input Frame Context in Next-Frame Prediction Models for Video Generation", "limitations": "Limitations have been discussed in the paper", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- This paper identifies two key challenges in autoregressive video generation: compounding errors and insufficient memory. Some analyses are given for addressing these challenges.\n- This paper incorporates methods from LLMs to investigate their performance in the context of video generation, which is a promising direction.\n- The proposed method is sound. I like the idea of history retrieval, which could enhance the in-context learning ability for video generative models.\n## Weaknesses\n- Writing in this paper needs to be improved. The paper should highlight the insights and contributions. However, there are many paragraphs describing different methods, training details, baselines, and so on.\n- Experimental results are not sufficient to validate the effects of the proposed method. I noticed that the background and diversity of the Minecraft videos are limited. Therefore, **I suspect the trained model can be easily overfitted and can be difficult to generalize to other domains**. Moreover, the paper only presents the empirical results for in-domain performance, where the metrics are calculated by comparison with the ground truth videos.\n- In line 160, this paper claims that global state conditioning is significant for the final performance. However, for many realistic cases where simulators are not available, this kind of information is not accessible.\n- Miss references. The method of action-conditioned block has been discussed by previous works [1,2], and the idea of leveraging video models for world simulation has been proposed by a recent work [1]. This paper should add a discussion section to distinguish the differences and advantages compared to existing works.\n\n[1] Pre-Trained Video Generative Models as World Simulators, Arxiv 2025\n\n[2] IRASim: Learning Interactive Real-Robot Action Simulators, Arxiv 2024", "was_revised": true, "final_justification": "The authors addressed some of my concerns in the discussion phase, thus I increased the rating accordingly. However, the main contributions and insights provided are still not clear in this version. I believe this paper should incorporate the additional experiments, explanations, and discussions in the next version to reach the acceptance threshold.", "rebuttal": "> W1: Writing in this paper needs to be improved...\n\n**Reply to W1:** We appreciate the reviewer’s feedback on further improving the writing. We will modify accordingly to highlight more on our contributions and leave more details of different methods and implementation in supplementary materials.\n\n> W2: Experimental results are not sufficient to validate the effects of the proposed method...\n\n**Reply to W2:**  We appreciate the reviewer’s concerns regarding the diversity of Minecraft videos and the generalizability of our method. To address these points, we have conducted additional experiments in a real-world setting. Specifically, following the experimental setup of **Diffusion Forcing Transformer (DFoT)** [4], we trained our diffusion model on the RealEstate10K dataset and used the pretrained model from [4] as the baseline. After fine-tuning for just 2 epochs (10% of the original training steps), our method significantly outperforms the baseline in terms of memorization capability. The results (summarized in the table below) demonstrate that our approach generalizes effectively beyond the Minecraft domain.\n\n|Method\\Metric|SSIM↑|PSNR↑|LPIPS↓|FVD↓|\n|-|-|-|-|-|\n|DFoT|0.4436|13.03|0.4469|337.5|\n|VRAG|**0.9116**|**32.21**|**0.1146**|**221**|\n\nRegarding the concern about overfitting, we believe there may be a misunderstanding of our methods and evaluation protocal. Our test set is entirely distinct from the training set, and the context frames are different from those to be predicted for both training and test stages. Importantly, our goal is to let the model develop **in-context learning** capabilities at inference time—retaining and recalling predicted historical information during inference, instead of reusing ground truth frames. For example, if the model sees a tree in its historical context, we expect it to generate the same tree from the same viewpoint later, relying on retrieval and in-context learning rather than just memorization of training data. Overfitting on the training set would not provide this in-context learning capability.\n\nFinally, we acknowledge the reviewer’s observation that our evaluation metrics primarily compare generated videos to ground truth. While this is a valid consideration, we want to emphasize three points: (1) We did not claim to build a foundational world model across domains due to the limited resources, but focus on validating the effectiveness of VRAG in single domain (Minecraft) setting, therefore the evaluation is indeed in-domain performance. (2) One of our major contributions is improving the model’s memorization capability, therefore frame-wise measure against the ground truth video is a natural and valid metric for fidelity. (3) Specifically, for long-term video consistency, there is no commonly applied evaluation metric, and most existing work relies on ground truth comparisons, with metrics reported in our paper. To further strengthen our evaluation, we have incorporated additional metrics from **VBench** [5] to assess temporal quality and video frame quality. Since our work does not focus on text-to-video generation, we did not employ the Video-Condition Consistency metrics from VBench, but instead focused on the Video Quality metrics. The evaluation results on VBench (higher is better) are shown in the following table:\n\n|Method\\Metric|Background Consistency|Temporal Flickering|Motion Smoothness|Aesthetic Quality|Imaging Quality|\n|-|-|-|-|-|-|\n|DF20|0.9668|0.9485|0.9582|0.5272|0.6058|\n|YaRN|**0.9686**|0.9401|0.9523 |0.5252|0.6323|\n|Historical Buffer|0.9664|0.9475|0.9579|0.5167|0.6253|\n|**VRAG**|**0.9686**|**0.9511**|**0.9603**| **0.5295**|**0.6444**|\n\nAs demonstrated in the results, our method outperforms all other baselines across all metrics in both temporal quality and video frame quality. The Neural Memory baseline has Aesthetic Quality 0.343 and Imaging Quality score 0.3597 respectively, which are significantly lower than other baselines, therefore not listed here. Our VRAG shows better temporal consistency compared with all baseline methods, and the high video frame quality indicates the results are not over-smoothed.\n\nWe hope these clarifications and additional experiments alleviate the reviewer’s concerns.\n\n> W3: In line 160, this paper claims that global state conditioning is significant...\n\n**Reply to W3:** We sincerely appreciate the reviewer's concern. We would like to clarify that while explicitly modeling global state is crucial for accurately memorizing and retrieving generated videos, our VRAG framework does not solely rely on global state to be effective. The system can still retrieve relevant historical information based on frame similarity alone, which no longer requires the global state in retrieval process.\n\nMoreover, there are various methods to extract global state information from RGB videos, such as SLAM algorithms. As mentioned in W2's response, the RealEstate10K dataset actually utilizes SLAM to collect global state information. Same technique can be applied on other datasets as well. Additionally, we have trained a lightweight pose prediction model, to replace the ground-truth global state during model inference, and the experimental results (see table) demonstrate that the predictor can achieve comparable performance as our current methods for 300 frames prediction. We will provide this additional experiments in our paper to alleviate the concern.\n\n| Method\\Metric|SSIM↑|PSNR↑|LPIPS↓|\n|-|-|-|-|\n| DF20 |0.466|16.643| 0.538 |\n| VRAG (predicted pose)|0.500|**17.116**|**0.506**|\n| VRAG|**0.506**|17.097|**0.506**|\n\n> W4: Miss references...\n\n**Reply to W4:** We appreciate the reviewer for mentioning the missing references and will add discussions in our paper. The two papers focus on robotic manipulation video prediction, with less dynamic backgrounds. Although they also work on action-conditioned video prediction like our baseline methods (DF20), they do not try to address the memory issue in long-term world modeling. Our work distinguish from theirs in both dataset choices and techniques.\n\n> Q1: How's the performance of the proposed retrieval method compared to FramePack? This baseline should be included in the next version of this paper.\n\n**Reply to Q1:** We sincerely appreciate the reviewer's valuable suggestion. We have additionally conducted experiments using Frame Pack [3] as a baseline with same number of context frames and training steps. The comparative results shown in the following table:\n\n|Method\\Metric|SSIM↑|PSNR↑|LPIPS↓|\n|-|-|-|-|\n|Frame Pack|0.421|16.372|0.574|\n|DF20| 0.466|16.643|0.538|\n|VRAG|**0.506**|**17.097**|**0.506**|\n\nOur VRAG clearly outperforms this baseline across metrics under the same setting. We would also like to emphasize that, at the time of our original submission, the Frame Pack paper is a concurrent work. Also, the key techniques of Frame Pack and ours are orthogonal in nature - while Frame Pack primarily focuses on improving computational efficiency and reducing overhead, our method can actually address the memory issue more effectively. Our proposed video retrieval augmentation can be complementary to the Frame Pack technique for better performances as a future work.\n\nAs noted in the Frame Pack paper, their frame compression relies on importance sampling that typically prioritizes more recent frames, which is straightforward and not optimal. When combined with our approach, the similarity function from our VRAG framework could be a better alternative, which would significantly enhance its memory capabilities with recall of the most relevant information in historical frames. This potential synergy demonstrates how our methods can work together rather than a direct competitor.\n\n> Q2: Can the authors provide more qualitative demos for assessing the performance, such as the generated videos?\n\n**Reply to Q2:** We have prepared some video-based qualitative results for visualization. However, we regret to notice that due to NIPS's newly implemented policy this year, we are unable to provide any external links or any images/videos. We plan to create a project webpage in the future where these video demonstrations will be made available. If interested, please follow our work for updates.\n\n> Q3: Is the model trained from scratch or adapted from a pre-trained model? If this model is trained from scratch, why don't the authors choose a pre-trained model for fine-tuning?\n\n**Reply to Q3:** All models in our study are trained from scratch. We sincerely appreciate the reviewer's valuable suggestion regarding the use of pre-trained models. Indeed, fine-tuning from pre-trained models represents a promising direction that could potentially enhance both efficiency and performance. As shown in our additional experiments with DFoT fine-tuning (please refer to the results in W2's response), we have explored this approach.\n\nThere are two primary reasons for our original decision not to use pre-trained models. First, there are currently limited open-source models employing the diffusion forcing paradigm. While fine-tuning from full sequence denoising models is a viable strategy, it requires further investigation. Additionally, some of our baseline methods cannot be directly adapted from pre-trained models through fine-tuning due to their unique architectural designs, such as YaRN and Neural Memory. Second, the Minecraft environment in our study features relatively simple settings, where training from scratch can achieve satisfactory results, as demonstrated by our experimental outcomes.\n\n**Reference**\n\n[1] Pre-Trained Video Generative Models as World Simulators, Arxiv 2025\n\n[2] IRASim: Learning Interactive Real-Robot Action Simulators, Arxiv 2024\n\n[3] Packing Input Frame Context in Next-Frame Prediction Models for Video Generation, Arxiv 2025\n\n[4] History-Guided Video Diffusion, Arxiv 2025\n\n[5] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024"}, {"review_id": "XaVrzlHSrD", "reviewer": "Reviewer_4P6H", "summary": "This work introduces Video Retrieval Augmented Generation (VRAG), a framework for long-horizon, action-conditioned video generation with improved spatio-temporal consistency. The authors identify two key challenges in autoregressive video world models: compounding errors over time and insufficient memory mechanisms for maintaining long-term coherence. VRAG mitigates these by (1) conditioning on explicit global state (position and pose), and (2) retrieving and integrating past frames through a modified memory buffer during both training and inference. Through extensive experiments in Minecraft, the authors demonstrate that standard long-context techniques from language models (e.g. RoPE extensions, neural memory) are ineffective in the video domain, while VRAG substantially improves video quality and long-term consistency.", "questions": "* Would VRAG work in domains without access to structured global state (raw RGB-only settings)?\n\n* Have you evaluated whether global state prediction (for example from vision) can substitute the oracle state used in your setup?\n\n* What is the inference-time overhead introduced by retrieval, especially during long-horizon rollouts?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### __Strengths__\n\n1. The paper tackles two fundamental issues: compounding error and long term memory, through an interactive and long video generation setting.\n\n2. VRAG outperforms strong baselines (Diffusion Forcing, Neural Memory) across world coherence and compounding error evaluations.\n\n3. The authors convincingly show that in-context learning techniques from LLMs transfer poorly to video generation.\n\n### __Weaknesses__\n\n1. The effectiveness of VRAG depends on access to explicit global states (like coordinates and orientation), which may not be available or easily inferred in real-world settings.\n\n2. There is no qualitative video to support the claims of the paper, apart from the image figures. It would be much more compelling to have a side-by-side comparison of the LLM techniques and the proposed VRAG in a long horizon rollout.\n\n3. The method is only evaluated on Minecraft gameplay, a synthetic and stylised environment. It’s unclear whether the findings would hold in real-world video (robotics, driving).", "was_revised": true, "final_justification": "I would like to thank the authors for responding to my questions, which have all been well addressed. After reading the other reviews, that acknowledged similar strengths to what I've highlighted, and no particular strong weakness that have not been left unanswered in the rebuttal, I've decided to keep my initial rating.", "rebuttal": "> W1: The effectiveness of VRAG depends on access to explicit global states (like coordinates and orientation), which may not be available or easily inferred in real-world settings.\n> \n\n**Reply to W1:** We sincerely appreciate your insightful concern regarding the dependency on explicit global states. It is worth noting that VRAG can still be effectively employed even when such states are not directly available. For instance, retrieval can be performed based on image similarity to access relevant historical information without relying on explicit coordinates. Additionally, as mentioned in our response to Q2, a lightweight pose predictor can be trained to infer the necessary global state information. Furthermore, with the rapid advancements in SLAM algorithms, many RGB-only SLAM methods have proven highly effective in accurately estimating global states. In fact, as discussed in our response to W3, the RealEstate10K dataset itself leverages SLAM-derived camera trajectories to obtain global state information, demonstrating the feasibility of this approach in real-world settings.\n\n> W2: There is no qualitative video to support the claims of the paper, apart from the image figures. It would be much more compelling to have a side-by-side comparison of the LLM techniques and the proposed VRAG in a long horizon rollout.\n> \n\n**Reply to W2:** We have prepared some video-based qualitative results for visualization. However, we regret to notice that due to NIPS's newly implemented policy this year, we are unable to provide any external links or any images/videos. We plan to create a project webpage in the future where these video demonstrations will be made available. If interested, please follow our work for updates.\n\n> W3: The method is only evaluated on Minecraft gameplay, a synthetic and stylised environment. It’s unclear whether the findings would hold in real-world video (robotics, driving).\n> \n\n**Reply to W3:** We fully understand your concerns and appreciate your valuable suggestions. However, on one hand, driving datasets are extremely scarce, and most of them are internal data that is not open-source. On the other hand, most robotic manipulation scenarios are relatively monotonous with less dynamic backgrounds, and the phenomenon of forgetting occurs less frequently. Therefore, we follow the experimental setup of **Diffusion Forcing Transformer (DFoT)** [1]. We trained our diffusion model on the RealEstate10K dataset and used the pretrained model from [1] as the baseline. After fine-tuning for just 2 epochs (10% of the original training steps), our method significantly outperforms the baseline in terms of memorization capability. The results (summarized in the table below) demonstrate that our approach generalizes effectively beyond the Minecraft domain.\n\n| Method\\Metric | SSIM↑ | PSNR↑ | LPIPS↓ | FVD↓ |\n| --- | --- | --- | --- | --- |\n| DFoT | 0.4436 | 13.03 | 0.4469 | 337.5 |\n| VRAG | **0.9116** | **32.21** | **0.1146** | **221** |\n\n> Q1: Would VRAG work in domains without access to structured global state (raw RGB-only settings)?\n> \n\n**Reply to Q1:** As addressed in our response to W1, we believe VRAG remains applicable even in scenarios where explicit global states are unavailable.\n\n> Q2: Have you evaluated whether global state prediction (for example from vision) can substitute the oracle state used in your setup?\n> \n\n**Reply to Q2:** We trained a pose predictor model that takes actions and images as inputs. Its architecture consists of only a few convolutional layers and fully connected layers, resulting in a very small inference time overhead. We used the trained predictor to make predictions on inference data and generated videos using the predicted global state. The experimental results (for 300 frames prediction) are as follows:\n\n| Method\\Metric | SSIM↑ | PSNR↑ | LPIPS↓ |\n| --- | --- | --- | --- |\n| DF20 | 0.466 | 16.643 | 0.538 |\n| VRAG (predicted pose) | 0.500 | **17.116** | 0.506 |\n| VRAG | **0.506** | 17.097 | **0.506** |\n\nAs we can see, the evaluation results are nearly identical. This proves the feasibility of using the predicted global state. We will add this experiments in our paper for clarity. \n\n> Q3: What is the inference-time overhead introduced by retrieval, especially during long-horizon rollouts?\n> \n\n**Reply to Q3:** During single-step model prediction, VRAG incurs almost no additional inference-time overhead compared to standard diffusion forcing. The computational cost introduced by the retrieval mechanism is negligible, as it only involves similarity calculations between a set of vectors.\n\n| Method | DF20 | VRAG |\n| --- | --- | --- |\n| Memory usage (MB) | 4448 | 4452 |\n| Inference Time (min) | 12 | 12 |\n\nAs demonstrated in the table, VRAG's GPU memory usage and inference time overhead are nearly identical to DF20. The inference time is derived for autoregressive generation over 600 frames. Meanwhile, the computation for the retrieval part can be entirely performed on the CPU, with its memory footprint being only num_frame × action_dim × 4 Bytes = 9.4 KB, which is almost negligible.\n\n**Reference**\n\n[1] History-Guided Video Diffusion, Arxiv 2025"}]

Author rebuttal:

Meta-review: This paper introduces Video Retrieval Augmented Generation (VRAG),    to improve long-term consistency and mitigate compounding errors in interactive video generation. Initially, all reviewers expressed significant concerns about the limited evaluation, which was confined to a small Minecraft dataset, raising questions about generalizability and the suitability of the evaluation metrics. However, the authors provided a thorough rebuttal that included new experiments on a real-world dataset and additional metrics for temporal consistency, which successfully addressed the majority of the issues raised. While one reviewer remained concerned that the insights gained from the small-scale experiments might not translate to larger systems, the overall consensus is in favor of the paper. The proposed method is a neat and valuable contribution.
Decision: Accept (poster)