WorldMem: Long-term Consistent World Simulation with Memory
Authors: Zeqi Xiao
Yushi LAN
Yifan Zhou
Wenqi Ouyang
Shuai Yang
Yanhong Zeng
Xingang Pan
Venue: NeurIPS 2025 poster ()
Keywords: world simulation
memory
Primary area: deep_learning

Abstract: World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing state-aware memory attention that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.

WORLDMEM: Long-term Consistent World Simulation with Memory
Zeqi Xiao
Yushi Lan'
Yifan Zhou'
Wenqi Ouyang'
Shuai Yang" Yanhong Zeng* Xingang Pan
L-b,a,, Nanyang Technological University,
2 Wangxuan Institute of Computer Technology. Peking University *Shanghai AI Laboratory
(zeqi001, yushi001, yifan006, wenqi. ouyan,, xingang pan-@ntu. edu..edu..g... williamyangopku. hewpppluueede.e.ommm en, zengyh19000gmail.co com
Abstract
World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences actions. However, the limited tem- poral context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WORLD- MEM, framework that enhances scene generation with memory bank consisting memory units that store memory frames and states (e.g., poses and timestamps). By employing state-aware memory attention that effectively extracts relevant in- formation from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework only models static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simu- lated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach. Project page at https://xizaquu http://iiigggggiuuu.bouu github.iof www.ob.oo/ooooo wooowmmmmmmm
under significant
1 Introduction
World simulation has gained significant attention for ability to model environments and predict the outcomes of actions (Bar al. 2024 Decart et al., 2024 Alonso et al.) 2025 Feng al. 2024 Parker-Holder et al.) 2024 Valevski et al. 2024). Recent advances in video diffusion models have further propelled this field, enabling high fidelity rollouts of potential future scenarios based user actions, such as navigating through an environment or interacting with objects. These capabilities make world simulators particularly promising for applications in autonomous navigation (Feng et al. 2024, Bar et al. 2024) and as viable alternatives to traditional game engines (Decart et al. 2024
Parder-Holder al. (022)).
Despite these advances, a fundamental challenge remains: the limited probing horizon. Due to computational and memory constraints, video generative models operate within fixed context window and are unable to condition on the full sequence of past generations. Consequently, most existing methods simply discard previously generated content, leading to a critical issue of world inconsistency, which also revealed in Waan et al. (2025). As illustrated in Figure [T), when the camera moves away and returns, the regenerated content diverges from the earlier scene, violating the coherence expected a consistent world.
A natural solution is to maintain an external memory that stores and retrieves relevant historical information outside the generative loop. While intuitive, formulating such memory mechanism
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Temporal direction
Imioolce
Place pumpkin
Place pumpkin
Wonder around
Wander
Wonder
Wonder
Right
Left
Turn left
7urn left
Turn
Turn left
(a) without Memory
(b) with Memory
Figure 1: WORLDMEM enables long-term consistent world generation with an integrated mem- ory mechanism. (a) Previous world generation methods typically face the problem of inconsistent world due to limited temporal context window size. (b) WORLDMEM empowers the agent to explore diverse and consistent worlds with expansive action space, e-g. crafting environments by placing objects like pumpkin light or freely roaming around. Most importantly, after exploring for while and glancing back, find the objects we placed are still there, with the inspiring sight of the light melting the surrounding snow, testifying to the passage of time. Red and green boxes indicate scenes that should be consistent.
non-trivial. A direct approach might involve explicit 3D scene reconstruction to preserve geometry and detail. However, 3D representations are inflexible dynamic and evolving environments and are prone to loss of detail, especially for large, unbounded scenes (Wu et al. (025a).
Instead, we argue that geometry- -free representations offer more flexible solution. These representa- tions, however, pose their own challenges particularly in balancing detail retention with memory scalability. For example, implicit approaches like storing abstract features via LoRA modules (Hong et al. (2244) offer compactness but lose visual fidelity and spatial specificity. Some recent works represent visual scenes as discrete tokens encoding fine-grained visual information (Sajjadi et al.) 2022, et al. 2025), , but they are limited by fixed token and struggle to capture the complexity of diverse and evol ving environments. immediate future, only small subset of historical content is typically relevant. Based on this, propose a token-level memory bank that stores all previously generated latent tokens, and retrieves a targeted subset for each generation step based relevance.
address this issue, we observe that for generating
Conditioning on the retrieved memory requires spatial-temporal reasoning. In contrast prior work where memory aids local temporal smoothness (Zheng et al., (024a) or semantic coherence (Wu et al. 20256) Rahman al. 2023), long-term world simulation demands reasoning over large spatiotemporal gaps, e.g., memory and query may differ in viewpoint and time, and retain exact scenes with detail. facilitate this reasoning, we propose augmenting each memory unit with explicit state cues, including spatial location, viewpoint, and timestamp. These cues serve as anchors for reasoning and are embedded as part of the query-key attention mechanism. Through this state-aware attention, model can effectively reason the current frame with past observations, facilitating accurate and coherent generation. Importantly, such a design leverages standard attention architectures, enabling to scale naturally with modern hardware and model capacity.
Motivated by this idea, we build our approach, WORLDMEM, on top of the Conditional Diffusion Transformer (CDiT) (Peebles and Xie (023) and the Diffusion Forcing (DF) paradigm (Chen et al.) 2025), which autoregressively generates first-person viewpoints conditioned on external action signals. As discussed above, at the core of WORLDMEM is memory mechanism composed of a memory bank and memory attention. ensure efficient and relevant memory retrieval from the bank, we introduce confidence-based selection strategy that scores memory units based on field-of-view

---

(FOV) overlap and temporal proximity. In the memory attention, the latent tokens being generated act as queries, attending to the memory tokens (as keys and values) to incorporate relevant historical context. To ensure robust correspondence across varying viewpoints and time gaps, we enrich both queries and keys with state-aware embeddings. A relative embedding design is introduced to ease the learning of spatial and temporal relationships. This pipeline enables precise, scalable reasoning over long-range memory, ensuring consistency in dynamic and evolving world simulations.
We evaluate WORLDMEM on a customized Minecraft benchmark (aan et al. 2022) and on RealEstate 10K (Zhou al., 2018). The Minecraft benchmark includes diverse terrains (e.g., plains, savannas, and deserts) and various action modalities (movement, viewpoint control, and event trig- gers), which is a wonderful environment for idea verification. Extensive experiments show that WORLDMEM significantly improves 3D spatial consistency, enabling robust viewpoint reasoning and figh-fidelity scene generation, as shown fb). Furthermore, in dynamic environments, WORLDMMM accurately tracks and follows evolving events and environment changes, demonstrating its ability to both perceive and interact with the generated world. We hope our promising results and scalable designs will inspire future research on memory- world simulation.
2 Related Work
Video diffusion model. With the rapid advancement of diffusion models (Song et al. 2020 Peebles
and Xie
2023] Chen et al. 2025), video generation has made significant strides (Wang al.
2023a b
Chen et al.) 2023 Guo et al. 2023 OpenAl 2024] Jin et al.. 2024; Yin et al. 2024).
The field has evolved from traditional b-Net-based architectures (Wang et al. 2023a Chen et al.
2023 Guo et al. 2023) to Transformer-bases frameworks (OpenAl) 2024] Ma et al. [0244] Zheng
al. 2024b), enabling video diffusion models to generate highly realistic and temporally coherent videos. Recently, autoregressive video generation (Chen et al.) 2025 Kim et al. 2024) Henschel
jet al. (2024) has emerged as a promising approach to extend video length, theoretically indefinitely.
Notably, Diffusion Forcing (Chen et al. 2025) introduces a per-frame noise-level denoising paradigm. Unlike the full-sequence paradigm, which applics a uniform noise level across all frames, per- ber-frame noise-level denoising offers more flexible approach, enabling autoregressive generation.
Interactive world simulation. World simulation aims to model an environment by predicting the next state given the current state and action. This concept has been extensively explored in the
construction of world models (Ha and Schmidhuber (018b) for agent learning (Ha and Schmidhuberi 2018a| Hafner et al. 2 2020; Hu et al., [223]] Beattie et al., 2016) Yang et al.) (223)). With advances in video generation, high-quality world simulation with robust control has become feasible, leading to numerous works focusing on interactive world simulation (Bar et al. [024]] Decart et al.,
2024 Alonso et al. 2025 Feng et al. 2024) Parker-Holder et al.] 2024 Valevski el al.) [0224] Yu ct al. 2025c b). These approaches enable agents navigate generated environments and interact
with them based on external commands.
However, due to context window limitations, such methods discard previously generated content, leading to inconsistencies the simulated world, particularly in maintaining 3D spatial coherence.
Consistent world simulation. Ensuring the consistency of generated world crucial for effective world simulation Wang et al. (2025). Existing approaches can be broadly categorized into two types: geometric-based and geometric-free. The brannetrcc-bsedd methods explicitly reconstruct the
generated world into a 3D/4D representation (Liu al. 2024 Gao et al.! 2024; Wang and Agapito 2024 Ren et al. 2025 Yu et al. 2024b]a Liang et al. 20224). While this strategy can reliably maintain consistency, imposes strict constraints on flexibility: Once the world is reconstructed, modifying or interacting with it becomes challenging. Geometric-free methods focus on implicit learning. Methods like Alonso et al. (2025); Valevski et al. (2024) ensure consistency by overfitting to predefined scenarios (e.g., specific CS:GO or DOOM maps), limiting scalability. StreamingT22 (Henschel al. 2024) maintains long-term consistency by continuing on both global and local visual contexts from previous frames, while SlowFastGen (Hong et al., (024) progressively trains LoRA (Hu et al.) 2022) modules for memory recall. However, these methods rely on abstract representations, making accurate scene reconstruction challenging. In contrast, our approach retrieves information from previously generated frames and their states, ensuring world consistency without overfitting to specific scenarios.
3

---

Sittpe
DXT Block
DIT Black
DIT Block
Clear franc from memory bonk
Inference
Sample
Clear ftmn within the contect window
Menory Illock
Block
Block
Novey ftame
Differen: levels nose
Architecture Overview (Inference)
Input Difference
P
Timotarpp
Pttcker limbatding
Sinwoddil Embhoddnng
Laser Noen
Timesip Embrddinx
Scale, Stuft
MLP
MLP
AdaLN
Embeskling
Scale
(6) State Embedding
Memory Block
Figure 2: Comprehensive overview WORLDMEM. The framework comprises a conditional diffusion transformer integrated with memory blocks, with a dedicated memory bank storing memory units from previously generated content. By retrieving these memory units from the memory bank and incorporating the information by memory blocks to guide generation, our approach ensures long-term consistency in world simulation.
WORLDMEM
This section details the methodology of WORLDMEM. Sec. introduces the relevant preliminaries, while Sec. 3.2 describes the interactive world simulator serving as our baseline. Sec. 3.3 and 3.4 present the core of proposed memory mechanism.
3.1 Preliminary
Video diffusion models. Video diffusion models generate video sequences by iteratively denoising Gaussian noise through learned reverse process:
po xx
(x/)
(x uo (x k), of
where all frames (eS) share the same noise level k, and is context window length. This full-sequence approach enables global guidance but lacks flexibility in sequence length and autoregressive generation.
Autoregressive video generation. Autoregressive video generation aims to extend videos over the long term by predicting frames sequentially (Kondratyuk 2024 Wu et al. 2023). While various methods exist for autoregressive generation, Diffusion Forcing (DF) (Chen et al. [2255] provides neat and effective approach achieve this. Specifically, DF introduces per-frame noise levels kG;
pa (
k
x k = Nx -1 (x ki), 0 I),
Unlike full-sequence diffusion, DF generates video flexibly and stably beyond the training horizon. Autoregressive generation is special case when only the last one or few frames are noisy. With autoregressive video generation, long-term interactive world simulation becomes feasible.
3.2 Interactive World Simulation
Before introducing the memory mechanism, we first present our interactive world simulator, which models long video sequences using auto-regressive conditional diffusion transformer. Interaction is achieved by embedding external control signals, primarily actions, into the model through dedicated
conditioning modules (Parker-Holder al. 2024 Decart et al.! 2024 Yu al. 2025c).
Following prior work (Decart et al. 2024). we adopt conditional Diffusion Transformer (DiT) (Pee bles and Xie 2023) architecture for video generation, and Diffusion Forecasting (DF) (Chen et al.

---

2025) for autoregressive prediction. As shown in Figure 2 a), our model consists of multiple DiT blocks with spatial and temporal modules for spatiotemporal reasoning. The temporal module applies causal attention to ensure that each frame only attends to preceding frames.
The actions injected by first projected into the embedding space using multi- perceptron (MLP). The resulting action embeddings are added to the denoising timestep embeddings and injected into the temporal blocks using Adaptive Layer Normalization (AdaLN) (Xu et 2019), following the paradigm of Barr et al. (202)); Decart et al. (2024). In our Minecraft experiments, the action space contains 25 dimensions, including movements, view adjustments, and event triggers. We also apply timestep embeddings to the spatial blocks in the same manner. although this is omitted from the figure for clarity. Standard architectural components such as residual connections, multi-head attention, and feedforward networks are also not shown.
The combination of conditional DiT' and DF provides strong baseline for long-term interactive video generation. However, due to the computational cost of video synthesis, the temporal context window remains limited. As result, content outside this window is forgotten, which leads to inconsistencies during long-term generation (Decart et al. 2024).
3.3 Memory Representation and Retrieval
To address the limited context window of video generative models, we introduce a memory mechanism that enables the model retain and retrieve information be- yond the current generation window. This mechanism maintains memory bank com- posed of historical frames and their associ- ated state information: {xx"" Pr, t=)) where x denotes a memory frame, Pi E Re.5 (x, y. z, pitch, yaw) is its pose, and 6; is the timestamp. Each tuple is referred to as a memory unit. We save m; in token-level, which is compressed by the visual encoder but retains enough details for reconstruc- tion. The corresponding states {(p,t) t}} play a critical role not only in memory retrieval but also in enabling state-aware memory conditioning.
Algorithm 1: Memory Retrieval Algorithm
Input: Memory bank of N historical states
{(x!", Pi, ()) Current state (xe, fe); fe) memory condition length LM;;
Similarity threshold tr; weights Wap Wit. Output: A list of selected state indices S Compute Confidence Score:
Compute FOV overlap ratio o via Monte Carlo sampling. Compute time difference d Concat {1 - (=])==1) Compute confidence a = o d wt.
Selection with Similarity Filtering: Initialize S = Ø
for m = 1 to LM do Select i" with highest a,-
Append j to S
Remove all j where similarity(i", ,j) > tr
return S
Memory Retrieval. Since the number of memory frames available for conditioning is limited, an efficient strategy required sample memory units from the memory bank. We adopt a greedy matching algorithm based frame-pair similarity, where similarity is defined using the field-of- view (FOV) overlap ratio and timestamp differences as confidence measures. Algorithm presents our approach to memory retrieval. Although simple, this strategy proves effective in retrieving relevant information for conditioning. Moreover, the model's reasoning over memory helps maintain performance even when the retrieved content is imperfect.
3.4 State-aware Memory Condition
After retrieving necessary memory units, unlike prior methods that use memory mainly for temporal smoothness (Zheng et al.) 2024a) or semantic guidance (Wu et al. 2025b Rahman et al.) 2023), our goal is to explicitly reconstruct previously seen visual content or scene changes. This requires the model to perform spatiotemporal reasoning to extract relevant information from memory, which we model using cross-attention (Vaswani al. 2017). Since relying solely on visual tokens can be ambiguous, we incorporate the corresponding states as cues to enable state-aware attention.
under significant viewpoint
State Embedding. State embedding provides essential spatial and temporal context for memory retrieval. To encode spatial information, we adopt Plücker embedding (Sitzmann et al. 2021 1) to convert 5D poses p E RR5 into dense positional features PE(p) E k following (He et al. 2024 Gao et al. 2024 Temporal context is captured via a lightweight MLP over sinusoidal embedded
5

---

Ous
GT
Oues
GT
Ours
GT
600 Frame
700 frame
Ouss
Initialize
hay
Place hay
Place bay
Move around ...
Tum left
Ouss
Initialize
Dig
Plant
Move around
Turn left
Figure 3: Qualitative results. We showcase WORLDMEM'S capabilities through two sets of examples. Top: A comparison with Ground Truth (GT). WORLDMEM accurately models diverse dynamics (e.g., rain) by conditioning 600 past frames, ensuring temporal consistency. Bottom: Interaction with the world. Objects like hay in the desert or wheat the plains persist over time, with wheat visibly growing. For the best experience, see the supplementary videos.
(SE) timestamps. The final embedding is (Figure (c)):
E (PE(()) + G(B(())),
where Gp and G2 MLPs mapping pose and time into shared space.
State-aware Memory Attention. To support reconstruction under viewpoint and temporal shifts, we introduce state-aware attention mechanism that incorporates spatial-temporal cues into memory retrieval. By conditioning attention on both visual features and state information, the model achieves more accurate reasoning between input and memory.
Let X, E xd xd denote the flattened feature map of input frames (queries), and X, E R4xxd the concatenated memory features (keys and values). We first enrich both with their corresponding state embeddings E, and Eg:
x,
E X
X
+ Eg-
Cross-attention is then applied to retrieve relevant memory content and output updated X':
X = CrossAttn(Q = (X,), K = (x())) V = P. (Xx)),
(5)
where Pg. Pk. and Pe are learnable projections.
To simplify the reasoning space, we adopt relative state formulation. For each query frame, the state is set to zero reference (e.g., the pose is reset to the identity and the timestamp to zero), while the states of key frames are normalized to relative values. This design, illustrated in Figure (d)), improves alignment under viewpoint changes and simplifies the learning objective.

---

Ous DF
Full
DF
Ouns
Initialize
Tum right
Tum left
Frame Frame
Frame
Figure 4: Within context window evaluation. The motion sequence involves turning right and returning to the original position, showing self- contained consistency.
Figure 5: Beyond context window evalua- tion. Cuffusion-Forcing suffers inconsistency over time, while ours maintains quality and re- covers past scenes.
Table I: Evaluation Minecraft
Table 2:
Ablation on embedding designs
Within context window
Pose type
Embed. type
PSNR
LPIPS |
rFID +
Methods
PSNR
LPIPS L
rFID 1
Sparse
Absolute
20.67
0.2887
39.23
Dense Dense
Absolute
23.63
0.1830
29.34
Full Seq.
20.14
0.0691
13.87
Relative
23.98
0.1429
15.37
DF
24.11
0.0094
13.88
Ours
25.98
0.0072
13.73
Table 3: Ablation memory retrieve strategy
Beyond context window
Methods
PSNR
LPIPS
rFID 1
Strategy
PSNR f
LPIPS 4
rFID +
Full Seq. DF Ours
Random
18.32
0.3224
47.35
17.32 23.98
0.4376
51.28
Confidence Filter
23.12
0.1863
24.33
0.1429
15.37
SSmilarity Filter
23.98
0.1429
15.37
Incorporating memory into pipeline. We incorporate memory frames into the pipeline by treating them as clean inputs during both training and inference. As shown in Figure ( during training, memory frames are assigned the lowest noise level kmin- while context window frames receive independently sampled noise levels from the range [Rmin kmmax] During inference, both memory and context frames are assigned kiiin while the current generating frames are assigned kmax-
To restrict memory influence only to memory blocks, we apply a temporal attention mask:
1, LM and j
Amask (i.
1, i> LM and j S
(6)
0, otherwise
where Las is the number of memory frames that are appended before frames within the context window. This guarantees causal attention while preventing memory units from affecting each other.
4 Experiments
Datasets. We use MineDojo (Fan et al. 2022) to create diverse training and evaluation datasets in Minecraft, configuring diverse environments (e.g., plains, savannas, ice plains, and deserts), agent actions, and interactions. For real-world scenes, we utilize RealEstatelOK (Zhou et al. 2018) with camera pose annotations to evaluate long-term world consistency.
Metrics. For quantitative evaluation, we employ reconstruction metrics, where the method of obtaining ground truth (GT) varies by specific settings. We then assess the consistency and quality of the generated videos using PSNR, LPIPS (Zhang et al. 2018), and reconstruction FID (rFID) (Heusel al. 2017), which collectively measure pixel-level fidelity, perceptual similarity, and overall realism.
Experimental details. For our experiments on Minecraft (Fan et al., 2022). we utilize the Oasis (Decart et al. 2024) the base model. Our model is trained using the Adam optimizer with fixed

---

Camerol 2tt
Viesocrafter
OuiS
Figure 6: Results RealEstate (Zhou et al. 2018). We visualize loop closure consistency
full camera rotation. The visual similarity between the first and last frames serves as a qualitative indicator of 3D spatial consistency.
Table 4: Evaluation on Reales 10K
Methods
PSNR
LPIPS +
rFID +
CameraCtrl (He al.
2024
13.19
0.3328
133.81
TrajAttn (Xiao
2024
14.22
0.3698
128.36
Viewcrafter Yu al. 2024c
21.72
0.1729
58.43
DFoT (Sng et al. Ours
2025
16.42
0.2933
110.34
23,34
0.1672
43.14
learning rate of x 10 Training is conducted at a resolution 640 x 360, where frames are first encoded into a latent space via VAE resolution of 32 x 18, then further patchified 16 9. Our training dataset comprises approximately 12K long videos, each containing 1500 frames, generated from Fan al. 2022). During training, employ 8-frame temporal context window alongside an 8-frame memory window. The model trained for approximately 500K steps using 4 GPUs, with batch size of 4 per GPU. For the hyperparameters specified in Algorithm of the main paper, set the similarity threshold (r to 0.9, W. to 1, and wy to 0.2/Le. For the noise levels in Eq. (5) and Eq. (6), set kimin to 15 and kmmx 1000.
For our experiments on RealEsttaee 10K (Zhou al. 2018), we adopt DFoT (Song et al. 2025) as the base model. The RealEstate 10K dataset provides training set of approximately 65K short video clips. Training is conducted at a resolution of 256 x 256, with frames patchified to 128 x 128. The model is trained for approximately 50K steps using 4 GPUs, with batch size of per GPU.
4.1 Results Generation Benchmark
Comparisons on Minecraft Benchmark. We compare our approach with a standard full full-sequence (Full Seq.) training method (He al. 2204)) Wang al. 2024) and Diffusion Forcing (DF) (Chen 2025). The key differences are as follows: the full-sequence conditional diffusion transformer (Peebles) and Xie 2023) maintains the same noise level during training and inference, DF introduces different noise levels for training and inference, and our method incorporates memory mechanism. To assess both short-term and long-term world consistency, we conduct evaluations within and beyond the context window. We evaluate both settings on 300 test videos. In the following experiments, the agent's poses are generated by the game simulator as ground truth. However, in real-world scenarios, only the action input is available, and the pose is not directly observable. In such cases, the next-frame pose can be predicted based on the previous scenes, past states, and the upcoming action. We explore this design choice in the supplementary material.
Within context window. For this experiment, all methods use a context window of 16, while our approach additionally maintains a memory window of 8. We test on customized motion scenarios (e.g., turn left, then turn right or move forward, then backward) assess self-contained consistency, where the ground truth consists of previously generated frames at the same positions. As shown Table T and Figure the full-sequence baseline suffers from inconsistencies even within its own context window. DF improves consistency by enabling greater information exchange among generated frames. Our memory-based approach achieves the best performance, demonstrating the effectiveness of integrating dedicated memory mechanism.

---

Table 5: Ablation on sampling strategy for training
Sampling strategy
PSNR
LPIPS +
rFID +
Small-range 19.23
0.3786
46.55
Large-range
21.11
0.3855
42.96
Progressive 23.98
0.1429
15.37
Beyond context window. In this setting, all methods use context window of 8 and generate 100 future frames; our method further employs a memory window while initializing a 600-frame memory bank. We compute the reconstruction error using the subsequent 100 ground truth frames after 600 frames. Full-sequence methods can not roll out that long so we exclude it. DF exhibits poor PSNR and LPIPS scores, indicating severe inconsistency with the ground truth beyond the context window. Additionally, its low rFID suggests notable quality degradation. In contrast, our memory-augmmentdd approach consistently outperforms others across all metrics, demonstrating superior long-term consistency and quality preservation. Figure further substantiates these findings.
Figure showcases WORLDMEN'S capabilities. The top section demonstrates its ability operate in a free action space across diverse environments. Given a 600-frame memory bank, our model generates 100 future frames while preserving the ground truth's actions and poses, ensuring strong world consistency. The bottom section highlights dynamic environment interaction. By using timestamps as embeddings, the model remembers environmental changes and captures natural event evolution, such as plant growth over time.
Comparisons on Real Scenarios. We compare method with prior works (He al. 2024 Xiao (et al. 2024 Yu et al. [202cc] Song et al.] (025) on the RealEstate 10K dataset (Zhou et al.) 2018). We design 5 evaluation trajectories, each starting and ending the same pose, across 100 scenes. The trajectory lengths range from 37 to 60 frames exceeding the training lengths of all baselines (maximum 25 frames).
CameraCtrl (He et al. 2024). TrajAttn (Xiao et al. 2024). and DFoT (Song et al. 2025) discard past frames and suffer from inconsistency. Viewerafter (Yu et al. 2024c) incorporates explicit 3D reconstruction, yielding better results, but is constrained by errors in post-procsssing such as reconstruction and rendering. As shown in Table and Figure our approach achieves superior performance across all metrics. However, the RealEstate dataset inherently limits the full potential of method, as it consists of short, non-interactive clips with limited temporal complexity. We leave evaluation under more challenging and interactive real-world scenarios for future work.
4.2 Ablation
Embedding designs. The design of embeddings within the memory block is crucial for cross-frame relationship modeling. We evaluate three strategies (Table 2): (1) sparse pose embedding with absolute encoding, (2) dense pose embedding with absolute encoding, and (3) dense pose embedding with relative encoding. Results show that dense pose embeddings (Plücker embedding) significantly enhance all metrics, emphasizing the benefits of richer pose representations. Switching from absolute to relative encoding further improves performance, particularly in LPIPS and rFID, by facilitating absolute embeddings accumulate errors over time, while relative embeddings maintain stability even beyond 300 frames.
relationship reasoning and information retrieval. As illustrated in Figure 7
Sampling strategy for training. We compare different sampling strategies during training in the Minecraft benchmark. Small-range sampling restricts memory conditioning frames within 2m in the Minecraft world, while large-range sampling extends this range to 8m. Progressive sampling, on the other hand, begins with small-range samples for initial training steps and then gradually expands to large-range samples.
As shown in Table both small-range and large-range sampling struggle with consistency and quality, whereas progressive sampling significantly improves all metrics. This suggests that gradually increasing difficulty during training helps the model learn to reason and effectively query information from memory blocks.
Time condition. We ablate the effectiveness of the timestamp condition (for both embedding and retrieval) T We curate 100 video samples featuring placing events and evaluate whether future generations align with event progression. As shown in the table, incorporating the time

---

30
Ours
wio
Relative embed.
PSNR
Memory block
Initbalize
Place bay
Walk around
Look
w/ Rendom
Figure 8: Results w/o and w/ time condition. Without timestamps, the model fails to differ- entiate memory units from the same location at different times, causing errors. With time con- ditioning, it aligns with the updated world state, ensuring consistency.
memory retrieve
800
Frame index
Figure 7: Long-term Generation Comparison. This figure presents the PSNR of different abla- tion methods compared to the ground truth over 300-frame sequence. The results show that our method without memory blocks or using ran- dom memory retrieval exhibits immediate incon- sistencies with the ground truth. Additionally, the model lacking relative embeddings begins to degrade significantly beyond 100 frames. In contrast, our full method maintains strong con- sistency even beyond 300 frames.
Table 6: Ablation on time condition
Time condition
PSNR
LPIPS
rFID +
w/o
23.17
0.1989
23.89
25.12
0.1613
16.53
condition significantly improves PSNR and LPIPS, indicating that adding temporal information helps the model faithfully reproduce event changes in world simulation. Since events like plant growth are inherently unpredictable, we do not conduct quantitative evaluations on such cases but instead provide qualitative illustrations in Figure g
Memory retrieve strategy. We analyze memory retrieval strategies in Table 3 Random sampling from the memory bank leads to poor performance and severe quality degradation, as evidenced by sharp drop rFID and rapid divergence from the ground truth (Figure The confidence- bassed filtering significantly enhances consistency and generation quality. Additionally, we refine retrieval by filtering out redundant memory units based on similarity, further improving all evaluation metrics and demonstrating the effectiveness of our approach.
Limitations and Future works
Despite the effectiveness of our approach, certain issues warrant further exploration. First, we cannot guarantee that we can always retrieve all necessary information from the memory bank In some corner cases (e.g. when views are blocked by obstacles), relying solely on view overlap may be insufficient. Second, our current interaction with the environment lacks diversity and realism. In future work, we plan to extend our models to real-world scenarios with more realistic and varied interactions. Lastly, our memory design still entails linearly increasing memory usage, which may impose limitations when handling extremely long sequences.
Conclusion
In conclusion, WORLDMEM tackles the longstanding challenge of maintaining long-term consistency in world simulation by employing memory bank of past frames and associated states. Its memory attention mechanism enables accurate reconstruction of previously observed scenes, even under large viewpoints temporal gaps, and effectively models dynamic changes over time. Extensive experiments in both virtual and real settings confirm WORLDMEM'S capacity for robust, immersive world simulation. We hope work will encourage further research on the design and applications of memory- based world simulators.
Acknowledgements. This research is supported by the National Research Foundation, Singapore, under its NRF Fellowship Award ENRF-NRFFF16-2224.00 NTU SUG-NAP, as well as cash and in-kind funding from NTU S-Lab and industry partner(s).
This research also supported by
10

---

References
Eloi Alonso, Adam Jelley. Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion world modeling: Visual details matter in atari. Advances Neural Information Processing Systems, 37:58757-58777 2025.
Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models, 2024.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab, arXiv preprint arXiv:1612.0:80) 2016.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081 24125,, 2025.
Haoxin Chen, Menghan Xia, Yingging He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Vidcocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512, 2023.
Decart, Julian Quevedo, Quinn McIntyre, Spruce Camphell, Xinlei Chen, and Robert Wachen. Oasis: A universe in transformer. 2024. Project website.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De- An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances Neural Information Processing Systems, 13:33-1888,, 2022.
Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang. Liu, and Hongyang Zhang. The matrix: intinite-horizan world generation with real-time moving control. arXiv preprint arXiv:2412.00 033.00388, 2024.
Ruiqi Gao, Alcksander Holynski, Philipp Henzler, Arthur Brussce, Ricardo Martin-Bruale Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Crcate anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.10:: 10314 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang. Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint wxXiv:230.004725 2023.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018a.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 10/22, 2018b.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 191200160 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.0/993, 2020.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.00 0010000 2024.
Roberto Henschel, Levon Khachatryan. Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan. Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403. 14773, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Yining Hong. Beide Liu, Maxine Wu, Yuanhao Zhai Kai-Wei Chang, Linjie Li, Kevin Lin, Chung Chang-CChng Lin, Jianfeng Wang. Zhengyuan Yang, Ying Nian Wu, and Lijuan Wang Wang. Slowfast-vgen. Slow-fast learning for action-driven long video generation. arXiv preprint 241.::2410.23277, 2024.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. Gaia-l: 1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.1708) 2023.
11

---

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):, 2022.
Hanwen Jiang, Hao Tan, Peng Wang, Haian Jin, Yue Zhao, Sai Bi, Kai Zhang. Fujun Luan, Kalyan Sunkavalli, Qixing Huang, et al. Rayzer: A self- supervvee large view synthesis model. arXiv preprint arXiv:2505.0070 2025.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint 24X1v:2410.05954 2024.
Jihwan Kim, Junoh Kang, Jinyoung Choi, and Bohyung Han. FIFO-diffusion: Generating infinite videos from text without training. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Dan Kondratyuk, Lijun Yu, Xiuye Gu. José Lezama, Jonathan Huang. Grant Schindler, Rachel Hornung. Vighnesh Birodkar, Jimmy Yan, Ming ChangC Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta. Meera Hahn. Anja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang. Hartwig Adam, Ming-Hsuan Hsuan Yang, Irfan Essa, Huisheng Wang, David A. Ross, Bryan Scybold, and Lu Jiang. Videopoet: large language model for zero-shot vidco generation, 2024.
Hanwen Liang, Junli Cao, Vidit Goel, Guocheng Qian, Sergei Korolev, Demetri Terzopoulos, Konstantinos N Plataniotis, Sergey Tulyakov, and Jian Ren. Wonderland: Navigating 3d scenes from a single image. arXiv preprint arXiv:2412. 12091, 2024.
Fangfu Liu, Wenqiang Sun, Hanyang Wang. Yikai Wang, Haowen Sun. Junliang Ye, Jun Zhang, and Yueqi Duan. Reconx: Reconstruct any scene from sparse views with video diffusion model. arXiv preprint arXiv:2408. 16767, 2024.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for vidco generation. arXiv preprint arXiv:2401..3048, 2024.
OpenAI. Video generation models as world simulators.
video llleoo-gneeaaiin-noone wwwaanmmmelllla--ooooeeeee- www.-aawwwood-smmmaase-ea------e simulatoro 2024.
Jack Parker-Holder, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Christos Kaplanis, Alexan- dre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung, Michael Dennis, Sultan Kenjeyev, Shangbang Long. Vlad Mnih, Harris Chan, Maxime Gazeau, Bonnie Li, Fabio Pardo, Luyu Wang. Lei Zhang, Frederic Besse. Tim Harley, Anna Mitenkova, Jane Wang, Jeff Clune, Demis Hassabis, Raia Hadsell, Adrian Bolton, Satinder Singh, and Tim Rocktäschel. Genie 2: A large-scale foundation world model. 2024.
William Peebles and Saining Xic. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
Tanzila Rahman, Hsin- Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Iconid Sigal. Make-a-story: Visual memory conditioned consistent story generation. In Proceedings of the IEEE/CVF conference on 2023.
computer vision and puttern recognition, pages
Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao. Gen3e: 3d-informed world-consistent video generation with precise camera control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario I. Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 0229-6233 2022.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313 19325,,999.1111 2021.
Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du Russ Tedrake, and Vincent Sitzmann. History- guided video diffusion. arXiv preprint 202204250..06764, 2025.
12

---

Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 3.iv:2011.13455 2020.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408. 14837, 2024.
Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory, arXiv preprint 1100:22408.16611 2024.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang. and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.0657 2023a.
Jing Wang, Fengzhuo Zhang, Xiaoli Li, Vincent YF Tan, Tianyu Pang, Chao Du, Aixin Sun, and Zhuoran Yang. Error analyses of auto-regressive video diffusion models: A unified framework. arXiv preprint 2rXiv:2533.10704 2025.
Yaohui Wang, Xinyuan Chen, Xin Ma. Shangchen Zhou, Ziqi Huang. Yi Wang, Ceyuan Yang. Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309 2023b.
Zhouxia Wang. Ziyang Yuan, Xintao Wang. Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionetrl: A unified and ficxible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
Sibo Wu, Congrong Xu, Binbin Huang, Andreas Geiger, and Anpei Chen. Genfusion: Closing the loop between reconstruction and generation via videos. arXiv preprint arXiv:2503.22219 2025a.
Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. Ar-diffusion: Auto-regressive diffusion model for text generation, 2023.
Xindi Wu, Uriel Singer, Zhaojiang Lin, Andrea Madotto, Xide Xia, Yifan Xu, Paul Crook, Xin Luna Dong, and Seungwhan Moon. Corgi: Cached memory guided video generation. In 2025 IEFE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4585-4594, IEFF, 2025b.
Zeqi Xiao, Wenqi Ouyang, Yifan Zhou, Shuai Yang, Lei Yang, Jianlou Si, and Xingang Pan. Trajectory attention for fine-grained video motion control. arXiv preprint arXiv:2411. 193224, 2024.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances neural information processing systems, 32, 2019.
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint urXiv:2310.06114, 1(2):6, 2023.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast causal video generators. arXiv preprint ar 2r12:2441..07772, 2024.
Hong-Xing Xing Yu, Haoyi Duan, Charles Hermmann William T Freeman, and Jiajun Wu. Wonderworld: Interactive 3d scene generation from single image. arXiv preprint arXiv:2406.09339 2024a.
Hong-XXngg Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, et al. Wonderjourney: Going from anywhere to everywhere. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6658 66667, 2024b.
Jiwen Yu, Yiran Qin, Haoxuan Che, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Hao Chen, and Xihui Liu. A survey of interactive generative video. arXiv preprint arXiv:2504.21853, 2025a.
Jiwen Yu, Yiran Qin, Haoxuan Che, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Position: Interactive generative video as next-generation game engine. arXiv preprint arXiv 17359, 2025b.
Jiwen Yu, Yiran Qin, Xintao Wang. Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos. arXiv preprint arXiv:2501.08325, 2025e.
13

---

Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu. Xiaoyu Li, Zhipeng Huang. Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. Viewerafter: Taming video difflssin models for high-fidelity novel view synthesis. arXiv preprint arXiv:2409.02044 2024c.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
Longtao Zheng. Yifan Zhang, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An. and Shuicheng Yan. Memo: Memory-guider diffusion for expressive talking video generation. arXiv preprint arXiv:2412. 04448, 2024a.
Zangwei Zheng, Xiangyu Peng. Tianji Yang, Cheahui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024b.
Tinghui Zhou, Richard Tucker, John Flynn. Graham Fyffe, and Noah Snavely, Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] " is generally preferable to "[No]" it is perfectly acceptable to answer " "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often
nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims: Long-term consistent world generation with memory.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
is discussed the supplemental material due to page limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
15

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss our experimental setting in the experiments sections. More details are disclosed in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
16

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do not provide open access to the data and code in the supplemental material, but will release codes and datasets publicly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https)
swi.iiossiippcccc.oo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
17

---

Answer: [Yes]
Justification: We specify the experimental setting in the experiment section, and provide more details about training in the supplemental material
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: statistics.
We do not report error bars. However, our results are obtained from large
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide more information about computing resources the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
18

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pss//moor.....ooooooo//da
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
19

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit and use the datasets and codes in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprssiipteeeeoooccppppp thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new datasets in the paper and will release them publicly and document them.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standard components.
Please refer to our LLM policy fattps://neun http:/pppp..p/ooooeeeeee/////22 for what should or should not be described.
21


Reviews:
[{"review_id": "HGE5WeeQ0N", "reviewer": "Reviewer_Zecv", "summary": "This paper presents a novel framework that incorporates a memory bank consisting of state-aware memory cells.  By modeling temporal and spatial states, this approach aims to address the challenge of maintaining long-term consistency in world model video prediction.  The core of the framework utilizes a specially designed memory attention mechanism that enables the memory unit to store not only information from memory frames, but also pose and timestamp information.  The experimental results in virtual scenes are impressive, with PSNR, LPIPS, and rFID metrics evaluated on long sequences across different scenes and interaction modes in the Minecraft game, achieving state-of-the-art (SOTA) performance.  Additionally, the framework demonstrates promising application prospects in the gaming field.  Similarly, on the real-world RealEstate10K dataset, the approach also achieves SOTA performance.", "questions": "1. It would be beneficial to include additional results across a wider range of styles and scene types in both virtual and real-world environments, in order to further demonstrate the generality and robustness of the proposed method.\n\n2. The paper would be strengthened by explicitly stating the practical limitations of the method, including but not limited to memory usage, training time, and inference speed, to better illustrate its real-world applicability.\n\n3. A clear analysis of failure cases, along with a discussion of the underlying reasons for the method's limitations in these scenarios, would provide valuable insight and help guide future improvements.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1: This paper is well-written and clearly articulates the inherent challenge of long-term 3D spatial inconsistency in world generation, which effectively motivates the proposed approach and establishes a clear research direction. The authors introduce a compelling framework based on a state-aware memory attention mechanism, which enhances long-term world generation by explicitly modeling temporal and spatial states, providing a solution to the consistency problem.\n\n2: The method demonstrates strong performance both quantitatively and qualitatively, with comprehensive metrics including PSNR, LPIPS, and rFID that validate the effectiveness of the approach.\n\n3: The visual results are impressive and provide clear evidence of the method's capabilities, showing substantial improvements in maintaining visual quality over extended sequences.\n\n4: The method shows clear advantages in maintaining consistency over long temporal horizons, which is the core challenge the paper aims to address and represents a significant advancement in the field.\n\nWeaknesses:\n\n1: The paper lacks a comprehensive analysis of memory usage and the maximum duration for which high-quality reconstructions can be maintained.   Without this analysis, it remains unclear what the practical limitations and scalability boundaries of the proposed method are, which is crucial for evaluating its real-world applicability.\n\n2: The encoding methods employed for pose and timestamp information appear to follow conventional approaches without significant innovation.   The authors should provide stronger motivation for these design choices or explore more sophisticated encoding schemes that could better capture the temporal-spatial relationships.\n\n3: The experimental evaluation of dynamic scene reconstruction is somewhat limited in scope.   For virtual environments, the demonstration is confined to selected Minecraft scenarios, while real-world evaluation focuses primarily on indoor scenes.   A more diverse set of scenarios, including outdoor environments and different types of dynamic content, would strengthen the generalizability claims of the proposed framework.", "was_revised": true, "final_justification": "The authors introduce a compelling framework based on a state-aware memory attention mechanism, which enhances long-term world generation by explicitly modeling temporal and spatial states, providing a solution to the consistency problem.", "rebuttal": "We thank the reviewer for the detailed and insightful comments. Your feedback on memory usage analysis, encoding strategies, and scenario coverage has helped us better identify the limitations of our current design. We will revise the manuscript accordingly.\n\n---\n\n### **W1 & Q2: Lack of analysis on memory usage, temporal span, and practical constraints**\n\nWe appreciate the reviewer’s concern regarding scalability and real-world applicability. Below, we provide quantitative analysis covering memory usage, generation duration, training cost, and inference efficiency.\n\n- **Memory usage of the memory bank**:  \n  The memory bank is lightweight. Storing 600 visual memory tokens with shape `[600, 16, 18, 32]` in `float32` only takes approximately **21MB**.\n\n- **Generation duration**:  \n  Our method maintains high-quality reconstruction over long sequences. Below are PSNR values at various lengths:\n\n    | Number of Generated Frames | PSNR  |\n  |-------------|-----------|\n  | 100         | 23.1   |\n  | 300         | 22.8   |\n  | 500         | 22.7   |\n  | 800         | 21.4   |\n  | 1000        | 20.8   |\n\n  These results indicate that our method maintains strong reconstruction quality up to 500 frames, and while degradation is observed at 1000 frames, the generation remains structurally coherent without collapse. This demonstrates that our method is capable of handling relatively long sequences, beyond the typical short-range settings explored in prior work.\n\n- **Retrieval latency** (for 8 memory frames), as memory bank scales:\n\n    | Number of Memory Candidates | Retrieval Time (s) |\n    |------------------------|--------------------|\n    | 10                     | 0.04               |\n    | 100                    | 0.06               |\n    | 600                    | 0.10               |\n    | 1000                   | 0.16               |\n\n  The cost of generation (20 denoising steps) is ~0.9 seconds per frame (excluding retrieval). Thus, retrieval contributes only ~10–20% of the inference time, even when accessing 1000 candidates. \n\n\n\n- **Training setting**:\n  - ~500K training steps to converge on 4×H100 GPUs  \n  - Batch size: 4 (the original paper incorrectly reports this and will be corrected)  \n  - GPU memory usage per GPU: ~51GB  \n  - Total training time: ~4 days\n\n- **Cost Compared with Baseline**\n\n  To further address the reviewer’s concern regarding cost and practical feasibility, we include a detailed comparison below between our method and the baseline (without memory), under consistent settings.\n\n  **Setup**:  \n  - 8 context frames + 8 memory frames  \n  - Inference uses 20 denoising steps  \n  - No acceleration techniques (e.g., timestep distillation, sparse attention)\n  - Single H200\n\n  |                 | **Training**                 |                        | **Inference**              |                        |\n  |-----------------|------------------------------|------------------------|----------------------------|------------------------|\n  |                 | Memory Usage                 | Speed (it/s)          | Memory Usage               | Speed (it/s)          |\n  | **w/o memory**  | 33 GB                        | 3.19                   | 9 GB                       | 1.03                   |\n  | **with memory** | 51 GB                        | 1.76                   | 11 GB                      | 0.89                   |\n\n  Introducing memory increases the computational cost during training, with a moderate rise in memory usage and a reduction in training speed. However, the overhead during inference is relatively minor, with only a small increase in memory usage (from 9 GB to 11 GB) and a slight drop in speed (from 1.03 it/s to 0.89 it/s). Overall, the additional cost introduced by memory is acceptable, particularly during inference, where the impact is minimal.\n\n  Moreover, with modern acceleration techniques, such as timestep distillation, early exit, or sparse attention, the inference speed can be further improved to real-time levels (i.e., approximately 10 FPS), making the approach practical for deployment.\n\n\n\n---\n\n### **W2: Conventional encoding of pose and timestamp**\n\nWe intentionally adopt standard encoding schemes to represent camera pose and timestamp. This choice ensures better generalization across environments and reduces inductive bias, making our framework more adaptable. While these encodings perform well in our current setting, we agree that more expressive alternatives may further improve performance. Also, such complex designs may incur significantly higher training and inference costs. We consider more sophisticated designs a promising direction for future exploration.\n\n---\n\n### **W3 & Q1: Limited dynamic and real-world scenario evaluation**\n\nOur current experiments focus on synthetic environments and small-scale real-world scenes, primarily due to resource limitations. Nonetheless, our framework is conceptually applicable to real-world scenarios, provided that sufficient sensory input and temporal alignment are available. \n\nWhile our current setup does not include dynamic elements such as moving entities (e.g., zombies), this omission is mainly due to the significant computational cost required to train a video model capable of generating such dynamics with high fidelity. Importantly, our model architecture is inherently compatible with dynamic content. Each memory entry is timestamped, enabling the model to learn temporal evolution. For instance, recognizing that a walking zombie at time  $t_i$ should appear at a different location at time $t_{i+n}$​. This capability is a core motivation behind our design of state-aware memory, which we plan to further explore in future work involving dynamic scenes.\n\nWe agree that demonstrating results on more diverse environments, including outdoor and highly dynamic scenes, would strengthen our claims. We consider this an important direction for future work and are working toward scaling the framework to support such settings.\n\n---\n\n### **Q3: Failure cases and limitations**\n\nWe appreciate the reviewer’s suggestion and will include a dedicated limitations and failure case section in the revised paper. In fact, we have discussed several of these issues in the supplementary and will lift them into the main paper. Below, we summarize key failure modes observed in our current system:\n\n- **Retrieval under occlusion**: Our current memory retrieval strategy is based on view overlap, assuming that overlapping FoVs imply semantic relevance. However, this assumption breaks in the presence of occluders. For example, if the current viewpoint faces a wall, the retrieved frames with similar FoVs may be located behind the wall and thus contain irrelevant content. Our method does not yet model occlusion-aware visibility, which limits retrieval accuracy in cluttered environments.\n\n- **Long-horizon generation degradation**: While our method supports relatively long sequences (e.g., up to 1000 frames), it is still subject to common issues in autoregressive generation, such as quality degradation and error accumulation. As the sequence length increases, we observe texture blurring, loss of spatial detail, and occasional color artifacts (e.g., patchy or unnatural regions), especially beyond 1000 frames. These issues are consistent with failure modes observed in other long-range video generation systems.\n\nWe consider addressing these limitations in future work, such as incorporating visibility-awareness into memory retrieval and exploring strategies to improve long-horizon consistency.\n\n---\n\nWe hope our responses have addressed your concerns. If any points remain unclear, we would be happy to further clarify and discuss."}, {"review_id": "OkttNdZATN", "reviewer": "Reviewer_xLdM", "summary": "This paper proposes to integrate a memory bank with a video model to enable more consistent\nlong-horizon video simulation. To do this, the  authors construct a memory bank of past \nposes, times and images that the agent has seen. At generation time, relevant frames to the current\nlocation of the agent is achieved through FoV overlap, which are then used to render future frames.", "questions": "1) How would the approach illustrated in the paper work for dynamically changing scenes? I.e. in minecraft, there is a zombie that is moving around and disappears when the sun comes out. While the timestep conditioning could help model dynamics for a short amount of time, it seems like a more explicit dynamics model might be needed for large scale dynamic change, where an entity will move from its viewing context in the memory to a very different frame.\n2)  Since the paper is based off diffusion forcing, would the history guidance method in [1] also help improve the performance of the method?\n3)  The paper uses FoV overlap to compute relevant frames in a memory bank to retrieve. However, in many settings, we may not know the current pose of the agent and the past frames may not form a consistent world model (imagine learning a world model for the wrist camera of a moving robot) How would the authors imagine generalizing their method in this case?\n\n[1] Song et al. History-Guided Video Diffusion.", "limitations": "I did not find any discussion of the limitations of the approach. One big limitation might be that the retrieval mechanism is likely significantly slower than directly generating video.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper studies an important and timely problem\n- The results seem significant \n- The method is clearly explained and well motivated\n\nWeaknesses \n- The memory bank selector is hard-coded, preventing the approach from effectively working in settings where we do know the ground truth pose of the agent and where there isn't a well defined 2D map of the scene (i.e. in a multilevel house, the FoV overlap wouldn't make sense because multiple floors would have the same overlap)\n- All the results in the paper are done either in Minecraft or on small baseline Realestate 10k images. It would be good to illustrate the applicability of the method on more real world scenes (it seems the method is heavily limited by the need of knowing precise camera poses in a 2D floormap situation)\n- All results only show the visual generation results, without any illustrations of the application of the method to downstream decision-making tasks. For instance, can you use the model with MPC to make decisions in minecraft? \n- There is a lot of work in interactive world simulation and the authors have cited a limited number of works in the area. The authors should try to cite a more comprehensive set of existing works, for instance, [1] should be discussed. \n- The results on the website aren't fully 3D consistent (i.e. in Minecraft, the terrain of the ground is changing)\n\n[1] Yang et al. Learning Interactive Real World Simulators", "was_revised": true, "final_justification": "I remain positive about the paper.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We appreciate the detailed analysis and helpful suggestions on memory design, downstream applications, and generalization. We will revise the paper accordingly.\n\n---\n\n### **W1: The memory bank selector is hard-coded**\n\nWe agree that our current FoV-overlap-based memory retrieval is heuristic and may not generalize well to complex 3D environments (e.g., multi-level buildings). This limitation is acknowledged in the paper (Line 2-5 in Supp.). Nevertheless, it serves as a proof-of-concept showing that structured spatial priors can support long-term consistent world modeling. Despite its simplicity, the approach works well in controlled settings such as Minecraft and indoor scans. \n\n---\n\n### **W2: Limited applicability to real-world scenes**\n\nOur current experiments focus on synthetic and small-scale real scenes, mainly due to resource constraints. However, our framework is conceptually applicable to real-world scenarios as long as sufficient sensory input and temporal alignment are available. While relying on known camera poses may seem restrictive, such information is available in many practical settings, such as gaming, robotics, and embodied scanning, via built-in APIs or SLAM systems. As pose estimation tools (e.g., Vggt [1]) continue to improve, this constraint becomes increasingly manageable. We also plan to explore retrieval based on learned visual representations to further reduce dependence on explicit pose information.\n\n---\n\n### **W3: No illustration of downstream decision-making tasks**\n\nThank you for the suggestion. Although our current work focuses on generating quality and consistency, our model is naturally compatible with downstream decision-making frameworks such as Model Predictive Control (MPC). By simulating temporally grounded future observations, our model can be used to evaluate outcomes of candidate action sequences and support action selection. For example, in Minecraft, this allows integration into planning and control pipelines. While we have not demonstrated this in the current work, we consider it an important direction for future research.\n\n---\n\n### **W4: Missing related work in interactive world simulation**\n\nWe appreciate the reviewer highlighting this. We will extend our related work section to include recent contributions on interactive and long-context video world models, including but not restricted to [2,3,4,5].\n\n---\n\n### **W5: Incomplete 3D consistency in generation**\n\nWe acknowledge that some generation results, especially in long sequences, may not achieve perfect 3D consistency (e.g., terrain deformation). This remains a general challenge in world modeling. Nonetheless, to the best of our knowledge, our method achieves state-of-the-art consistency compared to existing methods, including very recent approaches [6,7]. We are actively exploring ways to further improve consistency.\n\n---\n\n### **Q1: Modeling dynamic scene changes**\n\nThis is an important and challenging direction. Our current experiments do not include moving entities (e.g., zombies) primarily due to the computational cost to train a video model that can generate such dynamics with high quality. However, our model architecture is compatible with such dynamics. Since each memory entry is associated with a timestamp, the model can learn temporal evolution, e.g., understanding that a walking zombie at time $t_i$ should appear in a different location at time $t_{i+n}$. This is a key motivation behind our design of *state-aware memory*, which we plan to explore further with dynamic scenes in future work.\n\n---\n\n### **Q2: Integration of history guidance**\n\nYes, history guidance [8] techniques are complementary to our method and can improve temporal coherence and visual fidelity. In our RealEstate10K experiments, we have already applied history guidance to stabilize generation (Table 4 in the main paper). These techniques are orthogonal and can be integrated seamlessly to further enhance results.\n\n---\n\n### **Q3: Generalization beyond pose-based retrieval**\n\nOur method leverages the broader concept of *state-aware memory*, with pose and timestep serving as a specific instantiation of the state. In scenarios where explicit poses are unavailable (e.g., egocentric wrist cameras), the same principle applies: define the state (e.g., the position and the rotation angle of robotic arms, past visual input), and learn associations between current states and stored memory. With sufficient training data, we believe a learned retrieval function can effectively generalize to such unstructured environments.\n\n---\n\n### **Limitation: Retrieval latency vs. direct generation**\n\nThank you for highlighting this limitation. While retrieval does introduce some latency, it remains minor even at scale. Below is the empirical timing on an H100 GPU for retrieving 8 memory frames from different memory pool sizes:\n\n| Number of Memory Candidates | Retrieval Time (s) |\n|------------------------|--------------------|\n| 10                     | 0.04               |\n| 100                    | 0.06               |\n| 600                    | 0.10               |\n| 1000                   | 0.16               |\n\nThe cost of generation (20 denoising steps) is ~0.9 seconds per frame (excluding retrieval). Thus, retrieval contributes only ~10–20% of the inference time, even when accessing 1000 candidates. \n\nWe believe this is a worthwhile trade-off, as retrieval enhances long-term coherence by anchoring generation in relevant context, which pure autoregressive or diffusion models struggle to maintain. We will move this limitation and analysis from the supplementary to the main paper as suggested.\n\n\n---\nWe hope our responses have addressed your concerns. If any points remain unclear, we would be happy to further clarify and discuss.\n\n---\n\n[1] Wang J, Chen M, Karaev N, et al. Vggt: Visual geometry grounded transformer.\n\n[2] Yang M, Du Y, Ghasemipour K, et al. Learning interactive real-world simulators.\n\n[3] Yu J, Qin Y, Che H, et al. A survey of interactive generative video.\n\n[4] Beattie C, Leibo J Z, Teplyashin D, et al. Deepmind lab.\n\n[5] Hafner D, Pasukonis J, Ba J, et al. Mastering diverse domains through world models.\n\n[6] Chen T, Hu X, Ding Z, et al. Learning World Models for Interactive Video Generation.\n\n[7] Po R, Nitzan Y, Zhang R, et al. Long-context state-space video world models.\n\n[8] Song K, Chen B, Simchowitz M, et al. History-guided video diffusion"}, {"review_id": "GJnEwc7NQa", "reviewer": "Reviewer_7GuY", "summary": "this paper proposes a method to use past visual and state information as a kind a memory to generated videos with improved content consistency. useful information is retrieved and injected through cross-attention.", "questions": "1. this paper proposed to use cross-attention to make use of the memory information, which treats the retrieved memory as condition (like input text). what if the retrieved memory is incomplete or flawed? i suspect using cross-attention may cause the whole generation model to be sensitive to errors of memory retrieval. please add more analyses on the pros and cons of using the cross-attention mechnism.\n2. figure 2(d) is not clear. in the \"State-aware Memory Attention\" part, how the cross attention is performed when clear frames within the context window and clear frames within the memory bank is the input?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strength:\nthis paper is well-written and easy to follow, and the proposed method does alleviates the content consistency problem in the experiment settings of this paper.\n\nweaknesses:\nin the experiments, the generated video has quite limited length - 16 to 100 frames - which is not a typical scenario for showing the content consistency issue for long video generation (minecraft game video in this paper). for longer (minute level) video generation, how will the performance of the propose method be?", "was_revised": true, "final_justification": "the rebuttals addressed my concerns to a large extent. i will keep my original rating.", "rebuttal": "We sincerely thank the reviewer for the valuable suggestions and sharp observations. Your comments on long-horizon generation, robustness of memory conditioning, and architectural clarity are greatly appreciated. We will revise the paper to include more details.\n\n---\n\n### **W1: Limited generation length**\n\nTo further assess long-range consistency, we provide the following PSNR results for longer sequences:\n\n| Number of Generated Frames | PSNR  |\n|-------------|--------|\n| 100         | 23.1   |\n| 300         | 22.8   |\n| 500         | 22.7   |\n| 800         | 21.4   |\n| 1000        | 20.8   |\n\nThese results indicate that our method maintains strong reconstruction quality up to 500 frames, and although quality degrades at 1000 frames, the generation does not collapse and remains structurally reasonable. This demonstrates that our method is capable of handling relatively long sequences, beyond the typical short-range settings explored in prior work.\n\n\n\n---\n\n### **Q1: On the robustness and design of memory cross-attention**\n\nThank you for the thoughtful question. Our use of cross-attention to incorporate memory is motivated by a balance between robustness and efficiency.\n\n- **Robustness to noisy memory**: During training, memory is sampled randomly from large video pools, which often include unrelated frames. The model thus learns to selectively attend to informative content while ignoring irrelevant memory. This naturally increases robustness to retrieval imperfections.\n\nUsing cross-attention for memory conditioning is not strictly necessary. One could alternatively apply full self-attention over both the current frame and the memory. However, we adopt cross-attention for the following reasons:\n\n- It introduces a clear directional inductive bias, where the current denoising frame attends to the memory without allowing information to flow back.\n- It significantly reduces computational and memory cost, particularly when the context or memory length is large. \n\nThus, the training and inference would both be more effective and efficient.\n\nThat said, cross-attention is less standard than full self-attention in typical Transformer architectures. Full self-attention offers greater capacity and flexibility for learning rich interactions between memory and context, which might be suitable for large-scale training.\n\n\n\n\n---\n\n### **Q2: Clarification on Figure 2(d)**\n\nWe appreciate your comments. In the \"State-aware Memory Attention\" module, we use all tokens, including clean frames from the context window, clean frames from the memory bank, and noisy frames, as queries. The clean frames within the memory bank are used as keys and values in the cross-attention operation. We will revise Figure 2(d) to highlight the query-key-value assignments more clearly, including visual markers for which tokens are used in each role.\n\n\n---\n\n\nWe hope our responses have addressed your concerns. If there are any unclear points or remaining questions, we would be happy to further clarify and discuss."}, {"review_id": "uN8zlm044v", "reviewer": "Reviewer_sC3T", "summary": "The manuscript introduces a key-frame based memory model for image-based world models. The memory consists of previous state anchored in space via camera pose and time. When generating a new frame, first a set of memory/key frames are retrieved based on viewpoint overlap and relative time. The next frame is generated using these additional frames for context. This approach proves very effective in generating videos beyond the training time horizon and performs better than related works with fixed context windows or that use reconstruction to fuse a memory representation.", "questions": "Fig 6: it is hard to tell what the expected frames should be - it would be good to add some GT frames here to anchor and be able to compare.", "limitations": "limitations are not written in the main paper. The limitations summarized in the supplement are solid. Especially the linear memory complexity. Consider lifting them to the main paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Overall I am very excited about this line of work and the very promising results in generating consistent video over long time horizons with the flexible key-frame-based memory setup. I think this is a significant paper due to the simplicity of the approach and the demonstrated performance. \n\n- Strengths:\n    - The key-frame-based memory approach is flexible and induces minimal inductive biases. Only the most fundamental quantities of space and time are used. The experiments on 600 frame memory context and generation of 100 frames into the future demonstrate very clearly the value and power of the approach in long term generation. \n    - The qualitative evidence that temporal dynamics like growing grass can be generated based on the world memory is exciting. It will be very interesting to see if more intricate dynamics can be generated as well in future work.\n    - The paper is well and clearly written with good illustrations to help convey the message and the approach. Illustrating video generations in picture captions is hard. The supplemental videos are really helpful in this regard. \n    - The experiments both quantitatively and qualitatively support the claims and design choices well. In particular the long-term generation beyond the context length is a powerful signal (Tab 1 and Fig 7). \n- Weaknesses:\n    - One of the main limitations and potential weaknesses for very long time horizon generation of this approach is the linear memory growth since all past frames are stored into the memory bank. Reconstruction-based approaches can theoretically overcome this limitation but as shown in the paper lead to lower quality generations. \n    - Currently WorldMem gets 16+8 frame input and related work gets only 16. This makes it not 100% comparable. I think we need to see an ablation where WorldMem gets 8+8 frames input to get the same number of frames. This would clearly show the difference of getting 8 frames from the memory bank vs getting just 8 frames from the past (other methods). \n    - I would have appreciated a little more detail on the per frame noise levels used for generation and how those affect the outcome.", "was_revised": true, "final_justification": "I have read the rebuttal and the other reviews.\n\nI appreciate the authors clarifications and am happy to stay with my original rating of accept. I did not see anything major in the other reviews either to change my positive rating.\n\nI trust that the authors will include the additional studies on retrieval duration, long generation and ablation of the number of context frames into the publication.", "rebuttal": "We thank the reviewer for the thoughtful feedback and helpful suggestions regarding fairness of comparisons, memory scaling, and evaluation clarity. We will revise the paper accordingly.\n\n---\n\n### **W1: Linear memory growth**\n\nWe agree that the memory bank grows linearly with video length. However, in practice, we find the cost to be manageable:\n\n| Number of Memory Candidates | Retrieval Time (s) |\n|------------------------|--------------------|\n| 10                     | 0.04               |\n| 100                    | 0.06               |\n| 600                    | 0.10               |\n| 1000                   | 0.16               |\n\nThe memory bank itself is lightweight. For example, storing 600 visual memory tokens of shape `[600, 16, 18, 32]` in `float32` takes only about **21MB**. Furthermore, since memory tokens are detached and not used in backpropagation during inference, the actual GPU memory footprint is low. Retrieval time increases slowly with memory size, making our method scalable in practice even for long sequences.\n\n---\n\n### **W2: Input length mismatch in comparison**\n\nThank you for pointing this out. While our method uses 16 context frames plus 8 retrieved memory frames, other baselines typically use 16 continuous past frames. We agree this makes direct comparison not entirely fair. However, it is worth noting that the primary role of context frames in our method is to maintain short-term temporal smoothness, rather than to provide rich content information, since we have memory as context.\n\nWe provide the following ablation to show how the number of context frames affects performance. The results suggest that as long as the context includes a few relevant frames (e.g., more than 3), the generation quality remains stable:\n\n| Number of Context Frames | PSNR  |\n|---------------------|--------|\n| 1                   | 23.12  |\n| 4                   | 27.09  |\n| 8                   | 26.84  |\n| 16                  | 27.01  |\n\n\n\n---\n\n### **W3: Per-frame noise level**\n\nWe adopt the standard diffusion forcing setup, where each frame is trained with independently sampled noise levels. During training, we do not perform specific ablations to optimize per-frame-level noise strategies. Instead, we follow the default setting where each frame is independently perturbed with randomly sampled noise levels. This setup simplifies training and encourages the model to generalize across a range of noise conditions.\n\nOur main motivation for using per-frame noise levels is to enable more flexible control during inference. In particular, we apply an **autoregressive inference strategy**, where frames are denoised sequentially.\n\n\n---\n\n### **Q1: Missing ground truth in Figure 6**\n\nThank you for the suggestion. As this is a synthetic rotation trajectory not observed in training data, there is no ground truth to compare with. Instead, the goal of Fig. 6 is to visualize loop closure consistency over one full rotation, where the visual similarity between the first and last frames provides a qualitative indicator of 3D spatial consistency. We will make this clearer in the revised version.\n\n---\n\n### **Limitations missing in main paper**\n\nWe appreciate this suggestion. We will lift this section into the main paper in the revised version.\n\n\n\n---\n\nWe hope our responses have addressed your concerns. If there are any unclear points or remaining questions, we would be happy to further clarify and discuss."}]

Author rebuttal:

Meta-review: The authors introduce an approach to view generation for longer-sequence world modelling, in which a memory bank is used of past frames and states (including time and pose). The approach reaches state-of-the-art performance on MineDojo and RealEstate. 

Multiple issues were raised and addressed to the satisfaction of the reviewers, including the scalability of the method in terms of memory and retrieval latency, and the generation of longer-term videos. Other issues are rightly delegated to future work, such as the application of the method to more diverse worlds or more dynamic scenes.

Wit the incorporation of a more elaborate discussion of the limitations in the main manuscript (scaling limitations, dependence on ground-truth) and clarifications on the attention modules and noise properties, this should form a valuable contribution to NeurIPS.
Decision: Accept (poster)