Title: WorldMem: Long-term Consistent World Simulation with Memory
Venue: NeurIPS 2025
Accepted: yes
Keywords: world simulation

memory
Abstract: World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing state-aware memory attention that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript introduces a key-frame based memory model for image-based world models. The memory consists of previous state anchored in space via camera pose and time. When generating a new frame, first a set of memory/key frames are retrieved based on viewpoint overlap and relative time. The next frame is generated using these additional frames for context. This approach proves very effective in generating videos beyond the training time horizon and performs better than related works with fixed context windows or that use reconstruction to fuse a memory representation.\n\nSTRENGTHS AND WEAKNESSES:\nOverall I am very excited about this line of work and the very promising results in generating consistent video over long time horizons with the flexible key-frame-based memory setup. I think this is a significant paper due to the simplicity of the approach and the demonstrated performance. \n\n- Strengths:\n    - The key-frame-based memory approach is flexible and induces minimal inductive biases. Only the most fundamental quantities of space and time are used. The experiments on 600 frame memory context and generation of 100 frames into the future demonstrate very clearly the value and power of the approach in long term generation. \n    - The qualitative evidence that temporal dynamics like growing grass can be generated based on the world memory is exciting. It will be very interesting to see if more intricate dynamics can be generated as well in future work.\n    - The paper is well and clearly written with good illustrations to help convey the message and the approach. Illustrating video generations in picture captions is hard. The supplemental videos are really helpful in this regard. \n    - The experiments both quantitatively and qualitatively support the claims and design choices well. In particular the long-term generation beyond the context length is a powerful signal (Tab 1 and Fig 7). \n- Weaknesses:\n    - One of the main limitations and potential weaknesses for very long time horizon generation of this approach is the linear memory growth since all past frames are stored into the memory bank. Reconstruction-based approaches can theoretically overcome this limitation but as shown in the paper lead to lower quality generations. \n    - Currently WorldMem gets 16+8 frame input and related work gets only 16. This makes it not 100% comparable. I think we need to see an ablation where WorldMem gets 8+8 frames input to get the same number of frames. This would clearly show the difference of getting 8 frames from the memory bank vs getting just 8 frames from the past (other methods). \n    - I would have appreciated a little more detail on the per frame noise levels used for generation and how those affect the outcome.\n\nQUESTIONS:\nFig 6: it is hard to tell what the expected frames should be - it would be good to add some GT frames here to anchor and be able to compare.\n\nLIMITATIONS:\nlimitations are not written in the main paper. The limitations summarized in the supplement are solid. Especially the linear memory complexity. Consider lifting them to the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
this paper proposes a method to use past visual and state information as a kind a memory to generated videos with improved content consistency. useful information is retrieved and injected through cross-attention.

STRENGTHS AND WEAKNESSES:
strength:
this paper is well-written and easy to follow, and the proposed method does alleviates the content consistency problem in the experiment settings of this paper.

weaknesses:
in the experiments, the generated video has quite limited length - 16 to 100 frames - which is not a typical scenario for showing the content consistency issue for long video generation (minecraft game video in this paper). for longer (minute level) video generation, how will the performance of the propose method be?

QUESTIONS:
1. this paper proposed to use cross-attention to make use of the memory information, which treats the retrieved memory as condition (like input text). what if the retrieved memory is incomplete or flawed? i suspect using cross-attention may cause the whole generation model to be sensitive to errors of memory retrieval. please add more analyses on the pros and cons of using the cross-attention mechnism.
2. figure 2(d) is not clear. in the "State-aware Memory Attention" part, how the cross attention is performed when clear frames within the context window and clear frames within the memory bank is the input?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to integrate a memory bank with a video model to enable more consistent\nlong-horizon video simulation. To do this, the  authors construct a memory bank of past \nposes, times and images that the agent has seen. At generation time, relevant frames to the current\nlocation of the agent is achieved through FoV overlap, which are then used to render future frames.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper studies an important and timely problem\n- The results seem significant \n- The method is clearly explained and well motivated\n\nWeaknesses \n- The memory bank selector is hard-coded, preventing the approach from effectively working in settings where we do know the ground truth pose of the agent and where there isn't a well defined 2D map of the scene (i.e. in a multilevel house, the FoV overlap wouldn't make sense because multiple floors would have the same overlap)\n- All the results in the paper are done either in Minecraft or on small baseline Realestate 10k images. It would be good to illustrate the applicability of the method on more real world scenes (it seems the method is heavily limited by the need of knowing precise camera poses in a 2D floormap situation)\n- All results only show the visual generation results, without any illustrations of the application of the method to downstream decision-making tasks. For instance, can you use the model with MPC to make decisions in minecraft? \n- There is a lot of work in interactive world simulation and the authors have cited a limited number of works in the area. The authors should try to cite a more comprehensive set of existing works, for instance, [1] should be discussed. \n- The results on the website aren't fully 3D consistent (i.e. in Minecraft, the terrain of the ground is changing)\n\n[1] Yang et al. Learning Interactive Real World Simulators\n\nQUESTIONS:\n1) How would the approach illustrated in the paper work for dynamically changing scenes? I.e. in minecraft, there is a zombie that is moving around and disappears when the sun comes out. While the timestep conditioning could help model dynamics for a short amount of time, it seems like a more explicit dynamics model might be needed for large scale dynamic change, where an entity will move from its viewing context in the memory to a very different frame.\n2)  Since the paper is based off diffusion forcing, would the history guidance method in [1] also help improve the performance of the method?\n3)  The paper uses FoV overlap to compute relevant frames in a memory bank to retrieve. However, in many settings, we may not know the current pose of the agent and the past frames may not form a consistent world model (imagine learning a world model for the wrist camera of a moving robot) How would the authors imagine generalizing their method in this case?\n\n[1] Song et al. History-Guided Video Diffusion.\n\nLIMITATIONS:\nI did not find any discussion of the limitations of the approach. One big limitation might be that the retrieval mechanism is likely significantly slower than directly generating video.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework that incorporates a memory bank consisting of state-aware memory cells.  By modeling temporal and spatial states, this approach aims to address the challenge of maintaining long-term consistency in world model video prediction.  The core of the framework utilizes a specially designed memory attention mechanism that enables the memory unit to store not only information from memory frames, but also pose and timestamp information.  The experimental results in virtual scenes are impressive, with PSNR, LPIPS, and rFID metrics evaluated on long sequences across different scenes and interaction modes in the Minecraft game, achieving state-of-the-art (SOTA) performance.  Additionally, the framework demonstrates promising application prospects in the gaming field.  Similarly, on the real-world RealEstate10K dataset, the approach also achieves SOTA performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1: This paper is well-written and clearly articulates the inherent challenge of long-term 3D spatial inconsistency in world generation, which effectively motivates the proposed approach and establishes a clear research direction. The authors introduce a compelling framework based on a state-aware memory attention mechanism, which enhances long-term world generation by explicitly modeling temporal and spatial states, providing a solution to the consistency problem.\n\n2: The method demonstrates strong performance both quantitatively and qualitatively, with comprehensive metrics including PSNR, LPIPS, and rFID that validate the effectiveness of the approach.\n\n3: The visual results are impressive and provide clear evidence of the method's capabilities, showing substantial improvements in maintaining visual quality over extended sequences.\n\n4: The method shows clear advantages in maintaining consistency over long temporal horizons, which is the core challenge the paper aims to address and represents a significant advancement in the field.\n\nWeaknesses:\n\n1: The paper lacks a comprehensive analysis of memory usage and the maximum duration for which high-quality reconstructions can be maintained.   Without this analysis, it remains unclear what the practical limitations and scalability boundaries of the proposed method are, which is crucial for evaluating its real-world applicability.\n\n2: The encoding methods employed for pose and timestamp information appear to follow conventional approaches without significant innovation.   The authors should provide stronger motivation for these design choices or explore more sophisticated encoding schemes that could better capture the temporal-spatial relationships.\n\n3: The experimental evaluation of dynamic scene reconstruction is somewhat limited in scope.   For virtual environments, the demonstration is confined to selected Minecraft scenarios, while real-world evaluation focuses primarily on indoor scenes.   A more diverse set of scenarios, including outdoor environments and different types of dynamic content, would strengthen the generalizability claims of the proposed framework.\n\nQUESTIONS:\n1. It would be beneficial to include additional results across a wider range of styles and scene types in both virtual and real-world environments, in order to further demonstrate the generality and robustness of the proposed method.\n\n2. The paper would be strengthened by explicitly stating the practical limitations of the method, including but not limited to memory usage, training time, and inference speed, to better illustrate its real-world applicability.\n\n3. A clear analysis of failure cases, along with a discussion of the underlying reasons for the method's limitations in these scenarios, would provide valuable insight and help guide future improvements.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}