Learning 3D Persistent Embodied World Models
Authors: Siyuan Zhou
Yilun Du
Yuncong Yang
Lei Han
Peihao Chen
Dit-Yan Yeung
Chuang Gan
Venue: NeurIPS 2025 poster ()
Keywords: Embodied World Models
Primary area: applications

Abstract: The ability to simulate the effects of future actions on the world is a crucial ability of intelligent embodied agents, enabling agents to anticipate the effects of their actions and make plans accordingly. While a large body of existing work has explored how to construct such world models using video models, they are often myopic in nature, without any memory of a scene not captured by currently observed images, preventing agents from making consistent long-horizon plans in complex environments where many parts of the scene are partially observed. We introduce a new persistent embodied world model with an explicit memory of previously generated content, enabling much more consistent long-horizon simulation. During generation time, our video diffusion model predicts RGB-D video of the future observations of the agent. This generation is then aggregated into a persistent 3D map of the environment. By conditioning the video model on this 3D spatial map, we illustrate how this enables video world models to faithfully simulate both seen and unseen parts of the world. Finally, we illustrate the efficacy of such a world model in downstream embodied applications, enabling effective planning and policy learning.

Learning 3D Persistent Embodied World Models
Siyuan Zhou"
Yilun Du
Yuncong Yang" Lei Han
Peihao Chen
Dit- Yan Yeung"
Chuang Gan
Abstract
The ability to simulate the effects future actions on the world a crucial ability intelligent embodied agents, enabling agents to anticipate the effects of their actions and make plans accordingly. While a large body of existing work has explored how to construct such world models using video models, they often myopic in nature, without any memory of a scene not captured by currently observed images, preventing agents from making consistent long-horizon plans in complex environments where many parts of the scene are partially observed. We introduce new persistent embodied world model with explicit memory of previously generated content, enabling much more consistent long-horizon simulation. During generation time, video diffusion model predicts RGB-D video of the future observations of the agent. This generation is then aggregated into a persistent 3D map of the environment. By conditioning the video model this 3D spatial map, we illustrate how this enables video world models to faithfully simulate both seen and unseen parts of the world. Finally, illustrate the efficacy of such a world model in downstream embodied applications, enabling effective planning and policy learning.
Introduction
By training vast and diverse datasets from the internet, large video generation models have demonstrated impressive capabilities that expand the horizons of computer vision and AI [2, 5, 12, 39]. Such models are especially useful in embodied settings, where they can serve as world models, enabling simulation of how the dynamics of the world will evolve given actions [6, 37]. Such an embodied world model can then significantly benefit tasks like path planning and navigation, enabling agents to make decisions based on simulated interactions before acting in the real world and enabling agents to be purely trained on simulated real-world interactions.
However, fundamental challenge exists for embodied world models: the underlying state of the world is represented as a single image or chunk of images, preventing existing world models from consistently simulating full 3D environments where much of the generated world remain unobserved any given moment [16]. As a result, models are myopic and often generate new elements that conflict with its historical context-thereby harming the internal consistency required for real-world fidelity. These contradictions undermine the world models' capacity to track evolving environments and plan over extended horizons, ultimately constraining their utility for navigation, manipulation, and other tasks demanding stable, long-term interaction. video defining a 3D scene, the world model without 3D memory fails to retain the unobserved regions once they move out of view. Compared to the ground truth frames, not only do items like the painting and table disappear, but the room structure turns into an entirely different sooocesshocssinng contradiction with the provided context.
illustrated in Figure 1, given context
To solve this issue, we present Persistent Embodied World Model, an approach to learning an accurate embodied world model by explicitly incorporating a 3D memory into world models. Real-
HKUST
UMass Amherst
MIT
Independent researcher
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Future imagination by World Models WD Memory (Qusii
Context Path
Mamory
Future Path
Context Path Video
Ground Truth Future Path Video
Figure I:
3D Persistent Video Generation. Given the context video that defines the top-left 3D scene, the baseline world model deviates from this layout and introduces contradictory elements. In contrast, with 3D memory, our approach preserves observed structures and generates content consistent with the original context.
world scenes naturally unfold in three dimensions, representation by populating 3D grids with DINO features [22] representing previously generated video frames, thereby capturing the spatial relationships within the environment. To enable accurately capture and generate the 3D geometry in an environment, our model processes and generates RGB-D data to preserve critical geometric cues in the memory's 's 3D structure. To enable 3D consistent simulation, our model converts an agent's actions into corresponding relative camera pose change in the map, enabling effective retrieval from the 3D memory and ensuring consistent viewpoints and content across frames. By uniting 3D- -t-uuuuutrrrd memory, depth- aware generation, and precise camera control, our model not only predicts future observations under action control but also faithfully reconstructs past scenes with high degree of spatial coherence. In Figure 1, the 3D memory module allows our method to preserve previously observed 3D structure, maintaining coherent scene content throughout generation. In contrast, model without the 3D memory progressively deviates from the given context.
model constructs volumetric memory
Our empirical results show significant enhancement for both the visual quality and consistency of video generation, highlighting the effectivvness of our proposed model and its associated memory mechanisms. In particular, these results confirm that incorporating a 3D memory promotes 3D persistence of embodied video generation, ensuring consistency with both previously generated and observed frames. Additionally, this persistence offers substantial advantages for the downstream robotic applications, including ranking the sampled action trajectories, planning with model predictive control, and policy training in the video simulators.
In summary, contributions
follows:
1. We propose the Persistent Embodied World Models, an embodied world model system which incorporates 3D memory into video diffusion models, enabling persistent video generation.
2. We empirically demonstrate that model offers significant improvement for both the visual quality and consistency of video generation.
3. We show how our model can be used in downstream embodied application such as planning and policy training.
2 Method
In this section, we present our proposed method in detail. We first give some formulations about world models with memory and action-driven video generation models Section 2.1. We then propose our 3D memory approach, which introduces the capability of 3D spatial understanding and

---

Input
Memory Maintenance
b) DIT Block
RGBD
Nokyy
Extraction
Full Arention
JD Festures
2D Festurcs
harward
3D Menlory
New 3D Memory
Output
RGBD Video
Block
Action (&R, (2)
Food Forwadd
Tideen States:
Camera Pose
Firbakking
Figure 2: Overview of our framework. (a) Our model takes the current RGB- (GB-Doo observation, action, and 3D memory as imput and synthesizes an RGB-D video. (d) The memory is incrementally updated after each vidco generation. (b, c) We train the memory blocks only and freeze DiT blocks of the video models the second training stage.
3D map features into the video models in Section 2.2. Finally, we describe the training pipeline and potential applications of our model for robotics in Section 2.3.
2.1 Formulation
World Models with Memory. We formulate our proposed model action action-guided guided video generation models with a 3D feature map memory. Specifically, our model takes the current image observations O,, agent actions A, and 3D feature map memory M input, and then predicts the future observations {Ot+1. O++H) H that the agent will see if it executes the given action A, In this paper, the observations O, refer to RGB images and depths. Agent's actions A, refer the navigation commands (e.g., move aaward/backward. turn left/right) and interaction commands (e.g., pick objects, move objects). The 3D feature map memory M is initialized all zero values, and then is incrementally updated given the observations of the agent. The memory M enables our model to not only imagine unknown areas but also maintain consistent generations in previously seen areas.
Action Representation. In this paper, we convert action commands A, into corresponding relative camera pose transformation C'. We then explicitly condition video generation on the computed relative camera pose of each action. Each relative camera pose is represented as the intrinsic matrix K and extrinsic matrix E t.. However, some works [14] indicate that direct conditioning video generation on the raw camera poses complicates the correlation between these values and image pixels, restricting the ability to control visual details precisely. To more accurately condition relative camera transform, we use the Plücker embedding [29] to represent each camera transform (more details in Appendix A.2).
Action-Driven Video Generation. To leverage the pre-trained generation abilities, we use Cog VideoX [39] as the backbone of our model. Previous works [14] mostly utilize cross-attention for the oction-driven and camera-control. However, we found that directly concatcnating Plücker camera embeddings with input image channels is better for precise pixel-wise control for CogVideoX.
Given a video of shape T x H x W. 3D VAE of Cog VideoX will compress the video and generate the video latent of shape T x H x W' We also downsample the Plücker camera embeddings with the same compression rate using UnPixelShufile layers [28]. We concatenate the downsampled camera embeddings with video latent and extend the original patch embeddings to accept new channels.
2.2 Video Generation with 3D Map
To enhance the representation of scene information, we create a volumetric memory representation by filling 3D grids with DINO features [22]. Constructing 3D map from one several images includes
Parts of an action that do not correspond camera pose transformation be encoded separately

---

several steps: (1) feature extraction, (2) feature unprojection, (3) aggregation. We first extract image features F' via DINO-v2 [22]. a pre-trained image encoder, for each image I. Then, we lift 2D image feature F into the 3D space. We unproject each pixel to 3D grid in the world coordinate space using depth, intrinsic and extrinsic matrix. Finally, we aggregate each grid the DINO-Map through max-pooling. For efficiency, we extract the meaningful grids from the 3D map. To maintain 3D) spatial relations, we then concatenate each grid with corresponding 3D sinusoidal absolute position embeddings.
key challenge in combining video models with the 3D spatial feature map that most video models are limited in the abilities of 3D spatial understanding. In order to accurately model such relationships, we inject depth information into the video models. We generate RGB-D videos instead of only RGB videos to introduce 3D-aware information into the video latent. With generating and processing depth information, our model can maintain the 3D geometry and spatial information. Drawing from TesserAct [42]. we separately encode RGB and depth with 3D VAE. We concatenate RGB and depth and extend the input and output layers to accept and output depth channels.
To inject DINO-Map into the video diffusion models, we design the cross-attention expert block as our memory block. We additionally concatenate camera embeddings C the video hidden states H such that the model has the information of both camera pose and depth. Thus, the model is able to correlate the hidden states with the corresponding 3D feature grids. Drawing from the Cog VideoX, we use expert adaptive layernorms to improve the alignment across two feature spaces. We regress the different scale and shift parameters a, B. and y from the time embeddings for the video latent and map latent separately. Specifically, we define our memory blocks as:
H Horrm M OH ars norm, M,
H H + aay Hnom Mnorm) H = Н + (nff((orrm(HH)
where norm; is expert adaptive layernorm, normy is layernorm, Attn is the cross-attention layer and ff is the feed forward layer. As illustrated in Figure 2, we inject the memory blocks after each original transformer block.
2.3 Training and Application
Training. Training large video diffusion models directly 3D feature maps is computationally prohibitive. To address this, our approach involves two training stages. We begin by fine-tuning CogvideoX without the memory block our dataset. The training objective of this stage is:
C = Bug [llooll) iioo,, de; c) <||²]
where 6p is video models, 2f is video latents, and 0z, at c represent observation, action and camera pose. We use superscripts i E j0, N] to denote diffusion steps (for example, 2²) and the added noise. The aim of this training stage is threefold: 1) Adapt pre-trained video models to the specific domain of our dataset. 2) Integrate action-conditioned controls to enable precise guidance over the generated video. 3) Optimize video models to generate RGB-D videos, producing both color (RGB) and depth (D) frames simultaneously. which is critical for further map construction.
Next, in the second training stage, we train the memory blocks only and freeze other parameters. The training objective of this stage is formulated as:
L Egi, ,.. Illlo (a ilo,, o2, c, M) e||²]
(3)
where M is the 3D feature map. This process teaches the model exploit the information in maps but also maintains the generation abilities of the original model.
Applications in Embodied Domains. We next describe how to leverage our model in downstream embodied applications such as model predictive control (Algorithm 1 (c)) or policy training. Overall, our model with memory serves as temporally consistent dynamics model to simulate agent- environment interactions.
To enable an agent to accurately choose actions in an environment, we can combine our model with model predictive control. Given a reward function R(V) that the agent is optimizing, we can optimize an action
arg max R(V),

---

Table 1: Video Generation Results. Our model outperforms baselines in both visual quality and consistency.
Method
PSNR
SSIM f
LPIPS +
FVD 4
DreamSim
SRC t
NWM
17.5
0.66
0.31=0.011
19414.3
0.247:0.008
63.4
Image Memory
19.0+a.25
0.68
0.27=0.01
124
0.184
69.4
Ours (w/o depth)
20.6:t0112
0.67=0.01
0.22#t0.1
11442.5
0, +0.002
77.8=0.24
Ours (w/ 2D-Map)
21.6:10.15
0.75
0.17.0.01
98:2.3
0.097
79. 79.2.1.22
Ours
22.5
0.76
0.16.0..0
92.1200
0.086
81.7.10.10
GT
GT
Ours
Ours
NWM
NWM
Figure 3: Qualitative Comparison of Ours with baselines. Given 3D memory and camera trajectory, the videos generated by our model are high-quality and closely match the ground truth. while NWM [2], without the memory mechanism, generates new contents that conffict with the ground truth. We use green boxes to show consistency and red boxes to show conflict.
which then execute in the environment. The above optimization objective searches for sequences of actions a so that the image observations the generated video from our method maximizes the reward function r(s). When optimizing Equation 4, we can either use sampled action chunks from a pre-trained policy [23] or directly search in the space of the actions using optimization methods such the cross entropy method [9]. In comparison to directly learning policy, using MPC to obtain actions enables the agent to more precisely select actions subject to the dynamics of the world, where the addition of memory our model enables more faithful optimization in partially observable embodied environments.
Alternatively, our model can also be used grounded simulator to generate data for policy learning in unseen environments. In this setting, we initialize the map M from a few-shot images of the unseen environment we wish to adapt to. We can then simulate trajectories of interaction in this environment by repeatedly using the policy to generate actions, using our video model to generate future observations given these actions, and then updating the map M with these observed future observations. We can then use hindsight relabeling [36] to label trajectories with goals and rewards, and use the downstream data train and improve the policy iteratively.
Experiment
We describe the experimental setting and our design choices and compare model to previous approaches. We report the results using three random seeds.
3.1 Dataset
We collect our dataset in the Habitat Simulation [32] with about 1,000 scenes from HM3D [24]. We split the scenes into training scenes and test scenes. Though the visual quality of Habitat is not perfect, the reason we still choose this is that the scenes from Habitat include multi-rooms and are much larger than the real-world dataset [3, 45J. In addition, Habitat supports the potential to collect data that how the agents interact with the environment. Our dataset includes about 50k trajectories with the most 500 steps. Additional details are included in Appendix A.
5

---

NWM
Ous
NWM
NWM
Qualitative Results of Consistency Generation. We autoregressively generate videos four times and present the final images from each generation. Our model can generate consistent content after revisiting the same location.
Figure 4:
3.2 Video Generation
We first evaluate the capability of our model and other baselines to generate temporally coherent and geometrically consistent videos from memory.
Baselines. We compare our model with several baselines:
Navigation World Model (NWM) [2]*. NMW is controllable egocentric video generation model. It predicts future observations based on past observations and actions.
Image Memory. This baseline is drawn from 3D-Mem [38], which previous used for VQA. leverages a compact set of informative snapshot images as 3D scene representation. We adopt the snapshot images as the image memory for the video models.
Ours (w/o depth). This an ablative baseline trained with RGB videos without depth information.
Ours (w/ 2D-Map). This is also an ablative baseline trained with 2D feature maps.
We evaluate the overall performance of video generate according to FVD [33]. To assess the persistence of the scene information between the videos and memories, we use peak signal-to-noise ratio (PSNR), SSIM [34], LPIPS [41] and DreamSim [11] to measure the frame-wise similarity between the generation and ground truth. We mainly focus on how well our model and other baselines generate coherent and consistent content from memory. Thus, we construct maps based on the previously visited observations as memory. For a fair comparison, all baseline models were configured to have an equal number of parameters. And NWM would take empty memory as the conditioning input.
Evaluation.
We first assess the effectiveness of our model. The results are highlighted in Table 1. Our method gains a significant improvement over the baseline without memories (NWM) and surpasses all other baselines with memories across all metrics. Specially, our model reaches notable FVD score of 92, significantly lower than those of other methods, such as NWM (194) and Image Memory (124). Furthermore, the better PSNR and SSIM scores also reflect that our model excels in visual fidelity. The poor performance of NWM highlights the critical importance of incorporating memory mechanisms, with baselines with an underlying memory mechanism substantially outperforming it. Image Memory and ours (with 2D-Map), which consider 2D representations for the memory, lack 3D spatial awareness, resulting in the degradation of the performance. Comparison with ours (w/o depth) indicates that incorporating depth information enhances the correlation between the hidden states of the video and 3D feature grids. As shown
Generation Results.
Since the original paper has not released the codes and the architecture of NMW is conditional diffusion transforerr model, similar our backbone CogvideoX [39].. implemented based our backbone.

---

Figure 5: Qualitative Examples of Long Video Generation. Our model generate long videos with memory,
in Figure 3, NWM generates new elements that conflict with the real scene while model can accurately replicate elements.
Persistent Generation. Next, we evaluate models using the metric Scene Revisit Consistency (SRC) from SlowFast [16] to access the consistency of video observed in the same location when revisited through reverse actions. SRC is determined by calculating the cosine similarities between visual features of the first visit and subsequent visits. We use DINO-v2 [22] to extract visual features. The results are highlighted in Table 1. We find that baselines with memory mechanisms all have better performance than NWM, among which our model achieves the best performance. Figure shows the quantitative results of our model and baselines without memory. Our model maintains long consistent generations after turning left twice and right back.
Table 2: Experimental Results of Ranking Trajectories. Our model exhibits superior performance compared with baselines.
Method
ATE +
RPE +
SIM
NoMaD
4.94+0.2
4.86+0.4
36,8+1.2
NWM (x8)
4.82+0.2
3.62+0.2
59.743.5
Ours (x8)
4.80+0.1
3.58=0.1
62.5+3.5
NWM (x16)
4.54+0.1
3.51
68.7+1.4
Ours (x16)
4.47+0.1
3.28=0.1
70.8+1.7
3.3 Planning with World Models
In this section, we describe how our model can help improve the performance of robotic navigation policies as illustrated in Section 2.3, One major challenge for the previous world model approaches is that they don't 't have ability to capture the 3D structure the entire environment, leading to unreliable guidance to the robotic policies. Our model, with 3D) persistent memory map, can ground generated videos to the real environment. Similar to NWM [2], we plan with the world models by generating videos of length 8.
Ranking Trajectories. We leverage the video models to rank the multiple trajectories generated by an existing policy. We use NoMal) [30], a state-of-the-ar diffusion policy for robotic navigation, to sample N 8, 16 action trajectories. We use video models to generate future observations under the action trajectories. We rank trajectories by computing the cost function, which is LPIPS similarity between the goal observation and the last frame of the generated videos.
We use Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) [2, 31] to access the global consistency and local accuracy of the final predicted action trajectories. We also evaluate the similarity (SIM) between the final pose and final rotation after exccuting the action trajectories and real pose and rotation. As shown in Table 2, model outperforms navigation policies and the baselines without memory mechanisms, demonstrating the efficacy of our approach.
Model Predictive Control. We assess how our model can be used with MPC. For simplicity, we use the discrete actions (e.g., move forward, turn left/right). We initialize the uniform distribution over the action space. We randomly sample N 60 action chunks from the current distributions.

---

90
70
0.-
SIM
50
10
20
200
400
600
800
500
1000
1500
2000
2500
3000
3500
Step
Rollout
(a) MPC Results
(b) Policy Learning Results
Figure 6: Experimental Results of MPC and Policy Learning. (a) Through generating more samples, our model with MPC achieves a 17% performance gain compared to ranking action trajectories. (b) By leveraging few-shot images stored memory prior, our model significantly enhances policy learning performance.
Tum Left
Tum Lefl
Turn Right
Turn Right
A
Turn Right
Turn Right
Turn Left
Turn Left
Figure 7: Qualitative Examples of Unseen Persistent Generation. Our model generate persistent videos even in unseen environments.
We generate the potential observations given each action chunk through our model and compute the cost function. We select the top A 30% action chunks to update the distributions. The metrics ATE and RPE measure the trajectory-Ievel distance and are not suitable for MPC since two different trajectories reach similar final goal. Thus, we only report SIM metries. The results shown in Figure 6a. After about 350 iterations, our model with MPC can achieve results similar with ours with NoMad. Our model with MPC achieves competitive results, SIM of 87.5 after about 720 iterations.
Policy Learning in New Environment. As described Section 2.3, our proposed model has the ability to support the policy training in new environment with few-shot images. Then, we provide proof-of-concept by fine-tuning the previous agent policies the new environments. The results are shown in Figure 6b. Our model can boost policy learning.
3.4 Additional Video Generation Results
Finally, we demonstrate the additional capabilities of our model, particularly in generating extended video sequences and generalizing to novel scenes.
Long Video Generation. We find that approach is able synthesize very long video trajectories while maintaining 3D consistency across extended time horizons by incrementally updating memory. To illustrate this, generate total of 1 112 video frames autoregressively in Figure 5.
Simulation in Unseen Scenes. We further illustrate how our approach can generalize well and simulate new unseen scenes. The results are shown in Figure 7, while modest decline in visual

---

fidelity occurs compared to training domains, the framework robustly preserves object permanence and scene consistency, underscoring its adaptability to novel scenes.
4 Related work
Embodied World Models. Significant research efforts have developed generation models as world models in gaming [1, 6]. autonomous driving [12, 20], and robotics [4, 8, 23, 36, 43]. Prior works [17, 44] have primarily focused on training models using low-dimensional state and action representations on simulation data. These approaches leverage trajectory generation to enable robot planning. However, such methods face significant challenges in scaling to high-dimensional observation spaces. UniPi [10], addresses this limitation by employing video diffusion model directly predict future visual frames subsequently coupled with an inverse dynamics model to infer action. Uni- Sim [36] and Genie [6] demonstrate the utility of world models for training agents by enabling policy learning without direct environmental interaction during training. However, existing world model frameworks in robotics remain largely restricted to simplified settings, such table-top manipulation tasks, limiting their applicability in real-world scenarios requiring complex spatial reasoning. Some works [18, 19] propose visual world models for the indoor navigation tasks. Most similar our work, Navigation World Models [2] introduces a controllable video generation framework conditioned on navigation commands simulate environment dynamies. While promising, this method lacks an explicit memory architecture, leading to scene inconsistency. Our framework integrates a memory mechanism that enables consistent long- horizon imagination, ensuring coherence with previously generated content.
Video Generation and Memory. A fundamental challenge in persistent video generation stems from computational memory limitations, which restrict the number of frames processed in a single forward pass. Previous approaches [13, 15, 26, 37] condition each generated video chunk solely on its immediate predecessor, creating truncated context windows that discard prior scene history. This results in temporal fragmentation and inconsistencies when revisiting earlier spatial contexts. Recent work [16]. utilizes a temporary LoRA module that embeds the episodic memory in its parameters in video diffusion models. However, it requires additional training time during inference, and its reliance on 2D latent representations neglects 3D spatial priors. Persistent Nature [7] models the 3D world as 2D terrain and render the video via NeRF [21]. Alternttve approaches like InfiniCube [20] leverage structured high-definition (HD) maps to enforce geometric consistency in long videos, but such maps are inherently static, simplistic, and confined autonomous driving. To overcome these limitations, we propose 3D feature map that serves as a dynamic memory buffer, explicitly encoding semantic attributes and spatial geometry to maintain coherent environmental representations across extended temporal horizons.
5 Conclusion
In conclusion, present the Persistent Embodied World Model, a novel framework that integrates 3D memory structures into video diffusion models to achieve persistent, spatially coherent video generation. Our model demonstrates significant advancements in both visual fidelity and temporal consistency for generation. Crucially, the system not only synthesizes plausible content for unexplored environments but also maintains geometric and semantic coherence in previously observed scenes. Furthermore, we showcase the practical utility of our model in embodied AI to enhance robotic planning and policy training.
Limitations and Future Works One limitation of our work is that we need the data with depth. Most datasets either don't 'tnt include depth information [45] or are limited in the diversity of trajectories. [3, 40J. To apply our approach to larger real-world datasets, one direction to leverage pre-trained depth estimation models (e.g., Depth Anything [35]) to estimate depth in videos. We can further use mixture of simulation data and real-world data to further improve data diversity.
Additionally, our constructed 3D maps of the environment do not model the dynamic evolution of the environment over time. This prevents our approach from simulating dynamies of many embodied environments where the surroundings in the environment are constantly changing i.e. moving cars in traffic or other inhabitants in house. Further learning an additional dynamics model on top of the 3D spatial memory to model such changes would be an interesting direction of future work.

---

Acknowledgments and Disclosure of Funding
Siyuan Zhou was partially supported by a Research Impact Fund project (RIF R6003-21) and General Research Fund project (GRF 16203224) funded by the Research Grants Council (RGC) of the Hong Kong Government.
References
[1] Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-58899 2024.
[2] Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models, 2024.
[3] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimny, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitscenes a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[4] Homanga Bharadhwaj. Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch. Ted Xiao. Dhruv Shah, Fei Xia. Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409. /6283, 2024.
[5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[6] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richic Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty- first International Conference Machine Learning, 2024.
[7] Lucy Chai, Richard Tucker, Zhengqi Li, Phillip Isola, and Noah Snavely. Persistent nature: A generative model of unbounded 3d worlds. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 20863-20874, 2023.
[8] Xiaowei Chi, Hengyuan Zhang. Chun-Kai Fan, Xingqun Qi. Rongyu Zhang, Anthony Chen, Chi-min Chan, Wei Xue, Wenhan Luo, Shanghang Zhang, et al. Eva: An embodied world model for future video anticipation. arXiv preprint arXiv:2410.15461 2024.
[9] Pieter-Tjerk De Boer, Dirk Kroese, Shie Mannor, and Reuven Rubinstein. tutorial on the cross- entropy method. Annals of operations research, 134:19-67, 2005.
[10] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Picter Abbeel. Learning universal policies via text-guided vidco generation. Advances in neural information processing systems, 36:9156-9172. 2023.
[11] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data, arXiv preprint arXiv:2336.09334 2023.
[12] Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. Advances in Neural Information Processing Systems, 37:91560-91 100--9199..900 2025.
[13] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilhach, and Frank Wood. Flexible diffusion modeling long videos. Advances in Neural Information Processing Systems, 35:27953-27965, 2022.
[14] Hao He, Yinghao Xu, Yuwci Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint 2000::::444002000, 2024.
[15] Vingqing Hc, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long vidco generation. arXiv preprint arXiv:2211. 12.22221,1 2022.
[16] Yining Hong, Beide Liu, Maxine Wu, Yuanhao Zhai, Kei-Wei Chang. Lingjie Li, Kevin Lin, Chung Ching Lin, Jianfeng Wang. Zhengyuan Yang. et al. Slowfast-vgen: Slow-fast learning for action- long video generation. arXiv preprint 2100:224 2024.
10

---

[17] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint wrXiv:2205.09911, 2022.
[18] Jing Yu Koh, Honglak Lee, Yinfei Yang. Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[19] Klemen Kotar, Aaron Walsman, and Roozbeh Mottaghi. Entl: Embodied navigation trajectory learner. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10863-10872, 2023.
[20] Yifan Lu, Xuanchi Ren, Jiawei Yang. Tianchang Shen, Zhangjie Wu, Jun Gao, Yue Wang. Siheng Chen, Mike Chen, Sanja Fidler, et al. Infinicube: Unbounded and controllable dynamic 3d driving scene generation with world gorld-guided video models. arXiv preprint arXiv:2412.03934, 2024.
[21] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Commanications of the ACM, 65(1):99 106, 2021.
[22] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szaframiec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023.
[23] Han Qi, Haocheng Yin, Yilun Du, and Heng Yang. Strengthening generative robot policies through predictive world modeling. arXiv preprint arXiv:2502.06622, 2025.
[24] Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg. John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterppor 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2109.08233. 2021.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[26] David Ruhe, Jonathan Heek. Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models. arXiv preprint arXix:2402.0 09470, 2024.
[27] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00 005120 2022.
[28] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop. Daniel Rueckert, and Zehan Wang. Real-time single image and video super- resolttino using an efficient sub sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883, 2016.
[29] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-ovaluation rendering. Advances in Neural Information Processing Systems, 34:19313- 2021.
[30] Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In 2024 IEFE International Conference on Robotics and Automation (ICRA), pages 63-70. IARF, 2024.
[31] Jürgen Sturm, Wolfram Burgard, and Daniel Cremers. Evaluating egomotion and structure- fromonottoon approaches using the tum rgb-d benchmark. In Proc. the Workshop on Color-Depth Camera Fusion in Robotics at the JEEE/RJS International Conference on Intelligent Robot Systems (IROS), volume 13, page 6, 2012.
[32] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner. Noah Maestre, Mustafa Mukadam, Devendra Chaplot. Oleksandr Maksymets. Aaron Gokaslan. Vladimir Vondrus, Sameer Dharur. Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira. Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[33] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metrie & challenges. arXiv preprint arXiv:1812.01717. 2018.
11

---

[34] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing. 13(4):600-612, 2004.
[35] Lihe Yang, Bingyi Kang, Zálong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 57:21875-21999 2024.
[36] Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators, arXiv preprint 4rXiv:2310006114, 2023.
[37] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion prohabilistic modeling for video genera- tion. Entropy, 25(10):1469 2023.
[38] Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning. 2024.
[39] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong. Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion madels with an expert transformer. arXiv preprint urXiv:2408.0600 06072, 2024.
[40] Chandan Yeshwanth, Yuch-Cheng Liu, Mathias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 12-22, 2023.
[41] Richard Zhang, Phillip Isola, Alexei Efros, Shechtman, Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1866595,10 2018.
[42] Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du and Chuang Gan. Tesseract: Learning 4d embodied world models. arXiv preprint arXiv:2544.2099, 2025.
[43] Siyuan Zhou, Yilun Du, Jiahen Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377 2024.
[44] Siyuan Zhou, Yilun Du, Shun Zhang, Mengdi Xu, Yikang Shen, Wei Xiao, Dan Yeung, and Chuang Gan. Adaptive online replanning with diffusion models. Advances in Neural Information Processing Systems, 36:44000 44016, 2023.
[45] Tinghui Zhou. Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 1rXiv:1805.09817, 2018.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Section I, we conclude our main contribution in the last paragraph: learn the embodied world model with 3D memory to generate persistent content, enabling effective planning and policy learning.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations and future work Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
We don't 't include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details in the Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code and dataset in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss cttppsccccpbccccccceececcccccccc /twpe/ggiiddeeeeeeebiiiiiieeee /Cooeeeooiissseeeeee cessionnoocyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide experimental details in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the error bars calculated as standard deviation across three random
seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computing resources in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:////../.u./eeeeeee wwtpeeewwws....pu.eeeeeeeeeeeeee Seppulic//tticsuuiiiiii
Answer: [Yes]
Justification: The research conducted in the paper conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We describe broader impacts in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper about the backbone and the simulation that we used to generate the data and provide the licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We provide the experimental details in the paper and will release the code and dataset in the future.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy bttpp::/neeuips... httpr//ps..c/ooofeeeeee//////55/ for what should or should not be described.
19

---

A Experimental Details
Map Expert AdaLN
Scale Shift
Memacy
Embeoliin
Timestep
Vision Expert
AdaLN
Hidden States
S Shift
Closs Cross-Attcntion Attention
Scale & Shiit
Foed Forward
Gate
Gate
Hidden States
Embolkings
Figure 8: Detailed Implementation of the Memory Block.
A.1 Dataset
We compare different datasets in Table 3.
Our dataset collected from HM3D is a large dataset satisfying multi-room and depth support. Though our dataset mainly includes navigation actions, it's feasible to be extended to interaction commands since is collected from the simulation Habitat [32]².
We collect our dataset by randomly sampling start and goal positions in the scene and following the shortest path at most 500 steps.
Sometimes, the agent would already have much information about the scenes. Thus, the agent has possibility to around to get scene information initially when collecting data.
Table 3: Comparison with Other Datasets.
Dataset
Num. Videos
Room
Depth
Allowed Interaction
Ours
50K
Many
RealEstate [45]
10K
Many
x
x
Scannet++ [40]
1.8K
Many
V
ARKitScenes [3]
3.1K
Few
Table 4: Details about 3D VAE.
RGB Depth
Camera
Input
512 512
9 x 512 x 512
Compression
3D VAE
Mcan-Pooling & UnPixelShuflle
Latent
64 X 64 x 16
3 64 x x
A.2 Implementation Details
We use CogVideoX [39]$ our backbone. CogVideoX is transformer-base architecture. We modify the PatchEmbed Layer to accept the depth latents and camera embeddings. In particular, we expand the input channel of the original convolution network (or MI. in Cog VideoX 1.5) and copy the original weights. Expanded parameters are zero initialized. Similarly, we modify the output layer to output the depth latents.
The architecture our memory block is illustrated in Figure We concatenate video hidden states with camera embedding. Similarly to CogvideoX [39], we normalize the hidden states and 3D
http://hhw...oneeeeeoooossasaaaa MIT license. http://gittbb.oom/TTUUUMMMCoy
Apache-2. 2.0 license
20

---

Tum Right
Turn Right
Tum Left
Tur Left
Tum Left
Tum Left
Tum Right
Tum Right
Tum Right
Right
Left
Turn Right
Right
Figure 9: Consistency Results.
memory via the expert adaptive layernorm separately. Instead of 3D full attention of CogvideoX, we use Cross-Attention Layer for efficiency. The number of additional parameters for all memory blocks nearly 1B.
3D VAE. We use 3D VAE from CogVideoX to compress RGB-D videos to video latents. We feed RGB and depth separately to 3D VAE, which compresses 9 512 x 512 frames to 3 x 64 x 64 latents. We also downsample the plücker embedding with the same compression rate. In particular, we use mean-pooling to downsample the spatial dimension and UnPixelShuffle layers [28] to compress the temporal dimension.
Memory. we create a volumetric memory representation by filling 3D grids with DINO features. The shape of 3D grid map is 256 32 256 384 and the size of each grid is 0.25m 1m 0.25m. We first extract image features via DINO-v2 [22] and upsample the features by bilinear interpolation. Then, we lift 2D image feature into the 3D space. We unproject each pixel 3D grid in the world coordinate space using depth, intrinsic and extrinsic matrix. Finally, we aggregate each grid in the DINO-Map through max-pooling.
Our memory can be easily extended to outdoor environments. We can maintain a large and unbounded grid map and extract the local map that is fed into the video diffusion models based on the camera pose. We leave for future work.
Action Representation. To more accurately condition on relative camera transform, we use the Plücker embedding [29] to represent each camera transform. Specifically, an intrinsic matrix and extrinsic matrix E is encoded as an image P E R The value of pixel (u, v) in the image is represented as a concatenated tuple (o x d, d) where o the ray origin for the pixel and d(u.e) the ray direction for the pixel. Specifically, is the camera center in the world coordinate space, which is
21

---

Figure 10: Generation Results.
Algorithm 1 Persistent Embodied World Model.
(a) Video Generation
1: Definition: Video Model po: Map M, Action Chunk a,, Observation with Depth ot, State St.
Generated Video with Depth V, Future Camera Poses {c}
2: Generate video V po(Ot, Or: M)
3: Compute camera pose {c} through 8; and dz
4: Construct new map M from the generated video V and camera poses {c}
5: Update map M with M
(b) Construct Map
1: Input: Video V. Camera Poses c, Image Encoder E
2: Output: 3D Feature Map M
RGB {f}. Depth {D} <-
4: Image feature F1 &- E(I)
5: Compute 3D grid {G} via {D} and camera poses
6: Compute 3D feature F via F, and G and concatenate with 3D position embedding p
7: Construct Map M by aggregating
8: Return: M
(c) Model Predictive Control
Requirement: Video Model pa. 3D Feature Map M, Agent Policies क, MPC Distributions 1
2: Sample action chunks {ar} from the agent policies π or the distributions IP from MPC
3: Generate videos V + (o ag MM)
4: Calculate the cost functions for each sampled video V
5: Rank action chunks {a,}, update the distributions F or policies
t, and d(u,v) is calculated as:
d(a.)) = RK- 1)T +
(5)
This image representation of actions enables more accurate 3D spatial control from the video model.
A.3 Training Details
We train our models with frame skip, where training video clips are subsampled by a specific stride. We use various frame stride from 1 to help the model learn various camera poses. We use AdaM optimizer, with línear warmup and a learning rate of le 4. Additionally, we utilize 6 precision for computational efficiency and clip gradients to a maximum norm of 1.0 to stabilize training. We utilized 8 H100 GPUs for training video diffusion models in approximately 3 days. We
22

---

adopt v-prediction [27] and use the DDIM sampler [25]. The inference sampling step is set to 50, and the inference time for generating the videos of 9 frames is 5 seconds.
A.4 Training and Application
We illustrate the process of training and application in Algorithm 1.
B Additional Experimental Results
B.1 Ablation Study on Voxel Size
We evaluated the model with different voxel sizes, shown in the table 5. Our results indicate that when the voxel size is 256 or larger, model performance remains stable. However, when the voxel size is set below 256, we observe slight decrease in performance.
Table 5:
Ablation Study Voxel Size.
Voxel Size
LPIPS +
PSNR f
SSIM
128
0.20
20.6
0.73
256 (ours)
0.16
22.5
0.76
384
0.16
22.2
0.77
B.2 Ablation Study on Representations
We evaluate the model with different 3D representation methods, shown in the table 6. Compared to point cloud representations and the use of CLIP features, our proposed 3D memory approach using DINO features consistently achieves the best performance.
Table Ablation Study on Representations.
Representation
LPIPS +
PSNR
SSIM
point cloud
0.20
21.2
0.73
clip DINO (ours)
0.18
21.5
0.75
0.16
22.5
0.76
C Additional Qualitative Examples
We show more examples about visual quality and consistency in Figure 9 and 10.
D Broader Impacts
This paper introduces research aimed at pushing the boundaries Machine Learning in the realm of robots. There is potential risk of collisions with objects and people if the robot system is not properly configured the real world. Implementing collision detection strategies is a possible approach to mitigate this risk.
23


Reviews:
[{"review_id": "bXuabDy3z0", "reviewer": "Reviewer_jqaK", "summary": "This paper introduces a persistent embodied world model --  a new method for keeping spatial memory of a room baked into the world model. The authors propose incorporating an explicit 3D memory mechanism into video diffusion models, enabling consistent long-horizon simulation of embodied agents navigating complex environments. The key innovation is maintaining a persistent 3D feature map that gets updated as the agent explores. The model generates RGB-D (color + depth) videos conditioned on current observations, actions, and this 3D memory map. This allows the model to accurately reconstruct previously seen areas when revisiting them, rather than hallucinating new content.", "questions": "What type of information is stored in this model volume? Can you linearly decode a semantic map of the room? \n\nWhat is the sensitivity to voxel size ? How bit of a maximum scene can this approach handle?\n\nHow does this compare to existing novel view synthesis and world models out there such as stable video camera? Does the explicit memory approach only show benefits when the trajectories are very long? And in that case how does it compare to simply using an every updating NERF to store a representation of the environment?", "limitations": "None", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a novel 3D feature map integrated as conditioning with video diffusion models to act as spatial memory.\n- The model shows significant improvements across multiple metrics and demonstrates practical benefits in downstream tasks like trajectory ranking and model predictive control.\n- The paper presents a complete pipeline \n\nWeakness:\n- There could be some more evaluations relating the quality of the generations against Novel view synthesis baselines such as Stable Virtual Camera\n- There could be some evaluations of the physical collisions this model learns, or other world modeling properties that could be baked into this approach -- otherwise the functionality is quite similar to a nerf\n- There could be more analysis of the memory volume produced, what sort of information is store in there, what can we lineary decode, how interpretable is it.", "was_revised": true, "final_justification": "I will keep my score as a borderline accept. Though this still means I recommend acceptance (as the name implies). My main concerns with the work are that the evaluations do not exactly match the types of navigation and embodiment tasks that this method would be most useful for, and including them would significantly improve the cause", "rebuttal": "Dear Reviewer,\n\nWe appreciate the reviewer for the detailed comments and insightful suggestions. We are glad to hear you find that our idea is appealing and our model has significant improvements. Here is the response to your question.\n\n> some evaluations of the physical collisions\n\nThanks for the suggestions. To evaluate how well our model handles physical collisions, we measure frame-wise similarity using LPIPS when collisions occur. We compare the performance of our model to a baseline that renders videos in the simulation without physical collision handling. As shown in the table, our model is able to accurately simulate future frames after a collision.\n\n|  | LPIPS $\\downarrow$ |\n| -------- | -------- | \n| Simulation w/o collision | 0.62 |\n| Ours |  0.25 |\n\n\n> What type of information is stored in this model volume? Can you linearly decode a semantic map of the room?\n\nOur model constructs a volumetric memory representation by populating 3D grids with DINO features. These features capture detailed structural and texture information, which aids in reconstructing the geometry and color of objects in 3D space [1].\n\nYes. The information stored in our memory can be linearly decoded into a semantic map of the room. This is supported by prior work [2, 3] showing that DINO features are highly effective for dense recognition tasks such as detection and semantic segmentation. Additionally, some works [4] have explored using a similar memory volume as a semantic map for vision-language navigation tasks.\n\n\n[1] Hong, Yicong, et al. \"Lrm: Large reconstruction model for single image to 3d.\" arXiv preprint arXiv:2311.04400 (2023).\n\n[2] Oquab M, Darcet T, Moutakanni T, et al. Dinov2: Learning robust visual features without supervision[J]. arXiv preprint arXiv:2304.07193, 2023.\n\n[3] Ren T, Chen Y, Jiang Q, et al. Dino-x: A unified vision model for open-world object detection and understanding[J]. arXiv preprint arXiv:2411.14347, 2024.\n\n[4] Wang, Zihan, et al. \"Gridmm: Grid memory map for vision-and-language navigation.\" Proceedings of the IEEE/CVF International conference on computer vision. 2023.\n\n\n> What is the sensitivity to voxel size ? How bit of a maximum scene can this approach handle?\n\nThanks for the suggestion. We evaluated the model with different voxel sizes, as shown in the table. Our results indicate that when the voxel size is 256 or larger, model performance remains stable. However, when the voxel size is set below 256, we observe a slight decrease in performance.\n\n| Voxel Size | LPIPS $\\downarrow$ | PSNR $\\uparrow$ | SSIM $\\uparrow$ | \n| -------- | -------- | -------- | -------- |\n| 128     |    0.20  | 20.6 | 0.73 |\n| 256 (ours) | 0.16 | 22.5 | 0.76 |\n| 384 | 0.16 |  22.2 | 0.77  |\n\nIn principle, our approach is capable of handling very large—and even unbounded—scenes. This is achieved by maintaining a large, potentially infinite, grid map and dynamically extracting the relevant local region based on the camera to be fed into the video diffusion model. This design allows our method to scale effectively with scene size.\n\n> How does this compare to existing novel view synthesis and world models out there such as stable video camera? \n\nWe compare our model with the stable video camera as shown in the table. We find that stable video camera is good at novel view synthesis, while it also generates new content quite easily without memory.\n\n| Model | LPIPS $\\downarrow$ | PSNR $\\uparrow$ | SSIM $\\uparrow$ | \n| -------- | -------- | -------- | -------- |\n| stable-virtual-camera     |   0.37   | 18.3 |  0.55 |\n| NWM | 0.31  | 17.5 | 0.66 |\n| Ours | 0.16 | 22.5 | 0.76 |\n\n> Does the explicit memory approach only show benefits when the trajectories are very long? \n\nNot only. Our memory approach can improve the 3D spatial consistency of the generated videos. We evaluate the performance of the model for short-horizon generation. The results are shown in the table.\n\n| Short Horizon | LPIPS $\\downarrow$ | PSNR $\\uparrow$ | SSIM $\\uparrow$ | \n| -------- | -------- | -------- | -------- |\n| Ours without memory |  0.20  | 19.8 | 0.74 |\n| Ours | 0.15 | 22.5  | 0.77 |\n\n\n> And in that case how does it compare to simply using an every updating NERF to store a representation of the environment?\n\nOur model not only predicts future observations under action control but also faithfully reconstructs past scenes with a high degree of spatial coherence. NeRF-based approaches are limited in their generative capabilities and cannot synthesize novel elements beyond their original observations. However, constructing and updating NeRF representations is computationally expensive—typically requiring several minutes for a single scene, which makes it infeasible to use NeRF during training or for scalable, interactive applications. In comparison, our method enables more efficient and flexible scene representation and generation, especially in action-conditioned and open-ended settings."}, {"review_id": "VKxKNymjql", "reviewer": "Reviewer_VbjC", "summary": "- The authors incorporate 3D memory features into video diffusion models to enhance agent's understanding of the environment. The goal is to build an embodied agent with long horizon and spatial consistency even during occlusions or revisits.\n- Instead of direct 3D supervision, the model extracted dino features per image and projected them into a 3D volumetric representation. The output predictions are in RGBD format with built in depth understanding, making the model more robust than 2D approaches. \n- The results section presented a clear improvement of visual quality and consistency. The paper also empirically demonstrated that the memory aware model benefits robotics path planning.", "questions": "- What are the rationales in choosing the proposed 3D representation method, as opposed to alternatives like point cloud, nerf, InfiniCube? Are there ablation studies run on alternative 3D representations, or ablation studies on not using dino features?\n- What’s the long horizon boundary of the given method? While the paper qualitatively illustrates good video generation results up to t=112, at what point does the memory module start to fail?\n- What is the end-to-end inference latency for future frame generation and dynamic memory updates, and how does the latency compare with other methods in the ablation? Is the inference time real-time?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- The paper is well motivated to incorporate persistent 3D understanding in the video diffusion model and embodied agent path planning. While there is existing literature on 3D aware video generation, the specific implementation details (e.g. using DINO feature, training memory modules with freezed DiT backbones) provide insights for future papers.\n- The paper is well-written, clear and provides solid evidence to support the claims. \n\nWeakness\n- The paper lacks discussion on the method scalability. The volumetric representation could be intractable in large and complex scenes, limiting the usage of this representation in real scenarios.\n- The paper would benefit from further discussion and experimentation on the specific 3D representation methodology. While briefly mentioned in the related works section, it is unclear how the proposed methodology is superior to alternatives like Nerf and InfiniCube. There is also a lack of ablation study to justify the choice of dino feature, as opposed to alternatives such as using point cloud.\n- Like the authors stated in the limitations section, the current setup is constrained, and does not apply to dynamic and changing scenes.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe thank the reviewer for the detailed reviews and insightful suggestions. We are glad to hear that you find our paper is well motivated and well-written. Here is the response to your question.\n\n> The volumetric representation could be intractable in large and complex scenes, limiting the usage of this representation in real scenarios.\n\nWe acknowledge the potential scalability challenges of volumetric representations in large and complex environments. However, our video dataset is based on HM3D [1], which contains large-scale 3D reconstructions of diverse real-world locations. Each HM3D scene comprises multiple rooms and a variety of objects, with the largest scene covering an area of 2,172 m². Our approach has demonstrated effective scaling to these large and complex environments. We believe this indicates the practical applicability of our method to real-world scenarios, though we agree that further work on memory efficiency and scalability could further enhance its usability in even larger-scale settings.\n\n[1] Ramakrishnan, Santhosh K., et al. \"Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.\" arXiv preprint arXiv:2109.08238 (2021).\n\n\n> What are the rationales in choosing the proposed 3D representation method, as opposed to alternatives like point cloud, nerf, InfiniCube? Are there ablation studies run on alternative 3D representations, or ablation studies on not using dino features?\n\nThanks for the suggestions. We evaluate the model with different 3D representation methods as shown in the table. We experimented with integrating NeRF into the video generation pipeline; however, NeRF requires several minutes to reconstruct a scene, making it impractical for use during training. InfiniCube, on the other hand, relies on structured high-definition maps that are primarily tailored for autonomous driving scenarios, which limits its applicability to our more general settings. Compared to point cloud representations and the use of CLIP features, our proposed 3D memory approach using DINO features consistently achieves the best performance.\n\n| Representation | LPIPS $\\downarrow$ | PSNR $\\uparrow$ | SSIM $\\uparrow$ | \n| -------- | -------- | -------- | -------- |\n| point cloud     |  0.20    | 21.2 | 0.73 |\n| clip | 0.18 |  21.5 | 0.75 |\n| DINO (ours) | 0.16 | 22.5 | 0.76 |\n\n\n> What’s the long horizon boundary of the given method?\n\nThe long-horizon capability of our method depends on the complexity and scale of the scene. There is an increasing probability that the memory module may struggle to maintain temporal coherence, leading to degradation in generation quality when $t$ is larger than 112.\n\n> What is the end-to-end inference latency for future frame generation and dynamic memory updates, and how does the latency compare with other methods in the ablation? Is the inference time real-time?\n\nWe show the inference latency in the table. At present, our video generation model does not achieve real-time performance. However, we believe that real-time inference is feasible by incorporating recent acceleration techniques such as diffusion-forcing [1, 2]. We plan to explore these optimizations in future work to further reduce latency and enable real-time applications.\n\n|  | generation (second) | memory updates (second) | \n| -------- | -------- | -------- |\n| No Memory     |   7   | 0 |\n| 2d Memory | 10 | <1 |\n| Ours | 10 | <1 | \n\n\n[1] Yin, Tianwei, et al. \"From slow bidirectional to fast autoregressive video diffusion models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Chen, Boyuan, et al. \"Diffusion forcing: Next-token prediction meets full-sequence diffusion.\" Advances in Neural Information Processing Systems 37 (2024): 24081-24125."}, {"review_id": "knANEfXFTQ", "reviewer": "Reviewer_MRYi", "summary": "This paper addresses action-guided video generation from RGB-D inputs: given past observations and control signal, the goal is to synthesize future frames that are consistent with the observed world. Recently, such models are termed 'world models' in the literature (i.e., given past observations and control signal, the task is to predict unobserved video frames based on the internal representation of the world). \n\nThis paper's main contribution is persistent, spatial volumetric memory, built by storing visual (DINO) features extracted from newly synthesized frames in a voxel grid. Such construction is possible when input frames consist of image and depth pairs (in contrast to video/camera only inputs). \n\nTo generate output images (and corresponding depth maps), the proposed model encodes video + depth, combines encoded RGBD input with relative poses (wrt. camera pose) using Pluecker embeddings, and uses cross-attention layer that augments encoded representation with 3D spatial features, stored in the spatial memory. \n\nThe method is trained and evaluated on the Habitat Simulation dataset using synthetic data (and pixel-perfect depth maps).", "questions": "Papers states in 2.2: \"For efficiency, we extract the meaningful grids from the 3D map.\". Would you please explain?\n\nAlso, could authors elaborate on the statement: \"A key challenge in combining video models with the 3D spatial feature map is that most video models are limited in the abilities of 3D spatial understanding.\" To the best of my knowledge, this statement is inconsistent with findings in the literature. \n\n\nReferences:\n[A] Agarwal, Niket, et al. \"Cosmos world foundation model platform for physical ai.\" arXiv preprint arXiv:2501.03575 (2025).", "limitations": "The paper points out that key limitations are the static world assumption (present in the dataset), and assuming perfect depth maps as input.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* The paper is reasonably well written\n* The problem statement is crisp and clear: Does maintaining persistent spatial memory and conditioning video generation on this memory improve action-conditioned video generation?\n* Proposed approach with spatial memory presents improvements wrt. variety of metrics that asses visual generation quality, and is shown to work exceptionally well for long-term video generation, which is intriguing\n* I also appreciate experiments on using the trained model in robotic navigation and planning (solid improvements wrt. ATE and RPE).\n* Overall, experimental validation is extensive, covering video generation quality, ranking of trajectories generated by a pre-defined policy, model-predictive control and policy learning. Paper even provides visuals, showcasing long-term video generation, and generated views that were not covered by the camera trajectory (unseen scenes). \n\n### Weaknesses\n\n**Presentation**: \n* Fig 2 stages that \"Our model takes the current RGB-D observation, action, and 3D memory as input and synthesizes an RGB-D video\", but I understand that memory is part, not input, to the model; it is built implicitly. This presentation is confusing. \n\n* The paper often uses references with absolutely no context, making the paper not self-contained. For example: \"Drawing from TesserAct [40], we separately encode RGB and depth with 3D VAE.\"; a better statement would be \"we do XYZ, following [40].\" Another example of weird sentence without context is \"We also downsample the Plücker camera embeddings with the same compression rate using UnPixelShuffle layers [26].\" There are many such cases.\n\n**General:**\n\nThe proposed work can only be trained on synthetic data. The paper argues that this issue could be mitigated with depth generation models, but I am skeptical. The accumulation of multiple views with depth generation models in a voxelgrid will usually not lead to consistent results; it might be that the proposed work only works with perfect depth data. \n\nI would have hoped to see this method trained and evaluated on at least more challenging synthetic datasets (For example, Fabri et al., MOTSynth, ICCV'21, Virtual KITTI; these could be assessed for video generation without control). I know further work may be needed to handle such dynamic scenes, but it is also great to learn where/how the proposed methodology breaks down. \n\nNevertheless, I believe that the proposed contribution (and its significance wrt. geometry-consistent RGBD generation) was clearly established on at least one dataset. I think this idea should be discussed in the community and further explored in future work. \n\n\nI am not very familiar with the state of research in this field, so for the time being, I assume that the selection of dataset and baselines for the evaluation is reasonable. I will revisit this / check again when I see comments of other reviews that may be more familiar with action-conditioned video generation and state-of-the-art in this field.", "was_revised": true, "final_justification": "After the rebuttal, I am getting the sense that the authors did not take our recommendations on how to improve this paper seriously, and I don't think the paper is ready to be accepted in the current form. For example:\n\nRebuttal \"*We respectfully disagree with the notion that our method can only be trained on synthetic data. Recent advances in 3D feedforward models [1, 2] have demonstrated the ability to generate high-quality depth maps from multiple views in the real scenario.*\"\n\nThis is a claim that is not supported empirically. Either show that this method can be trained on real data, or acknowledge that this is a limitation. The table referenced above still applies for synthetic data only (correct me if I am wrong). \n\nRebuttal: \"*We acknowledge that models such as Cosmos demonstrate strong 3D spatial reasoning; however, these models are trained with extremely large-scale datasets and computational resources, typically accessible only to major companies. In contrast, our approach aims to enhance 3D spatial understanding without requiring massive data or GPU resources.*\"\n\nYes, indeed, but this is not the claim that the paper makes. If this is the claim, it should be stated precisely like this. This is not what the paper states: *A key challenge in combining video models with the 3D spatial feature map is that most video models are limited in the abilities of 3D spatial understanding.* (also, is this really the key challenge in \"in combining video models with the 3D spatial feature map\")? \n\nStatements in the paper raised above are **not** sufficient reason to recommend rejecting this paper. Still, the paper is full of such issues, and these do compound (also, see reviewer's **VcUr** comments: \"*Almost every equation in the paper confused me. The authors are imprecise with their equations and definitions, and it greatly hurts understanding. See the Questions section for more information.*\", -- I strongly agree). \n\n\nAt the same time, I do not agree with **VcUr** that the fact that the code wasn't released (or videos) is a sufficient basis for rejection. If so, we would need to hold all papers to the same standards, and these standards should have been outlined in the reviewer guidelines. \n\nI am not strongly objecting to the acceptance of this paper, but given several presentation issues and a lack of trust that these will be addressed, I think this paper should undergo another round of revision.", "rebuttal": "Dear Reviewer,\n\nWe appreciate the reviewer for the detailed comments and insightful suggestions. We are glad to hear that you find our paper is well motivated and our experimental validation is extensive. Here is the response to your question.\n\n> \"Our model takes the current RGB-D observation, action, and 3D memory as input and synthesizes an RGB-D video\", but I understand that memory is part, not input, to the model; it is built implicitly.\n\nWe would like to clarify that, in our method, 3D memory is explicitly constructed and integrated into the model by learning memory blocks, as illustrated in Figure 2 and Figure 8 (Appendix). Our memory blocks are composed of expert adaptive layernorms, cross-attention blocks, and feedforward blocks. We will revise the paper to better clarify it.\n\n> The paper often uses references with absolutely no context, making the paper not self-contained. For example: \"Drawing from TesserAct [40], we separately encode RGB and depth with 3D VAE.\"; a better statement would be \"we do XYZ, following [40].\" Another example of weird sentence without context is \"We also downsample the Plücker camera embeddings with the same compression rate using UnPixelShuffle layers [26].\" There are many such cases.\n\nWe appreciate your suggestions and will thoroughly review all references to ensure they are introduced with appropriate context, making the paper easier to follow.\n\n> The proposed work can only be trained on synthetic data. The paper argues that this issue could be mitigated with depth generation models, but I am skeptical.\n\n> I would have hoped to see this method trained and evaluated on at least more challenging synthetic datasets (For example, Fabri et al., MOTSynth, ICCV'21, Virtual KITTI; these could be assessed for video generation without control).\n\nThank you for your insightful feedback. We respectfully disagree with the notion that our method can only be trained on synthetic data. Recent advances in 3D feedforward models [1, 2] have demonstrated the ability to generate high-quality depth maps from multiple views in the real scenario. \n\nDespite the added challenges posed by dynamic scenes and noisy depth maps, our memory approach still demonstrates improvements in visual fidelity. These results suggest that our method is robust to moderate noise in the depth input, and we believe that further advancements in depth estimation will enhance the generality and applicability of our approach.\n\n| Virtual KITTI | LPIPS $\\downarrow$ | PSNR $\\uparrow$ | SSIM $\\uparrow$ | \n| -------- | -------- | -------- | -------- |\n| NWM | 0.1989 | 20.2 | 0.625 |\n| Ours | 0.1731 | 21.4 | 0.660 |\n\n[1] Wang, Shuzhe, et al. \"Dust3r: Geometric 3d vision made easy.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Wang, Jianyuan, et al. \"Vggt: Visual geometry grounded transformer.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n\n> Papers states in 2.2: \"For efficiency, we extract the meaningful grids from the 3D map.\". Would you please explain?\n\nWhen constructing the 3D memory, some grids may be empty or contain very few points. For efficiency, we filter out these empty or sparsely populated grids and only retain grids with meaningful information. This reduces computation and memory usage without compromising the quality of the 3D memory representation.\n\n> Also, could authors elaborate on the statement: \"A key challenge in combining video models with the 3D spatial feature map is that most video models are limited in the abilities of 3D spatial understanding.\" To the best of my knowledge, this statement is inconsistent with findings in the literature.\n\n\nWe acknowledge that models such as Cosmos demonstrate strong 3D spatial reasoning; however, these models are trained with extremely large-scale datasets and computational resources, typically accessible only to major companies. In contrast, our approach aims to enhance 3D spatial understanding without requiring massive data or GPU resources. By leveraging 3D-structured memory, depth-aware generation, and precise camera control, our method enables effective 3D reasoning in a more accessible and resource-efficient framework."}, {"review_id": "OXF3C144oW", "reviewer": "Reviewer_VcUr", "summary": "The authors suggest that prior work on world models ignore spatial coherency (i.e., room appearance may change upon revisit). In this work, the authors propose a transformer world model that utilizes a 3D map of the environment. Given an RBGD image $o$, action $a$, and memory representation $M$, the authors predict the next observation $o'$. The authors utilize a pretrained image encoder to learn a low dimensional representation of the current image, then use a pretrained transformer backbone to represent the transition function. The authors' major contribution is to model $M$ using a matrix of DiNOv2 features. They take the features from an RGBD image and project these features into the corresponding spatial location in the 3D memory tensor $M$. These features are extracted in the world model via cross attention, enabling better consistency in the latent state representation. Finally, the authors use a diffusion model to generate the next RGBD image.\n\nThe authors train their model via a two-step process. First, they finetune their model without $M$. Then, they freeze their model and train $M$. The authors construct a train and test dataset using 50k trajectories collecting from the Habitat simulator. Then, the authors compare to four baselines: two alternate models and two ablations. They ablate their model without depth and with a 2D map. They find that their model outperforms the two alternative models, as well their ablations. Next, the authors compare their world model to two alternative models, demonstrating that their model has better scene revisitation consistency. Finally, the authors evaluate their model for planning purposes in a policy learning and model-predictive control framework. They show that their method produces better policies than two other approaches.", "questions": "- Line 41: Accurately -> accurate\n- Line 128: \"is video models\" -> \"is the video model\"\n- Figure 2: What are DiT blocks? These are not defined anywhere in the paper.\nEquation 2: What is $\\epsilon$? Is this noise? If so, how do you define it? Should it be time-dependent (i.e., $e_i$?)\n- Equation 2, 3: What is $c$?\n- Equation 4: What is $R(V)$? You define the reward function as $r$. \n    - In the following paragraph, you say \"The above optimization objective searches for sequences of actions a\", but in the preceding paragraph, you find $a^*$ by \"optimiz[ing] an action\"\n- Line 133: You say you \"train the memory blocks only and freeze other parameters\". The memory blocks themselves are just tensors, not parameters. Which weights do you actually train here? It is just the cross attention weights? Do you train the feedforward network as well?\n- Line 178: You perform an ablation of your model without depth? How is a depth-free variant of your model expected to work, given that you use the depth information to project the features into the 3D memory tensor?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The writing style is generally good\n- The authors provide many useful figures comparing their approaches to others\n- Their topic, consistency in world models, is an important and timely topic\n\n### Weaknesses\n- Given the focus on video generation, it is suspicious that the authors do not provide any generated video. It is easy to cherry-pick a couple nice looking frames for figures, while having poor video trajectories. This alone is grounds for rejection, in my opinion.\n- The authors do not provide any code to validate their approach or compare the results to other baselines. Again, another reason for rejection. Given the recent explosion on the number of world model papers published, reproducability is a necessity.\n- Almost every equation in the paper confused me. The authors are imprecise with their equations and definitions, and it greatly hurts understanding. See the *Questions* section for more information.\n- The \"no-depth\" ablation does not make sense, please see *Questions* for more information.\n\n> However, a fundamental challenge exists for embodied world models: the underlying state of the world is represented as a single image or chunk of images\n\nThis is false. The paper that invented the term \"world model\" represents the state using an RNN [1]. A world model, by definition, relies on the POMDP framework and represents the  latent state using either an RNN or transformer.", "was_revised": true, "final_justification": "I am in-between a borderline reject and accept. Since the authors shared their code with the AC, I will give the authors the benefit of the doubt and vote for borderline accept.", "rebuttal": "Dear Reviewer,\n\nWe thank the reviewer for the detailed comments and questions. We are glad to hear that you find our paper is well-written and our problem is significant. Here is the response to your question.\n\n> \"the underlying state of the world is represented as a single image or chunk of images.\" ... The paper that invented the term \"world model\" represents the state using an RNN.\n\nWe would like to clarify that in this context, our use of \"embodied world models\" specifically refers to recent large video diffusion models, as discussed in [1, 2]. We acknowledge that the original \"world model\" framework employed RNNs for state representation, and we will make this distinction clearer in our revision.\n\n[1] Bar, Amir, et al. \"Navigation world models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Yang, Mengjiao, et al. \"Learning interactive real-world simulators.\" arXiv preprint arXiv:2310.06114 1.2 (2023): 6.\n> do not provide any generated video.\n\nIn the main paper and Appendix, we included detailed frame-by-frame visualizations to highlight both the strengths and potential failure cases that could be missed in video playback. Unfortunately, due to rebuttal submission policies, we are unable to upload or share additional video material during this stage.\n\n\n> do not provide any code\n\nAs stated in our submission, we will open-source both our code and dataset upon acceptance of the paper. We would send the anonymous link related to the code if available at this stage.\n\n\n> What are DiT blocks?\n\nDiT blocks [1] refer to using Transformers as the backbone of diffusion models. In our work, these are implemented in the backbone CogVideoX [2]. We will clarify this in the revision to make it explicit for readers.\n\n[1] Peebles, William, and Saining Xie. \"Scalable diffusion models with transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[2] Yang Z, Teng J, Zheng W, et al. Cogvideox: Text-to-video diffusion models with an expert transformer[J]. arXiv preprint arXiv:2408.06072, 2024.\n\n> Equation 2: What is $\\epsilon$? Is this noise? If so, how do you define it? Should it be time-dependent?\n\n$\\epsilon$ is noise and is sampled from a Gaussian distribution. This formulation is standard in diffusion models, as also used in CogVideoX [1] and Stable Video Diffusion [2]. We will clarify this notation and provide additional details in the revision.\n\n[1] Yang Z, Teng J, Zheng W, et al. Cogvideox: Text-to-video diffusion models with an expert transformer[J]. arXiv preprint arXiv:2408.06072, 2024.\n\n[2] Yao C H, Xie Y, Voleti V, et al. Sv4d 2.0: Enhancing spatio-temporal consistency in multi-view video diffusion for high-quality 4d generation[J]. arXiv preprint arXiv:2503.16396, 2025.\n\n> Equation 2, 3: What is $c$?\n\n$c$ refers to the camera pose transform, which is defined in Section 2.1.\n\n> Equation 4: What is $R(v)$? You define the reward function as $r$.\n\nReward function $r$ is the reward of a single step. Return $R(V)$ is the sum of the rewards in the videos. In our paper, $R(V)$ is defined as the reward of the last frame, following Navigation World Model [1].\n\nWe will clarify these definitions in the revision. Additionally, we note that it is common in reinforcement learning literature (e.g., Policy Gradient [2]) to denote the return or expected cumulative reward as $R$.\n\n[1] Bar A, Zhou G, Tran D, et al. Navigation world models[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 15791-15801.\n\n[2] Sutton R S, McAllester D, Singh S, et al. Policy gradient methods for reinforcement learning with function approximation[J]. Advances in neural information processing systems, 1999, 12.\n \n > \"The above optimization objective searches for sequences of actions a\", but in the preceding paragraph, you find $a^*$ by \"optimiz[ing] an action\"\n\nModel predictive control (MPC) will first initialize the solution space with a Gaussian distribution, then search candidate solutions in the solution space, which will be used to optimize the solution space based on the expected rewards $R(V)$.\n\n> Line 133: You say you \"train the memory blocks only and freeze other parameters\". The memory blocks themselves are just tensors, not parameters. Which weights do you actually train here? It is just the cross attention weights? Do you train the feedforward network as well?\n\nTo be precise, the memory features are tensors, whereas the memory blocks are neural networks that include both cross-attention and feedforward networks, as shown in Figure 2. When we state that we \"train the memory blocks only and freeze other parameters,\" we mean that we update the weights of the cross-attention and feedforward networks within the memory blocks, while keeping the rest of the model parameters fixed. We will clarify this in the revision.\n\n\n> How is a depth-free variant of your model expected to work?\n\nWe used pre-trained depth estimation models (Depth Anything v2 [1]) to estimate the depth of the generated videos, and then updated the 3D memory.\n\n[1] Yang, Lihe, et al. \"Depth anything v2.\" Advances in Neural Information Processing Systems 37 (2024): 21875-21911."}]

Author rebuttal:

Meta-review: The paper introduces an embodied world model by incorporating a 3D memory, which leads to better long-horizon generations. After the discussions, the paper received Accept, two Borderline Accept and one Borderline Reject ratings. 

Reviewer MRYi maintained their reject rating mainly due to the unjustified claims and lack of clarity. The AC checked the paper, the rebuttal and the discussions and found the paper to be of merit and recommended acceptance as it provides a novel approach for conditioning video generation on a spatial memory. However, the authors must revise the paper according to the comments. 

Also, the paper solves a problem similar to slightly older works such as the ones mentioned below. These papers or similar ones should be included in the related work.

[1] Koh et al., Pathdreamer: A World Model for Indoor Navigation\
[2] Kotar et al., ENTL: Embodied Navigation Trajectory Learner
Decision: Accept (poster)