Title: Learning 3D Persistent Embodied World Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Embodied World Models
Abstract: The ability to simulate the effects of future actions on the world is a crucial ability of intelligent embodied agents, enabling agents to anticipate the effects of their actions and make plans accordingly. While a large body of existing work has explored how to construct such world models using video models, they are often myopic in nature, without any memory of a scene not captured by currently observed images, preventing agents from making consistent long-horizon plans in complex environments where many parts of the scene are partially observed. We introduce a new persistent embodied world model with an explicit memory of previously generated content, enabling much more consistent long-horizon simulation. During generation time, our video diffusion model predicts RGB-D video of the future observations of the agent. This generation is then aggregated into a persistent 3D map of the environment. By conditioning the video model on this 3D spatial map, we illustrate how this enables video world models to faithfully simulate both seen and unseen parts of the world. Finally, we illustrate the efficacy of such a world model in downstream embodied applications, enabling effective planning and policy learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors suggest that prior work on world models ignore spatial coherency (i.e., room appearance may change upon revisit). In this work, the authors propose a transformer world model that utilizes a 3D map of the environment. Given an RBGD image $o$, action $a$, and memory representation $M$, the authors predict the next observation $o'$. The authors utilize a pretrained image encoder to learn a low dimensional representation of the current image, then use a pretrained transformer backbone to represent the transition function. The authors' major contribution is to model $M$ using a matrix of DiNOv2 features. They take the features from an RGBD image and project these features into the corresponding spatial location in the 3D memory tensor $M$. These features are extracted in the world model via cross attention, enabling better consistency in the latent state representation. Finally, the authors use a diffusion model to generate the next RGBD image.\n\nThe authors train their model via a two-step process. First, they finetune their model without $M$. Then, they freeze their model and train $M$. The authors construct a train and test dataset using 50k trajectories collecting from the Habitat simulator. Then, the authors compare to four baselines: two alternate models and two ablations. They ablate their model without depth and with a 2D map. They find that their model outperforms the two alternative models, as well their ablations. Next, the authors compare their world model to two alternative models, demonstrating that their model has better scene revisitation consistency. Finally, the authors evaluate their model for planning purposes in a policy learning and model-predictive control framework. They show that their method produces better policies than two other approaches.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The writing style is generally good\n- The authors provide many useful figures comparing their approaches to others\n- Their topic, consistency in world models, is an important and timely topic\n\n### Weaknesses\n- Given the focus on video generation, it is suspicious that the authors do not provide any generated video. It is easy to cherry-pick a couple nice looking frames for figures, while having poor video trajectories. This alone is grounds for rejection, in my opinion.\n- The authors do not provide any code to validate their approach or compare the results to other baselines. Again, another reason for rejection. Given the recent explosion on the number of world model papers published, reproducability is a necessity.\n- Almost every equation in the paper confused me. The authors are imprecise with their equations and definitions, and it greatly hurts understanding. See the *Questions* section for more information.\n- The \"no-depth\" ablation does not make sense, please see *Questions* for more information.\n\n> However, a fundamental challenge exists for embodied world models: the underlying state of the world is represented as a single image or chunk of images\n\nThis is false. The paper that invented the term \"world model\" represents the state using an RNN [1]. A world model, by definition, relies on the POMDP framework and represents the  latent state using either an RNN or transformer.\n\nQUESTIONS:\n- Line 41: Accurately -> accurate\n- Line 128: \"is video models\" -> \"is the video model\"\n- Figure 2: What are DiT blocks? These are not defined anywhere in the paper.\nEquation 2: What is $\\epsilon$? Is this noise? If so, how do you define it? Should it be time-dependent (i.e., $e_i$?)\n- Equation 2, 3: What is $c$?\n- Equation 4: What is $R(V)$? You define the reward function as $r$. \n    - In the following paragraph, you say \"The above optimization objective searches for sequences of actions a\", but in the preceding paragraph, you find $a^*$ by \"optimiz[ing] an action\"\n- Line 133: You say you \"train the memory blocks only and freeze other parameters\". The memory blocks themselves are just tensors, not parameters. Which weights do you actually train here? It is just the cross attention weights? Do you train the feedforward network as well?\n- Line 178: You perform an ablation of your model without depth? How is a depth-free variant of your model expected to work, given that you use the depth information to project the features into the 3D memory tensor?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses action-guided video generation from RGB-D inputs: given past observations and control signal, the goal is to synthesize future frames that are consistent with the observed world. Recently, such models are termed 'world models' in the literature (i.e., given past observations and control signal, the task is to predict unobserved video frames based on the internal representation of the world). \n\nThis paper's main contribution is persistent, spatial volumetric memory, built by storing visual (DINO) features extracted from newly synthesized frames in a voxel grid. Such construction is possible when input frames consist of image and depth pairs (in contrast to video/camera only inputs). \n\nTo generate output images (and corresponding depth maps), the proposed model encodes video + depth, combines encoded RGBD input with relative poses (wrt. camera pose) using Pluecker embeddings, and uses cross-attention layer that augments encoded representation with 3D spatial features, stored in the spatial memory. \n\nThe method is trained and evaluated on the Habitat Simulation dataset using synthetic data (and pixel-perfect depth maps).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* The paper is reasonably well written\n* The problem statement is crisp and clear: Does maintaining persistent spatial memory and conditioning video generation on this memory improve action-conditioned video generation?\n* Proposed approach with spatial memory presents improvements wrt. variety of metrics that asses visual generation quality, and is shown to work exceptionally well for long-term video generation, which is intriguing\n* I also appreciate experiments on using the trained model in robotic navigation and planning (solid improvements wrt. ATE and RPE).\n* Overall, experimental validation is extensive, covering video generation quality, ranking of trajectories generated by a pre-defined policy, model-predictive control and policy learning. Paper even provides visuals, showcasing long-term video generation, and generated views that were not covered by the camera trajectory (unseen scenes). \n\n### Weaknesses\n\n**Presentation**: \n* Fig 2 stages that \"Our model takes the current RGB-D observation, action, and 3D memory as input and synthesizes an RGB-D video\", but I understand that memory is part, not input, to the model; it is built implicitly. This presentation is confusing. \n\n* The paper often uses references with absolutely no context, making the paper not self-contained. For example: \"Drawing from TesserAct [40], we separately encode RGB and depth with 3D VAE.\"; a better statement would be \"we do XYZ, following [40].\" Another example of weird sentence without context is \"We also downsample the Plücker camera embeddings with the same compression rate using UnPixelShuffle layers [26].\" There are many such cases.\n\n**General:**\n\nThe proposed work can only be trained on synthetic data. The paper argues that this issue could be mitigated with depth generation models, but I am skeptical. The accumulation of multiple views with depth generation models in a voxelgrid will usually not lead to consistent results; it might be that the proposed work only works with perfect depth data. \n\nI would have hoped to see this method trained and evaluated on at least more challenging synthetic datasets (For example, Fabri et al., MOTSynth, ICCV'21, Virtual KITTI; these could be assessed for video generation without control). I know further work may be needed to handle such dynamic scenes, but it is also great to learn where/how the proposed methodology breaks down. \n\nNevertheless, I believe that the proposed contribution (and its significance wrt. geometry-consistent RGBD generation) was clearly established on at least one dataset. I think this idea should be discussed in the community and further explored in future work. \n\n\nI am not very familiar with the state of research in this field, so for the time being, I assume that the selection of dataset and baselines for the evaluation is reasonable. I will revisit this / check again when I see comments of other reviews that may be more familiar with action-conditioned video generation and state-of-the-art in this field.\n\nQUESTIONS:\nPapers states in 2.2: \"For efficiency, we extract the meaningful grids from the 3D map.\". Would you please explain?\n\nAlso, could authors elaborate on the statement: \"A key challenge in combining video models with the 3D spatial feature map is that most video models are limited in the abilities of 3D spatial understanding.\" To the best of my knowledge, this statement is inconsistent with findings in the literature. \n\n\nReferences:\n[A] Agarwal, Niket, et al. \"Cosmos world foundation model platform for physical ai.\" arXiv preprint arXiv:2501.03575 (2025).\n\nLIMITATIONS:\nThe paper points out that key limitations are the static world assumption (present in the dataset), and assuming perfect depth maps as input.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors incorporate 3D memory features into video diffusion models to enhance agent's understanding of the environment. The goal is to build an embodied agent with long horizon and spatial consistency even during occlusions or revisits.\n- Instead of direct 3D supervision, the model extracted dino features per image and projected them into a 3D volumetric representation. The output predictions are in RGBD format with built in depth understanding, making the model more robust than 2D approaches. \n- The results section presented a clear improvement of visual quality and consistency. The paper also empirically demonstrated that the memory aware model benefits robotics path planning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- The paper is well motivated to incorporate persistent 3D understanding in the video diffusion model and embodied agent path planning. While there is existing literature on 3D aware video generation, the specific implementation details (e.g. using DINO feature, training memory modules with freezed DiT backbones) provide insights for future papers.\n- The paper is well-written, clear and provides solid evidence to support the claims. \n\nWeakness\n- The paper lacks discussion on the method scalability. The volumetric representation could be intractable in large and complex scenes, limiting the usage of this representation in real scenarios.\n- The paper would benefit from further discussion and experimentation on the specific 3D representation methodology. While briefly mentioned in the related works section, it is unclear how the proposed methodology is superior to alternatives like Nerf and InfiniCube. There is also a lack of ablation study to justify the choice of dino feature, as opposed to alternatives such as using point cloud.\n- Like the authors stated in the limitations section, the current setup is constrained, and does not apply to dynamic and changing scenes.\n\nQUESTIONS:\n- What are the rationales in choosing the proposed 3D representation method, as opposed to alternatives like point cloud, nerf, InfiniCube? Are there ablation studies run on alternative 3D representations, or ablation studies on not using dino features?\n- What’s the long horizon boundary of the given method? While the paper qualitatively illustrates good video generation results up to t=112, at what point does the memory module start to fail?\n- What is the end-to-end inference latency for future frame generation and dynamic memory updates, and how does the latency compare with other methods in the ablation? Is the inference time real-time?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a persistent embodied world model --  a new method for keeping spatial memory of a room baked into the world model. The authors propose incorporating an explicit 3D memory mechanism into video diffusion models, enabling consistent long-horizon simulation of embodied agents navigating complex environments. The key innovation is maintaining a persistent 3D feature map that gets updated as the agent explores. The model generates RGB-D (color + depth) videos conditioned on current observations, actions, and this 3D memory map. This allows the model to accurately reconstruct previously seen areas when revisiting them, rather than hallucinating new content.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper proposes a novel 3D feature map integrated as conditioning with video diffusion models to act as spatial memory.\n- The model shows significant improvements across multiple metrics and demonstrates practical benefits in downstream tasks like trajectory ranking and model predictive control.\n- The paper presents a complete pipeline \n\nWeakness:\n- There could be some more evaluations relating the quality of the generations against Novel view synthesis baselines such as Stable Virtual Camera\n- There could be some evaluations of the physical collisions this model learns, or other world modeling properties that could be baked into this approach -- otherwise the functionality is quite similar to a nerf\n- There could be more analysis of the memory volume produced, what sort of information is store in there, what can we lineary decode, how interpretable is it.\n\nQUESTIONS:\nWhat type of information is stored in this model volume? Can you linearly decode a semantic map of the room? \n\nWhat is the sensitivity to voxel size ? How bit of a maximum scene can this approach handle?\n\nHow does this compare to existing novel view synthesis and world models out there such as stable video camera? Does the explicit memory approach only show benefits when the trajectories are very long? And in that case how does it compare to simply using an every updating NERF to store a representation of the environment?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}