Title: 3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal

llm

3D

embodied

memory

agent
Abstract: Humans excel at performing complex tasks by leveraging long-term memory across temporal and spatial experiences. In contrast, current Large Language Models (LLMs) struggle to effectively plan and act in dynamic, multi-room 3D environments. 
We posit that part of this limitation is due to the lack of proper 3D spatial-temporal memory modeling in LLMs. 
To address this, we first introduce 3DMem-Bench, a comprehensive benchmark comprising over 26,000 trajectories and 2,892 embodied tasks, question-answering and captioning, designed to evaluate an agent's ability to reason over long-term memory in 3D environments.
Second, we propose 3DLLM-Mem, a novel dynamic memory management and fusion model for embodied spatial-temporal reasoning and actions in LLMs. 
Our model uses working memory tokens, which represents current observations, as queries to selectively attend to and fuse the most useful spatial and temporal features from episodic memory, which stores past observations and interactions. Our approach allows the agent to focus on task-relevant information while maintaining memory efficiency in complex, long-horizon environments.
Experimental results demonstrate that 3DLLM-Mem achieves state-of-the-art performance across various tasks, outperforming the strongest baselines by 16.5\% in success rate on 3DMem-Bench's  most challenging in-the-wild embodied tasks.


Peer reviews:
comments: SUMMARY:
This paper first proposes a 3D benchmark that is called 3DMEM-Bench, covering 26000 trajectories and 2892 embodied tasks. Then this paper proposes the formulation of 3DLLM-EME, which is a model with dyanmic memory management and fusion for embodied spatial-temporal reasoning and actions. In details, this model us working memory tokens to select from episodic memory. Experiments on 3DLLM-MEM shows the effect of the proposed method.

STRENGTHS AND WEAKNESSES:
Strength:

This paper builds a new 3D benchmark, and the idea of working memory and episodic memory is interesting.

Weakness:

1. I think the idea of querying tokens from the episodic memory is rational. However, it may cause the huge computation for reasoning. The authors should list the efficiency comparison. Moreover, if the thinking process is too long, the environment will be changed a lot, and the past observations and interactions might be useless. 

2. Especially, I think this model can be only adopted to the static environment, while cannot handle the dynamic environment. Especially, the interactions with others are also not described.

3. This model is built on the base model of LLaVA-3D. Thus, I wonder whether the advantage of 3DLLM-MEM is derived from the base model or the memory setting or the training data? This is not analysized sufficiently.

4. The authors could discuss some failing cases.

QUESTIONS:
1. What is the efficiency of the proposed model compared with other methods?

2. How to adopt this model into scenes with more dynamics and interaction? 

3. What is the contribution of each component in your framework?

LIMITATIONS:
No limitation is claimed in this paper. I think this paper should discuss more about their limitations in dealing with various scenes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a 3D Vision-Language-Action (3D-VLA) model equipped with long-term memory capabilities. In the context of this work, a 3D-VLA model begins with random exploration, during which it samples scenes randomly and collects RGBD + pose pairs for selected actions (exploration phase). This is followed by task-trajectory execution, where the agent (i) performs actions, (ii) captures corresponding visuals, and iterates until the task is completed or terminated.\n\nThe novel contribution of this paper, compared to prior work, is its argument that working memory, commonly used in such agents, is insufficient for tasks requiring reasoning over extended horizons. It further posits that existing benchmarks are too simplistic to evaluate such capabilities effectively.\n\nTo address this, the paper introduces a new benchmark based on a synthetic Habitat dataset (covering 182 spaces and 2.6k rooms). Tasks are generated using Gemini, leveraging pre-known scene/object geometry and semantics, and are filtered/verified via a trajectory simulator. A key strength of the dataset is its categorization of trajectories into three difficulty levels (based on the number of rooms required to complete a task) and its division into in-domain and in-the-wild settings (the latter includes areas not seen during model training).\n\nOverall, this benchmark offers a robust testbed for evaluating long-term action planning in indoor environments.\n\nThe second contribution is a baseline model that extends LLAVA-3D with long-term memory capabilities. The base network projects patches (tokens) from image/RGBD pairs into the LLM’s embedding space, enabling them to be processed alongside instruction tokens by the base LLM. For spatial reasoning, these embeddings are augmented with 3D positional encodings. To incorporate long-term memory, the model maintains a queue of past observations (termed episodic memory) in addition to the standard LLM context window, enabling retrieval. A memory fusion module allows tokens from active memory to attend to episodic memory, producing a fused representation that is then fed to the base LLM. This enables the LLM to operate not only on active memory but also on context retrieved from long-term storage.\n\nSTRENGTHS AND WEAKNESSES:\n###  Strengths \n\n* The paper tackles a significant and underexplored challenge in the 3D-VLM domain: long-term exploration.\n* The proposed benchmark is a great contribution and has the potential to advance the field. The benchmark construction process is innovative, particularly with the inclusion of objects from Objaverse (to address the lack of objects in Habitat), alongside LLM-based trajectory generation and simulation-based verification.\n* The proposed methodology is a sound adaptation of prior work (LLAVA-3D) and long-term memory mechanisms established in the literature.\n* The experimental evaluation is convincing, and I appreciate the inclusion of well-constructed baselines.\n\n### Weaknesses\n**Prior work & contextalizing this paper w.r.t prior art.** \n\nThe paper effectively discusses related work in the 3D-VLM domain. However, its treatment of long-term memory techniques is incomplete and poorly contextualized. The proposed mechanism appears heavily inspired by existing work in the LLM domain, which extends the \"working memory\" window by attending to a subset of previously processed data (stored as a KQV queue) retrieved via kNN. For instance, Wu et al.’s “Memorizing Transformers” (ICLR ’22) employs a similar approach (note that the paper states in Section 3.2: “To manage episodic memory, we propose the use of a memory feature bank,” whereas it should acknowledge following established practice).\n\nWhile extending this concept to the 3D-VLM domain is a notable contribution, the omission of this discussion is concerning.\n\n\n**Presentation:**\n\n1. Certain sections of the paper are well-written, particularly those drawing parallels between memory modules and neuroscience, which I appreciate. However, the paper falters in addressing fundamental questions: What is the task at hand? What are the model’s inputs and outputs? What is the problem statement or task definition? The paper is so focused on the “how” that it fails to clearly communicate the “what.” While this information can be pieced together from the paper or related works, its absence in the introductory section places an unnecessary burden on the reader. I only fully understood the task definition in the “Embodied Data Collection” paragraph of Section 2.2 (Data Collection).\n\n2. The paper builds upon LLAVA-3D and augments it with long-term memory capabilities. To be self-contained, it should include a preliminary section briefly describing the LLAVA-3D architecture. Additionally, the paper would benefit from a clear problem setting and task definition section before delving into data collection, generation, and modeling, with these elements foreshadowed in the introduction.\n\n**Why is the proposed method superior to the Retrieval-Augmented Memory baseline?**\nThe aforementioned baseline seems very similar to the proposed method, yet experimentally, the proposed model performs significantly better. It is unclear from the experimental evaluation where this difference comes from, as well as what precisely is the key difference at the methodology level responsible for the performance gains. From the paper description, it appears to me that the key difference is in how the memory fusion is performed?  \n\n**Nit:**\n* Abstract: “reason over long-term memory in 3D environments” → consider rephrasing to “reason over extended temporal spans using long-term memory” for clarity.\n* Table 1 analyzes different datasets and includes a column labeled “Long-Term Memory.” Benchmarks do not possess memory, so this label is misleading.\n\n\nI am genuinely excited about this work, but its current presentation issues are significant. I am open to revising my rating upward after the rebuttal if my concerns are adequately addressed.\n\nQUESTIONS:\nOn the memory update, the paper states:\n\n\"The working memory is dynamic and updated online. As the agent interacts with the environment, changes in the environment are immediately reflected in the working memory through updated 3D representations. When the agent moves to a new environment, the previous working memory is transferred to the episodic memory bank. If the corresponding environment already exists in the memory bank and has been modified by the agent, the memory entry is updated accordingly\". \n\nCould you please clarify how precisely this is implemented/executed? This appears rather important, I imagine that 'carrying' around full past is in practice not feasible.  \n\nAlso, please see **Why is the proposed method superior to the Retrieval-Augmented Memory baseline?** above.\n\nLIMITATIONS:\nA single sentence in the conclusion: \"One limitation of our model is that currently 3DLLM-MEM does not involve low-level navigation and control policy, but utilizes high-level pre-defined policies in simulator for carrying out the actions\".\n\nI think this discussion should have been expanded. The success rate of the proposed method is still low (not surprising, the task is hard), but readers would greatly benefit from understanding where the proposed method still fails. Also, at the moment, this model is entirely constrained to synthetic data. There is no evidence for generalization outside of Habitat sandbox. I understand that evaluation outside of that may be difficult, but the bottom line is: no, limitations are not adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the challenge of 3D spatial understanding and memory retrieval in 3D-LLMs. To that end, the authors propose 3DLLM-Mem, a 3D LLM that features a novel memory fusion mechanism inspired by human memory and cognitive maps. In particular, the model uses working memory and episodic memory tokens to represent current and past observations and retrieve relevant memories.\n\nTo evaluate this method and support other research in this direction, the authors also introduce 3DMem-Bench, a benchmark with trajectories and tasks spanning embodied navigation, question answering, and captioning. They benchmark their method, along with recent 3D-LLMs and memory mechanisms, and show that their method significantly outperforms prior work on all tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Proposed memory fusion mechanism is novel and well-justified\n- The proposed 3DMem-Bench benchmark is the first benchmark focusing on long-term memory and fine-grained complexity, and good details about the benchmark design principles and dataset collection process are provided\n- Strong results on both navigation and embodied QA tasks, demonstrating significant improvement over prior works\n- Demonstrates strong generalization to unseen environments\n\n**Weaknesses**\n\nThat said, I have some concerns about the clarity of the method and generalization contribution.\n1. Throughout the paper, there are some terms that are not properly defined and missing details, which make parts of the method unclear. In particular:\n    - It’s unclear what the difference is between the in-domain vs in-the-wild tasks. A better understanding of this would be helpful for understanding the generalization capabilities of 3DLLM-Mem.\n         - The details in Section 2.3 about how the in-domain and in-the-wild datasets are collected are highly specific, which makes it harder to understand what the core difference is between in-the-wild and in-domain tasks. \n         - Also, this section references “instances” and “unseen memory context”, which are not defined earlier and also make the description unclear\n    - The definitions of the environment, working memory, and episodic memory are unclear\n         - I’m confused about what observations are part of the working versus episodic memory. In line 182, it says that the current observation at each time step is stored in working memory, implying that the working memory stores a single timestep. In lines 204-205 though, it seems like the working memory is longer since it explains that “When the agent moves to a new environment, the previous working memory is transferred to the episodic memory bank.” \n         - It would help to have a concrete formulation of these terms, as they’re core to the method\n    - Details about training are missing.\n\n2. The paper emphasizes that 3DLLM-Mem demonstrates strong generalization to unseen environments. While 3DLLM-Mem has much higher absolute performance on in-the-wild tasks, the relative drop in performance between in-domain and in-the-wild is only slightly lower than other methods. More insight into this would strengthen this contribution, such as explaining which components of the method lead to this improvement.\n\nQUESTIONS:\n1. What are the core differences between in-domain and in-the-wild tasks?\n2. How are the following terms defined: instance, environment, unseen memory context? \n3. What is the relationship between the working and episodic memory, and how are observations transferred from one to the other? Does the working memory contain observations from more than a single timestep, and if so, how is a new environment (as mentioned in lines 204-205) determined?\n4. Do you have insight into why 3DLLM-Mem demonstrates strong generalization capabilities? Is this simply due to better overall performance? Or are there specific components of the method that enable it to maintain performance compared to other approaches?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel benchmark 3DMEM-BENCH for reasoning, planning and acting with long-term spatial-temporal memory in embodied environments. In the meantime ,this paper introduces  a 3D embodied LLM with dynamic memory management capabilities designed specifically for embodied spatial-temporal reasoning, planning and acting, and evaluates popular 3D-LLMs and memory mechanisms on 3DMEM-BENCH.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n1. This paper collected a fine-grained benchmark with multiple difficulty levels and varying numbers of multi room scene settings, which can assist the model in evaluation.\n2. This paper proposes a new mechanism for modeling long-term spatio-temporal memory, which interacts with attention and historical memory to effectively improve the performance of the model in complex scenarios.\n3. The approach proposed in the paper when switching environments involves storing working memory in the episode memory bank, which greatly reduces costs through this mode of thinking.\n\nWeaknesses\n1. The paper describes how memories are added to and updated in the episodic memory bank, but not how they might be pruned or consolidated. In extremely long-horizon scenarios, this could lead to the information redundancy and the memory bank growing indefinitely, which might reintroduce the very problems of computational inefficiency.\n2. While a significant effort was made to validate the data , tasks generated by an LLM (Gemini)  and executed within a simulator may not fully capture the physical world. This creates a risk that the model may be learning to exploit artifacts of the simulation rather than developing skills that are robustly transferable to real-world problems.\n\nQUESTIONS:\nSection 2.2 notes a low 24% validation rate for Gemini-generated trajectories. What were the most common failure modes that caused these trajectories to be invalid (e.g., illegal navigation, violations of object interaction rules (such as the \"hold only one object at a time\" constraint mentioned in Appendix F.1)? Could you provide several failure case analysis?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}