3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model
Authors: Wenbo Hu
Yining Hong
Yanjun Wang
Leison Gao
Zibu Wei
Xingcheng Yao
Nanyun Peng
Yonatan Bitton
Idan Szpektor
Kai-Wei Chang
Venue: NeurIPS 2025 poster ()
Keywords: multimodal
llm
3D
embodied
memory
agent
Primary area: deep_learning
TL;DR: We introduce a new benchmark for long-term spatial-temporal memory in 3D embodied agent. We propose a novel model with memory fusion technique for enhanced memory capabilities.

Abstract: Humans excel at performing complex tasks by leveraging long-term memory across temporal and spatial experiences. In contrast, current Large Language Models (LLMs) struggle to effectively plan and act in dynamic, multi-room 3D environments. 
We posit that part of this limitation is due to the lack of proper 3D spatial-temporal memory modeling in LLMs. 
To address this, we first introduce 3DMem-Bench, a comprehensive benchmark comprising over 26,000 trajectories and 2,892 embodied tasks, question-answering and captioning, designed to evaluate an agent's ability to reason over long-term memory in 3D environments.
Second, we propose 3DLLM-Mem, a novel dynamic memory management and fusion model for embodied spatial-temporal reasoning and actions in LLMs. 
Our model uses working memory tokens, which represents current observations, as queries to selectively attend to and fuse the most useful spatial and temporal features from episodic memory, which stores past observations and interactions. Our approach allows the agent to focus on task-relevant information while maintaining memory efficiency in complex, long-horizon environments.
Experimental results demonstrate that 3DLLM-Mem achieves state-of-the-art performance across various tasks, outperforming the strongest baselines by 16.5\% in success rate on 3DMem-Bench's  most challenging in-the-wild embodied tasks.

3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model
Wenbo Hu les
Yining Hong'
Yanjun Wang'
Leison Gao
Zibu Wei Kai- Chang!
Xingcheng Yao
Nanyun Peng'
Yonatan Bitton
Idan Szpektor"
University of California, Los Angeles, Google Research
https://3d1 1m mem. .oinniiiee github.io
Prepare the most suitable gift box for the teddy
-Explore the environment> Found the wddy bear. <Pick up teddy bear?
-Navigate to the hodroom
-Naviggtt the living room search for gift "Try placing the teddy bear into gift bax-
-Nuvvigate the kitchen and search gift box- Try placing the feddy beur into the gift
and search for gifl bux- -Try placing the feddy beur into the gift bux- Too small.
It fits, but it's a bit loose.
Ii Lis, but too living
box the
suitable.
Navigaate to bux the living roon-- teuld bear into the gift
Task complete.
Figure 1: We propose 3DLLM-MEN, memory-ennhnnced 3D embodied agent that explores and incorporates feedback from the environment, interacts with objects, and incrementally builds and maintains a task-relevant long-term memory throughout its trajectory. For illustration purposes, agents from multiple time steps are shown simultaneously.
Abstract
Humans excel at performing complex tasks by leveraging long-term memory across temporal and spatial experiences. In contrast, current Large Language Models (LLMs) struggle to effectively plan and act in dynamic, multi-room 3D environments. We posit that part of this limitation due to the lack proper 3D spatial-temporal memory modeling in LLMs. To address this, we first introduce 5DMEM-BENCE comprehensive benchmark comprising over 26,000 trajecto- ries and 2,892 embodied tasks, question-answering and captioning, designed to evaluate an agent's s ability to reason over long-term memory in 3D environments. Second, we propose 3DLLM MEM, novel dynamic memory management and fusion model for embodied spatial pemporll reasoning and actions LLMs. Our model uses working memory tokens, which represents current observations, queries selectively attend to and fuse the most useful spatial and temporal fea- tures from episodic memory, which stores past observations and interactions. Our approach allows the agent to focus on task-relevant information while maintaining memory efficiency in complex, long-horizon environments. Experimental results
x< Contact at whu @ www......ccc..u.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

demonstrate that 3DLLM- MEM achieves state-of-the-an performance across var- ious tasks, outperforming the strongest baselines by 16.5% in success rate on SDMEM-BENCHT most challenging in-the-wild embodied tasks.
1 Introduction
Picture yourself traversing an unfamiliar home, as illustrated in Figure I, on a mission to explore multiple rooms and evaluate various gift boxes to find the most suitable one for wrapping teddy bear. As you navigate from room to room, your brain instinctively creates a 3D cognitive map of the environment, maintains a working memory of objects you' ve encountered, forms episodic memories that link observations across space and time, and plans efficient actions. This seamless integration of 3D spatial understanding, long-term memory encoding and retrieval, fluid switching between working and episodic memory, and purposeful action planning - cognitive processes that humans take for granted remain formidable challenges for embodied AI systems today.
Recent extensions of Large Language Models (LLMs) to 3D environments have birthed 3D- LLMs (Hong et al., 2023b; Guo et al., 2023; Gu et al., 2024; Huang et al., 2024b; Xu et al., 2025a) that can perceive and reason about 3D spaces, while 3D Vision-Language-A Action models (Zhen et al., 2024; Zhao et al., 2025; Intelligence et al., 2025) further incorporate the ability to plan and act within these environments. Despite these advances, several critical limitations persist that prevent models from performing the kinds of tasks described above. First, current models struggle to main- tain long-term memory chains when performing complex tasks that unfold across multiple visual scenarios, such as several rooms in house, and extended time frames. Real-world 3D physical scenes are remarkably vast and information-dense, where every detail can matter for embodied tasks for instance, in Figure 1, finding the most suitable gift box requires remembering all the gift boxes encountered along the way and their characteristics and interaction with teddy bear. Dense 3D representations are particularly valuable as they capture comprehensive spatial information, preserving intricate geometric relationships and environmental details that sparse or object-centric approaches might miss. However, how to accurately and efficiently store dense 3D memory remains a fundamental challenge- retrieving the entire history would overwhelm the model's 's context limits, while selective retrieval (Xie al., 2024; Wang et al., 2024; Yang et al., 2025b) risks omitting critical information needed for accurate reasoning and decision decision-making. The second challenge resides in the entanglement of spatial and temporal memory agents must track not only where objects are, but how they change over time through exploration and interaction. As environments evolve, maintaining coherent representations of previously seen spaces while incorporating new information continues to exceed the capabilities of current embodied AI models.
Our efforts solving this challenge are two-fold. First, we introduce novel benchmark for reason- ing, planning and acting with long-term spatial-temporal memory in embodied environments. Our benchmark. 3DMEM- -ENCHH encompasses multi-room 3D scenes from the Habitat environment, augmented with interactive objects to enable manipulation tasks across extended spatial-temporry horizons. Notably, we define fine-grained embodied tasks across varying levels of difficulty- from simple to hard hard-enabling deeper insight into model performance, which we believe is not addressed in prior benchmarks as shown in Table 1. Our task set spans a wide range of complexities, from straightforward object collection to challenging comparative reasoning tasks that require integrating observations across multiple rooms and time steps. Additionally. we include in-the-wild challenge tasks to evaluate the model's generalization capabilities beyond seen environments. The benchmark includes three evaluation categories: (1) embodied tasks requiring extended action sequences across multiple rooms, (2) spatial-temporal embodied question answering (EQA) that evaluates understand- ing of spatial relationships over time, and (3) long-term scene captioning that tests memorization of previously observed environments. Our dataset includes 26,000+ trajectory examples spanning 182+ unique scenes with an average of 18 rooms per scene.
Second, we introduce 3DLLM- MEM, 3D embodied LLM with dynamic memory management capabilities designed specifically for embodied spatial-temporal reasoning, planning and acting. To our knowledge, we are among the first to explore dense 3D representations as memory for embodied 3D LLMs addressing a significant gap in current research as noted in recent 3D memory studies (Yang et al., 2025b). Unlike standard approaches that rely solely on context windows (Hong et al., 2023b; Huang et al., 2024b; Zhu et al., 2024), 3DL.MMMFM implements a dual-memort system: limited-capacity working memory for current observations and an expandable episodic
2

---

Beuchmark
#Test Tasks #Train Trajectories Long-tem Memery Fine-grained complexity EQA Captioning
ALFWorld (Shridhar al., 2021)
274
3,553
NA
NA
Behavior- KK (1. ct al., 2024a)
1,000
NA
X
X
NA
NA
VisuulAgemtBench (Liu et al., 2024)
746
4,482
NA
EmbodiedBench (Yang et al., 2025a)
1,128
NA
NA
NA
3DMRM-BENCH (ours)
1,860
26,276
865
167
Table Comparison with related benchmarks. 3D BENCH focus on spatial-temporal memory through fine-grained embodied tasks and EQA that span multiple "pieces" of long-term memory, distinguishing from prior benchmarks that typically target single-step or short-horizon reasoning. Fine Fine-grained complexity indicates our embodied task spans from simple medium to hard.
memory that stores past spatial-tempon information as dense 3D representations. The key innovation is our memory fusion module that actively integrates information from both memory systems based on task relevance and spatial- ppatial-temporal relationships. This allows the model to leverage the benefits of dense 3D representations while mitigating their computational demands, maintaining coherent spatial-temporal understanding across extended task horizons. The fusion process preserves critical spatial relationships while accounting for their evolvement through agent interactions over time.
We evaluate popular 3D-LLMs and memory mechanisms on SAMMM--BENCH Experimental results demonstrate 3DLLM-M MEM significantly outperforms all cxisting approaches in both in in-domain and in-the-wild embodied tasks. Notably, while the performance of other methods drops sharply in the challenging in-the- in-the-wild setting, our method remains robust, achieving an average success rate of 32.15 %% demonstrating strong generalization capabilities. As task complexity increases from simple to hard, all existing approaches degrade significantly, achieving only ~5% success rate in hard in-the- wild tasks. contrast, 3DLLM- -MEM maintains a strong performance of 27.5 77, demonstrating its scalability and effectiveness in managing longer-term memory representations.
Our contributions can be summarized as below:
We propose novel task that requires agents execute action chains while maintaining and utilizing memory.
long-term 3DMEM-BENCH a comprehensive benchmark comprising over 26,000 trajectories
We construct and 1,860 fine-grained long-term memory embodied mankinganging from simple to hard-alomt with question-ansswrring tasks that target memory changes across time and space, and captioning tasks in complex 3D environments,
We propose 3DLLM-MEN embodied 3D LLM with novel memory fusion module for spatial- temporal reasoning, planning, and acting-which utilizes working memory tokens as queries to selectively fuse relevant features from episodic memory for efficient, task-aware decision-making. Experimental results on embodied tasks, question-answe answering, and captioning demonstrate that EDLLM-MEN outperforms baselines by a large margin.
2
The Embodied 3D Long-Term Spatial-Temporal Memory Benchmark
2.1 Overview of 3DMEM-BENCH
Design principles Long-term memory (Camina and Güell, 2017; Friedman et al., 2018; Zlomik and Vansintjan, 2019) can be categorized into explicit memory and implicil memory. Explicit memory includes semantic memory, which stores general knowledge and facts about the world, and episodic memory, which consists of personal experiences that are time-stamped and context-specific. In contrast, implicit memory primarily involves procedural memory, such as learned skills and habits.
To comprehensively evaluate 3D long-term memory for real-world applications, we design 3DMEM- BENCH following three core task categories: embodied tasks, long-term memory EQA, and captioning. As illustrated in Figure 2, embodied tasks require an embodied agent to solve realistic indoor environ- ment challenges by leveraging both implicit and explicit long-term memory. Long-tern memory EQA tests the agent's 's ability to answer complex embodied questions using task includes five subcategories: spatial reasoning questions, long-term object navigation, compara- tive reasoning, multi-room layout understanding. and semantic object counting. Cuptioning tasks involve summarizing the agent's 's episodic memory to highlight shared and distinctive features across experiences, enabling more informed decision decision-making making under the current task context.
memory. This
3

---

Long-term Memory Embodied Tasks
In-domain Tasks
In-the-wild Challenges
In-the-witl Tasks
Instruction: Find an ideal container all the cookic dough.
Instruction: Collect cookie dough and then rearrange them table of living from in descending order of size.
Fine-grained comparison
Instruction: Preparing simple breakfast. The break fast includes toust with buller and jam, cup coffee.
Instruction: Stop exploring additional rooms and serve the breakfast
Force Finish
need find toaster, coffee machine, butter, jam, bread, coffee powder......
was searching for the toaster but remembered seeing the mierowave in the kitchen. <Navigate kitchen:
Task
Reschedule
No coffee powder found whewhere... From my memory, the ten bag the counter and water ketile on the lable.
<Navigate to kitchen?
<Pick up bag
<PPul Lea bag into the iuug
<Navigute to the dining room?-
Serve hot tea instead. Task complete.
Long-term Memory EDA
Captioning
Object Counting
Comparative
Q: How many ociling lamps are there after task finished?
Q: Whick carpet has larger area now, carpet(() room carpet()) (1) room A: Carpel(O) in room is larger.
Episodic Memory Captioning
3
Q: Give a caption covering the summary common features previous exploration and the difference that matlers for your current observation.
A: There are coiling lamps in total.
Spatial Relation
Object Navigation
Q:After took execation.
Q: If you stand next the Plant(O) facing window(0)
you're standing at tablo(0).
in room 15, how to navigate to kitchen cabinct(0y)
facing chuir( I), is the dresser(()
A: You need first turn left and go struight to see stovetop(0)) tray(O) and then the kitchen cabinet(() right on your front.
The previous rooms contained storage cahinets, beds, dressers, and kitchen counters
your left, right, behind?
A/ (resser(() to your right.
Room Layout
In room 8, what is the relative layout of the refrigemtton(O), dining table(()), and stove(() bifore task execution?
This room uniquely features wardrobe extending along an cutire wall and does not contain any pictures, lamps
A: The (efrigerator(()) is located on one side, dining table(0) the middle, and the stove(0) on the side kitchen.
Figure Overview of 3DMEM-BENCH. BENCH. For long-term memory embodied tasks, we further incorporate in-the-win wild challenges to test 3D agent's generalization abilities. Text inside < indicates high-level action tokens. For complete embodied task trajectories, please refer to Appendix C.
2.2 Data Collection
Base environment construction We build our scenes on top of the Habitat-Matterpon 3D (HM3D) semantics dataset (Ramakrishnan et al., 2021), which has 1000 3D spaces and 10,600 rooms within those spaces. Pre-processing for the axis-aligned bounding box and using valid semantic label annotation, we filter to 182 3D spaces and 2,602 rooms. However, existing objects in HM3D scene not interactive in Habitat-sim (Szot et al., 2021). To expand our task diversity and enable embodied tasks, we add interactive objects from Objaverse (Deitke et al., 2023) which consists of 800K 3D objects spanning rich categories. More environment construction details are illustrated in Appendix B.
Generating task trajectories Following Hong et al. (2023b, 2024), we adopt dox-dmmonttration- instruction-basel prompting, which utilizes the axis-aligned bounding boxes (AABB) of both rooms and objects within the 3D scenes to prompt Gemini (Team et al., 2023) to generate diverse tasks. We further prompt Gemini to incorporate interactive objects based on task requirements and their appropriateness within indoor environments. Detailed prompt instructions and few-shot demonstration examples are provided in Appendix E. To ensure the validity of the generated trajectories, we develop a trajectory simulation pipeline that verifies each trajectory step-by-step. At every step, the simulator checks: (1) the correctness of the agent's 's location, (2) the existence and validity of referenced objects, and (3) the correctness of uick-up and put-down actions. Finally, we ensure that high-level actions can be executed in the simulator, following (Szot et al., 2024; Yang et al., 2025a). Details of this implementation are in Appendix F. On average, our filtering process yields a validation rate of approximately 24%, ensuring the correctness and feasibility of the generated trajectories.
Embodied data collection In our task settings, an embodied agent first performs random ex- ploration within the environment collect (GB-D observations and corresponding camera poses.

---

up breads <Put in toasters
Thinking: coffee found.
<Rick up bag? into tea mug?
Eptsodic Momory
LLM
Tea bag boiling kottle memory. cNavigate Michenn
chavigate
eNavigate dring room>
to kitcheno
Task Complete
Memory Fusion
3DLLM-Mem
Fpisodic Monory
Working Memory
Mewory Tokows
Memmory 1m1
Memory 2
Memory
Sample & Projection
Memory
Memory Fusion
3D Patch Tokens
PODOM Purch
Monury
3D Position
...
Memory I
Embeddings
Encoder
Encoder
3D Inputs
(a)
(b)
Figure 3: (a) We propose WOLLM-MEME a memory-enhanced 3D embodied agent that gradually form its long-term memory while executing tasks. Multiple timesteps shown together but in different colors, with each timestep's ss memory including the prior one. The task is "prepare a simple breakfast" as shown in Figure 2. (b) Overview of memory fusion mechanism.
Then the agent follows the task trajectory, incrementally exploring environments, executing interaction actions, and receiving feedback with
new RGB-D observation data. All interaction results are recorded and the reconstructed point cloud data is precomputed and stored locally to enable faster
loading during both training and inference.
2.3
Data Curation
As mentioned previously, we collect embodied data by prompting Gemini. To enable fine-grained analysis of long-term memory capacity, we divide the tasks into three subcategories: simple, medium, and hard, comprising of 3, 5 and 10 multi-room scene settings respectively. In total, we collect 51K trajectories, with 31K in the simple setting. 10K in the medium, and 10K in the hard.
To construct in-domain evaluation sets, we first remove training tasks and filter for instances that never shown the agent's working memory. For the in-the-wild evaluation set, we apply additional filtering to assess the agent's generalization capabilities. Specifically, we select instances involving unseen objects and entirely unseen memory context, and we introduce novel in-the-wild challenges that differ from those encountered during training, as illustrated in Figure 2.
For BQA data curation, we extract complete trajectories explored by agents and then prompt Gemini to generate question-answer pairs. The questions are categorized into spatial reasoning, long-term object navigation, comparative reasoning, multi-room layout understanding, and semantic object counting. As shown in Figure 2, these questions evaluate models on spatial-tempon memory during embodied task execution. For long-term memory captioning, which primarily targets semantic episodic memory, we collect data across multiple rooms before and after the execution of each trajectory, enabling comparison and summarization of memory-
changes in
experiences.
Quality control After constructing the entire benchmark, we implement two quality control procedures: automatic validation using trajectory simulation rules and a manual review of each benchmark instance. The automatic check involves re-running the trajectory simulation validation pipeline, as described in $2.2, particularly for the in-the-wild tasks. For human validation, four student experts in the field manually inspect each benchmark example. We render multi-vies -ielw images of the entire scene using the simulator and verify whether the benchmark annotations accurately correspond to the simulated environment. More details are in Appendix F.2.
3
3D Long-Term Spatial-Temporet Memory Model (3DLLM-MEM)
3.1
Preliminary
Recent work on 3D Large Language Models (3D-LLMs) has showcased robust capabilities. We choose LLaVA-3D (Zhu et al., 2024) the base model to build our long-tern memory 3D-LLM.
5

---

LLDVA-3D directly builds on 2D-LLM with multi-view images as imput and utilizing the 3D position embeddings bring the 2D patches within 3D spatial context to construct 3D patches. For each frame image, a CLIP encoder splits the image X E 23xWxH into patches at the patch size P. For each 3D scene, V multi-view image patch features are encoded and then projected into L.M space W as Xp E TR. Vxxxxxxx, where h = [P] w P . and d represents LLM's hidden dimension. The 3D positions in the 3D world are obtained with known depth image, camera intrinsic and extrinsic parameters and are further encoded into 3D position embeddings P E RV xxxxxxx These are directly added to the 2D patch visual tokens Xp. resulting in pixel-aligned 3D patches X3D E RR XxVxxxxx To reduce redundancy in 3D patches, adopt the Farthest Point Sampling (FPS) strategy to downsample the 3D features to a fixed number of tokens, resulting in XID Feat E R R xd
3.2 3DLLM-MEM Memory Module
A 3D embodied agent gradually explores the environment by collecting observations and interacting with surrounding environments. For humans, current observations are held in working memory, while longer-term observations and experiences are stored in episodic memory. Inspired by human cognitive structure, 3DLLM-MEM is designed with a similar paradigm as illustrated in Figure 3. The current observation time step f i. denoted as X((==) E R Nxd remains within the context window and serves as the agent's ' working memory. As the agent accumulates more experiences, past observations from time steps 1 to T. represented as X(==1:T] E TR xd are stored as part of its episodic memory, where T denotes the total number of timesteps.
Episodic memory To manage episodic memory, we propose the use of a memory feature bank. For each observation time step j. where و S T, we first apply multi-layer perceptron (MLP) layer to project the observation into memory-specilf feature space, which is then stored in the memory bank for future retrieval. To further enhance the temporal understanding of the agent's exploration, we incorporate sinusoidal positional embeddings to encode each time step t = j. and then directly added to the corresponding memory feature representations.
Memory fusion Our motivation is that an agent should leverage its current observations to recall the most relevant information from its episodic memory in order complete the current task. To achieve this, we propose mechanism called 3D memory fusion. Specifically, we encode the 3D features from the working memory into a shared memory space and use this representation as the query feature, denoted as 1 E p.N XM where M is the dimensionality of the memory feature space.
The episodic memory bank stores the corresponding key and value features from past observations: fk E ERT TR and ʃ RTxNxXM respectively. Here, T is the number of past timesteps and N is the number of memory tokens per timestep. This structure allows the agent to retrieve task-relevant information through memory-query attention. The fused memory feature is then concatenated with the working memory feature to produce the final
representation f. for the agent:
( VC
fuse
Softmax
PV,
fM
Concat
f
(I)
Memory update The working memory is dynamic and updated online. As the agent interacts with the environment, changes in the environment are immediately reflected in the working memory through updated 3D representations. When the agent moves to a new environment, the previous working memory is transferred to the episodic memory bank. If the corresponding environment already exists in the memory bank and has been modified by the agent, the memory entry is updated accordingly. Thus, the memory bank remains dynamic and reflects the latest state of the explored environments. As described in 52.2, environment changes and corresponding observations are pre-collected and stored locally to facilitate efficient data loading during both training and inference.
4 Experiments
In this section, we first introduce the experimental setup and existing memory management baselines in 1. Then, we benchmark existing approaches on HOMEM--ENNH and present comprehensive results on embodied tasks, EQA, and captioning tasks to demonstrate the effectiveness of our

---

Simple
Medium
Average
Model
In-domain
In-the-wild
In-domain
In-thewwild
In-domain
In-the-wild
In-domain
In-the-wild
SR
Sub-SR
SR
Sub-SR
SR Sub-SR
SR
Sub-SR
SR
Sub-SR
SR
Sub-SR
SR
Sub-SR
SR Sub-SR
3D-LLM (Finetuned)
10.4
20.3
9.1
18.5
Everything in
324
Most Recent Memory
52.8
62.3
23.4
38.6
20.1
34.8
124
25.3
20.7
12.1
21.1
39.3
13.7
25.3
Setrivaal-Aggmenne Memory
34,2
63.0
28.3
46.2
21.8
40.2
13.7
28.0
10.8
21.6
4.8
10.6
22.3
41.6
15.6
28.3
3DLLM MEN (Ours)
14555
7344
37.0
65.4
136.8
67.8
316
57.4
30.5
46.2
27.8
42.1
(3..6
62.5
32.1
55.0
(a) Results on 3DMEM-BENCH embodied tasks. SR stands for success rate. Sub-SR stands for sub-success rate.
Our model outperforms existing approaches by large margin.
Embodied Task
Embodied Question Answering (EQA)
Captioning
Model
In-domain In-the-wild Spatial Nav. Comparative Layout Count
BLEUI
BLEU4
METEOR
3D-I.LM (Finetuned)
2.9
5.8
0.0
7.7
0.0
42.3
12.0
30.6
3D-Mem (GPT4-o)
39.9
11.0
25.8
19.1
7.8
41.7
4.7
31.8
3D-Mem (eemini-2.--Flushh
41.6
18.2
37.6
30.2
12.7
42.8
4.8
29.6
3D-Mem (Gemini-2,5-Pro
39.7
27.7
36.0
35.2
16.4
41.5
3.0
28.6
Most Recent Memory
21.1
13.7
27.5
30.2
24.3
20.1
10.5
32.4
10.1
25.6
Memory
22.3
15.6
38.0
33.4
31.8
29.7
15.6
40.8
11.5
29.3
3DLLLM MEM (Ours)
37.6
32.1
62.8
40.6
41.4
39.9
26.3
58.2
18.8
37.3
(b) Results on all tasks in 3DMEM- BENCH. Average success rate is reported for embodied tasks, Nov stands for long-term term object navigation. We report accuracy score open open-ended EQA evaluation and follow the standard LLM-as-judde evaluation protocol by prompting Gemini. Evaluation details are provided in Appendix E.
Table 2: Comparison with 3D memory models and standard memory management approaches. Our model, BOLLM-MEM, achieves the best performance across embodied, EQA and captioning tasks.
3DLLM- MEM in $4.2, along with qualitative results. Finally, in 84.3, we conduct an ablation study of key design choices in 3DLMMMMEM, demonstrating the effectiveness of our proposed memory fusion mechanism.
4.1 Experimental Setup
Implementation details We implement model based LLaVA-3D (Zhu et al., 2024), modify- ing it to be compatible with Google TPUs with PyTorch/XLA frameworks (Paszke et al., 2019; team, 2017-2025). We first expand the model's context window to 8192 tokens to accommodate long-term memory inputs. We then fine-tune our proposed memory module along with the LLM decoder using our training split, initializing from LLaVA-3D's pretrained weights. Training is conducted on 8 Google Cloud TPU v5p cores with a batch size of 256. Our model is trained using supervised fine-tuning (SFT) with a standard language modeling loss. More details are provided in Appendix D.
Baselines We compare 3DLLM-MEM against broad range of memory management approaches:
Everything in Context. For a small subset of scenes, it is feasible to fit all observations directly into the model's context window.
Most Recent Memory. Since retaining all observations in context is infeasible, we keep only the most recent observations, assuming they are most relevant to the current task.
Setrieval-Augmentee Memory. Inspired by retrieval-based techniques, we adopt memory bank that stores past observations. During inference, the most relevant memory entries are retrieved and appended before the working memory to augment reasoning.
3D-LLM (Hong et al., 2023b). A popular 3D LLM recognized by the community. We finetune on our training data and report its performance using the "everything in context" strategy with the longest context window supported. Further details are provided in Appendix G.
3D-Mem (Yang et al., 2025b). A framework designed for 3D scene memory in embodied exploration and reasoning. However, this method does not support embodied interaction action execution.
4.2 Experimental Results
Results on embodied tasks As shown in Table 2a, 3DLLM-MEM significantly outperforms all existing approaches on both in-domain and in-the-wild embodied tasks. Notably, while the performance of other methods drops sharply in the in-the-wild setting, our method demonstrates strong generalization capabilities with average success rate of 32.1%. 3D-LLM showcases the lowest
7

---

Prepare cozy reading nook in the living room with two books and teacup.
D
Figure 4: Qualitative example of PDLLM-MENT which maintains and utilizes a long-ten memory to complete the task. Detailed task execution trajectory can be found in Figure 6.
performance even under simple task settings, highlighting the necessity of incorporating explicit memory module. Both the Most Recent Memory and Reerieval-Augmented Memory (RAG) baselines perform poorly in this setting, with RAG showing only slight improvement, highlighting the challenges of retrieving relevant episodic memory. Interestingly, the Everything in Context baseline performs better than both recent memory and RAG approaches, suggesting that when all information can fit within the context window, the model can effectively utilize it. However, 3DLLM- MEM still outperforms Everything in Context, indicating the benefits of selectively fusing task-relevant memory features to better guide embodied reasoning and execution. As task complexity increases from simple to hard, all existing approaches degrade significantly, achieving only ~5% success rate in hard in-the hild tasks. In contrast, 3DLLM- MEM maintains a strong performance of 27.8%, demonstrating its scalability and effectiveness in managing longer-term memory representations.
Results on long-term EQA and captioning As shown in Table 2b, 3DLLM- consistently outperforms all existing approaches across all tasks in our benchmark. Notably, 3D-LLM achieves the second -bett performance on the captioning task, highlighting its strong ability to summarize object-centric semantic memory. However, due to limited context length, it performs poorly on the EQA tasks, which require long-term spatial-temporal reasoning. 3D-Mem demonstrates improved performance in EQA over other baseline approaches. However, it falls short on spatial relation, navigation and object counting tasks, indicating the limitation of relying solely on aggregated image- centric memories. 3DLLM- MEM significantly outperforms both Most Recent Memory and RAG Memory, which further demonstrates the effectiveness of our memory fusion technique.
Qualitative results explanations in Figure (Appendix H), demonstrating that 3DLLM-MEM is capable of maintaining long-term memory and executing complex tasks embodied environments.
We provide qualitative examples in Figure 4 and a more detailed version with
4.3 Ablation Study
Our approach initializes the fused memory using working memory features, aiming fuse the most relevant memories for the current task. We ablate several design choices for initializing the fusion query, as shown in Table 3. When using either the most recent episodic memory or learnable zero parameters, performance degrades compared to our proposed method. Interestingly, using the most recent memory outperforms zero initialization in the simple setting but underperforms the hard setting. One possible explanation is that recent memory initialization encourages fusion with nearby observations, which may be sufficient for simple tasks and leads to faster convergence. contrast, zero initialization is guided solely by training supervision to learn which memories are most useful. In summary, the ablation results demonstrate that initializing fusion queries with working memory tokens provides the most effective and robust design choice for long-term memory fusion.
5 Related Works
3D Large Language Models 3D Large Language Models (3D-LLMs) have demonstrated promising results across a wide variety of tasks, including 3D scene understanding, object detection, and segmentation (Hong et al., 2023b; Zhou et al., 2024; Huang et al., 2024a; Chen et al., 2024b; Xu et al., 2025a). parallel, 3D embodied agents have expanded these capabilities to planning and action in interactive environments (Brohan et al., 2023; Huang al., 2024b; Chen et al., 2024a; Black et al.,
8

---

Simple
Medium
Hard
Average
Midel
In-domaiin
In-tho-wild
In-domain
In-th-wwil
In-domain
In-She-wild
In-domain
In-tho-wikd
Sub-SR
SR
Sub-SR
SR
Sub-SR
Sub-SR
Sub-SR
SR
Sob-SR
SR
Sob-SR
Sub-SR
3DLLM-MEM
66.4
3688
67.8
31.6
130.5
27.8
42.1
37.6
62.5
Init with Most Roceni Episudic Memorry
69.4
28.6
50.7
32.4
58.6
23.7
45.1
22.6
37.8
153
31.
32.
55.3
22.5
42.4
Init with Learmabl Zero Parameteos
41.4
67.2
27.9
500
33.0
59.3
234
43.8
24.2
40.4
186
35.6
32.9
55.6
23.3
Table 3: Ablation study of query initialization designs in memory fusion module.
2024). Yet, existing models face significant challenges when performing long-horizon embodied tasks in densely populated 3D environments that require reasoning over long-term spatial-tempont memory. To address this, we propose explicit memory module inspired by the structure of human implicit and explicit memory. Our model employs memory fusion mechanism that efficiently retrieves and learns task-relevant information, resulting in enhanced performance on complex embodied tasks.
Long-term Embodied Trajectories Embodied AI simulators (Chang et al., 2017; Kolve et al., 2017; Szot et al., 2021; Shen et al., 2021) have fostered the development of embodied Al agents. Grounded in these environments, some existing benchmarks focus on high-level planning tasks, typically involving short trajectories that can often be completed within single-room settings. thereby requiring minimal spatial-temporal memory (Shridhar et al., 2020, 2021; Li et al., 2024a; Szot et al., 2024; Li et al., 2024b; Yang et al., 2025a). Other benchmarks emphasize long-term scene exploration with extended trajectories, but are primarily centered around navigation tasks and often lack embodied interaction support (Deitke al., 2020; Ramakrishnan al., 2021; Krantz al., 2022; Khanna et al., 2024). To bridge this gap, we introduce PDMEM--BENHE a benchmark specifically designed to evaluate long-horizon task execution that requires rich spatial-temporal memory and full embodied lask support, as summarized in Table 1.
Embodied Question Answering Benchmark Embodied Question Answering (EQA) bench- marks (Das et al., 2018; Wijmans et al., 2019; Yu et al., 2019) have been developed to advance goal-driven agents that can perceive their environment. Some EQA benchmarks also include em- bodied memory QA evaluation, such as OpenEQA (Majumdar et al., 2024), which includes an episodic memory QA split, and Yang et al. (2024), which focuses on spatial memory QA. In contrast, our benchmark, 3DMEM- DOMEM-BENCH jointly targets both spatial and episodic memory, especially their changes over time, while also supporting embodied action tasks, EQA and captioning. For specific comparison on EQA, our long-term memory EQA tasks are designed to require reasoning over multiple "pieces" of memory and their changes across time and space. Additionally, we consider the agent's 's location in the scene at the moment of answering each question during evaluation.
Memory System Memory is a fundamental component of AI systems, with early work in the context of LLM agents that utilize memory for decision decision-making in web-based and sandbox environ- ments (Shinn al., 2023; Zhang al., 2023; Packer et al., 2023; Zhang et al., 2024). Most existing approaches construct an experience pool or memory bank and focus on improving the retrieval of useful past information (Zhao al., 2024; Gao et al., 2024; Xu et al., 2025b). the computer vision domain, temporal memory has been studied extensively in video understanding and generation tasks (Wang et al., 2021; Diao et al., 2025), while spatial memory has been applied to scene-level visual understanding and 3D reconstruction (Wang and Agapito, 2024; Zou et al., 2025). Recent work such as 3D-Mem (Yang et al., 2025b) has investigated 3D scene memory for exploration and reasoning by prompting vasion-language models. In contrast, our work focuses on dense 3D memory representations that are critical for real-world embodied scenarios, where task execution depends heavily on maintaining and reasoning over long-term
memory.
6 Conclusion
In this work, we introduce 3DMEM-BENCH a comprehensive benchmark containing fine- grained long-term memory embodied lasks-ranging from simple to hurd-along with tasks that target memory changes across time and space, and captioning task in complex 3D envi- ronments. We propose SDLLM-MEN an embodied 3D-LLM with novel memory fusion approach for spatial-temporal reasoning, planning, and acting. One limitation of our model is that currently 3DLLM-MER does not involve low-level navigation and control policy, but utilizes high-level pre-
9

---

defined policies in simulator for carrying out the actions. We think that such aspects are orthogonal to our study, and could be explored and seamlessly integrated into our framework in the future.
Acknowledgments and Disclosure of Funding
We thank anonymous reviewers and other members of UCLA-NLP+ group for their helpful comments. This work was partially supported by U.S. DARPA ECOLE Program No. HR001 12390060, ONR grant N00014-23-J- 1-23-1-2780 Amazon Research Award, and Google gift fund. Peng and Chang have financial COI with Google and Amazon and were supported in part by grant from DARPA to the Simons Institute for the Theory of Computing.
References
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. 2024. To! A libon-languago-actton flow model for general robot control.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gon- zalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee. Tsang-Wei Edward Lec, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. 2023. Rt-2: Vision- sirion-lnnuaag-acttin models transfer web knowledge to robotic control. volume 1ttp/2307.15888
Eduardo Camina and Francisco Güell. 2017. The neuroanatomical, neurophysiological and psycho- logical basis memory: Current models and their origins. Frontiers in Pharmacology, 8:438.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV).
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas J. Guibas, and Fei Xia. 2024a. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA. June 16-22, 2024, pages 14455-14465 14465. IEEE. 8
Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. 2024b. Grounded 3d-llm with referent tokens. ArXiv preprint, abs/2405. 10370.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling cntruction-finetunae language models. ArXiv preprint, abs/2210.1141e 24
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 1-10. IEEE Computer Society. 9
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, and Ali Farhadi. 2020. Robothor: An open simulation-to-real embodied Al platform. In 2020 IEFF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA. June 13-19, 2020, pages 3161-3171. IEEE. 9
10

---

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. 2023. Objaverse: A universe of annotated 3d objects. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 13142-13153. IEEE. 4
Xingjian Diao, Chunhui Zhang, Weiyi Wu, Zhongyu Ouyang, Peijun Qing, Ming Cheng, Soroush Vosoughi, and Jiang Gui. 2025. Temporal working memory: Query-guided segment refinement for enhanced multimodal understanding. 9
Gary N Friedman, Luke Johnson, and Zachary M Williams. 2018. Long-term visual memory and its role in learning suppression. Frontiers in Psychology, 9:1896.
Jinglong Gao, Xiao Ding, Yiming Cui, Jianbai Zhao, Hepeng Wang, Ting Liu, and Bing Qin. 2024.
Self-evolving gpt: A lifelong autonomous experiential learner. 9
Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. 2024. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021-5028. IEEE. 2
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, and Pheng-Ann Heng. 2023, Point-bind & point-llm: Align- ing point cloud with multi mudtiity for 3d understanding, generation, and instruction following. 2
Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan. 2023a. 3d concept learning and reasoning from multi-view images. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 9202-9212. IEEE. 21
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023b. 3d-lim: Injecting the 3d world into large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023. 2, 4, 7, 8, 24
Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, and Chuang Gan.2024
2024. Multiply: A multisensory object-centric embodied large language model in 3d world. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 26396-26406. IEEE. 4
Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang. Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, and Zhou Zhao. 2024a. Chat-scene: Bridging 3d scene and large language models with object identifiers. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024. 8
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. 2024b. An embodied generalist agent in 3d world. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. 2.8
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. 2025. To.s: vision-lagguage-action model with open-world generalization.
Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sriram Yenamandra, Théophile Gervet, Matthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mottaghi. 2024. Goat-bench: A benchmark for multi-modal lifelong navigation. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 16373-16383. IEEE 9
11

---

Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017. A12-THOR: An Interactive 3D
Environment for Visual AI. arXiv. 9
Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot. 2022. Instance- specific image goal navigation: Training embodied agents to find object instances. 9
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang. Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Kyu-Young Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. 2024a. Behavior- 1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. 3,
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. 2024b. Embodied agent interface: Benchmarking lims for embodied decision making. In Advances Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024.7 9
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. 2024. Visualagentbench: Towards large multimodal models as visual foundation agents. ArXiv preprint, 0ass2240.0062277. 3
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Vay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent-Pierre Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Rajeswaran. 2024. Openeqa: Embodied question answering in the era of foundation models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 16488-16498. IEEE. 9
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2023. Memgpt: Towards Ilms as operating systems. 9
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Florian Gribonval, Rafal Jozefowiez, et al. 2019. Pytorch. https:// pytorch. org/ 7, 23
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra, 2021. Habitat-matteryy 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. volume abs/2109.08233 4,
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Pérez-D''Ariino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent Vainio, Josiah Wong, Li Fei-Fei, and Silvio Savarese. 2021. igibson 1.0: a simulation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), page accepted. IEEE. 9
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16. 2023. 9
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10737-10746. IEEE. 9
12

---

Mohit Shridhar, Xingdi Yuan, Mare-Alexande Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. 2021. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. 3,9 9
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Tumer, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot. Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel X. Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. 2021. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 251-266. 4, 9
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, Devon Hjelm, and Alexander T. Toshev. 2024, Large language models as generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations, ICI. 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 4, 9
Gemini Team, Rohan Anil. Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. ArXiv preprint, abs/2312.11800 4
XLA team. 2017-2025.
Xla: Optimizing compiler for machine learning. https://www.
tensorflow.o wwt.fflwww.woox/l.o
owoooogg.....ooo..oo xla. 7.23
Hao Wang, Weining Wang, and Jing Liu. 2021. Temporal memory attention for video semantic segmentation. In 202/ IEEE International Conference on Image Processing (ICIP), pages 2254- 2258. IEEE, 9
Hengyi Wang and Lourdes Agapito. 2024. 3d reconstruction with spatial memory. ArXiv preprint, abs/2408. 16061. 9
Zixuan Wang, Bo Yu, Junzhe Zhao, Wenhao Sun, Sai Hou, Shuai Liang, Xing Hu, Yinhe Han, and Yiming Gan. 2024. Karma: Augmenting embodied ai agents with long-and-short term memory systems. 2
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. 2019. Embodied question answering in photorealistic environments with point cloud perception. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pages 6659-6668. Computer Vision Foundation IEEE.
Quanting Xie, So Yeon Min, Pengliang Ji, Yue Yang. Tianyi Zhang, Aarav Bajaj, Ruslan Salakhutdi- nov, Matthew Johnson-Roberson, and Yonatan Bisk, 2024. Embodied-rag: General non-parametric embodied memory for retrieval and generation.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. 2025a. Pointllm: Empowering large language models to understand point clouds. In Computer Vision ECCV 2024, pages 131-147. Springer Nature Switzerland. 2, 8
Wujiang Xu, Zujie Liang. Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025b. A-mem: Agentic memory for IIm agents. ArXiv preprint, abs/2502.12110.
Jihan Yang. Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. 2024. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. ArXiv preprint, 111/2 9
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. 2025a. Embodiedhench: Comprehensive benchmarking multi-modal large language models for rision-driven embodied agents. 3, 4,
13

---

Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 2025b. 3d-mem: 3d scene memory for embodied exploration and reasoning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR). 7, 26
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L. Berg, and Dhruv Batra. 2019. Multi-target embodied question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pages 6309-6318 Computer Vision Foundation IEEE.
Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao, and Kai Yu. 2023. Large language model is semi-parametrie reinforcement learning agent. ArXiv preprint, abs/2306.07929,
Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2024. A survey on the memory mechanism of large language model based agents.
Andrew Zhao, Daniel Huang. Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. Expel: LLM agents are experiential learners In Thirty-Eighth AAAI Conference Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Educational Advances in Artificial Intelligence, EAAI 2014, 4, February 20-27, 2024, Vancouver. Canada, pages 19632 19642. AAAI Press. 9
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, Ankur Handa, Ming- Yu Liu, Donglai Xiang, Gordon Wetzstein, and Tsung-Yi Yi Lin. 2025. Cot-vla: Visual chain-of-thought reasoning for vision-language- action models.
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 2024. 3d-vla: 3d assion-laguuage abs/2403.09631. 2
generative world model. ArXiv preprint,
Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. 2024. Uni3d: Exploring unified 3d representation at scale. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
Chenming Zhu, Tai Wang, Wenwei Zhang. Jiangmiao Pang, and Xihui Liu. 2024. Llava-3d: A simple yet effective pathway empowering imms with 3d-awareness. ArXiv preprint, abs/2400.18125 2, 5,
Guillermo Zlotnik and Aaron Vansintjan. 2019. Memory: An extended definition. Psychology, 10:2523. 3
Frontiers in
Xueyan Zou, Yuchen Song. Ri-Zhao Qiu, Xuanbin Peng, Jianglong Ye, Sifei Liu, and Xiaolong Wang. 2025. M3: 3d-spatial multimodal memory. In ICLR. 9
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Question:
Answer:[Yes Yes]
Justification:
We clearly state our main claims in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
15

---

Justification:
This paper doesn't 't introduce new theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we fully disclose all the information, please refer to Section and our experimental setup in Section 4.1
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the ssame dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We will release our code and data publicly after the review process. We also provide data sample and example code in the submitted supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. ctips.cc/public//uu guwdee/iee///ee CCodeSubeesiaceecceee simiisssinn ssssonoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to our implementation details in Section 4.1 and Appendix D. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We conduct experiments on Google TPU and the training of long- term-ttrmmmn memory 3D-LLM is expensive, we don't have the resources to run the experiments multiple times and calculate the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to our implementation details in Section 4.1 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tt/rrrrrs...// cccpublic//etcecccccccccceeeeeee ctw.eetteeehiileeeeeeeeeeeaeee
Answer: [Yes]
Justification:
We discuss ethics concern and broader impact in Appendix A.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss broader impact in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: We require users to follow the guidelines such as Llama's guidelines when release the model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly credited the original owners and followed their license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: We document our new introduced benchmark in Section 2.1 and Section 2.2. We also provide the documentation in dataset along with the data sample in the supple- mentary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: We authors conducted the human validation with clear instructions, no other research with external human subjects is performed.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
20

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA].
Justification: We only used LLMs for improving grammar in paper writing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https / //neurips..... Cetpurrpsscc//ontiieee CwCCooofeeenee///0 for what should or should not be described.
A Broader Impact
The deployment and release of 3DLLM-MEN carry both potential benefits and risks. These considerations include visual aspects as well common issues found in existing LLMs like Alpaca and Vicuna. Since 3DLLM-MEM built on LLaMA, Vicuna, and CLIP, it inherits certain challenges associated with LLMs and vision encoders. Below, we outline the risks and the mitigation strategies implemented for the release this model.
Hallucination Similar other LLMs, 3DLLM-MEM might produce outputs that are not based on factual information or input data. This raises concerns about the accuracy of inferences, particularly in critical applications such as medical fields.
Biases Biases present in the base models can be brought to 3DLLM-MEM. stemming from both the vision encoder (CLIP) and the language decoder (LLaMA Vicuna). This may result in biased outcomes or unfair representations of diverse content.
Energy Consumption We train our model on our training data split which contains about 26K trajectories. The training time only takes less than one day, which makes energy consumption not primary concern.
B Environment Construction
To support navigation-centric interaction, the agent requires precise knowledge of two things: the traversable layout of each scene and the exact locations of all movable objects. Following 3D-CLR (Hong et al., 2023a), we build this spatial substrate from HM3D's richly annotated indoor scans. We rely on the semantic surface mesh that accompanies each scene to calculate the room and objects' locations. The mesh scan has unique (24-bit) hexadecimal color for every surface triangle that provides a semantic table that links each color to surface label (e.g., floor or ceiling) and a room identifier.
We first derive an axis-aligned bounding box for every room in each HM3D scene. We query the semantic table to retrieve, for every room identifier, the hex colors assigned to its floor and ceiling surfaces. Before processing individual rooms aggregate the minimum heights of all floor surfaces in the entire scan, producing global set of candidate floor elevations. For each room we then load the point clouds of its floor and ceiling. When both are available, the vertical bounds are given by the floor's lowest point and the ceiling's 's highest point. If the floor is missing, the lower bound is set to the highest global floor elevation that still lies below the room't 's ceiling; the ceiling is missing, the upper bound defaults to the highest point in the available cloud (i.e., the surface immediately above the floor). Rooms for which both surfaces are absent are discarded because no reliable vertical evidence is present. Horizontal limits are simply the minimum and maximum coordinates of the
21

---

room's floor and ceiling points, and together these six coordinates constitute the room's 's axis- axis-aligned bounding box.
Based on the bounding boxes for each room, we calculate the bounding boxes of objects in that room by gathering the corresponding vertices and computing the bounding boxes of the object by taking the minimum and maximum coordinates along the three axes. We merge room-level and object-level bounding boxes by their shared room index. After the aforementioned processing, the dataset comprises 182 distinct 3-D scenes covering a total of 2,602 rooms.
C Embodied Task Data Examples
In this section, we provide complete embodied task trajectories.
Here an example of in-the-wild wild task:
Task: Rearrange desk items by size. The items to be rearranged are: vase, box, and printer. Trajectory:
"I'm in room 10. I will start exploring the room for the items I need.", "I see printer(O) on the (O) I also see a box(O) on the cabinet(O)."" ", "I need to remember their locations and sizes for comparison later.",
"<GO TO NEW ROOM>".
"Now I'm in room 11. I'm exploring this room for the items.", "I don't 't sec a printer(O) on rack(1).", ", "I will remember these objects and their locations.",
in the room, but I
"<GO TO NEW ROOM>",
"Now I'm room 6. I'm exploring this room.", "I do not see any of the items I am looking for.",
"<GO TO NEW ROOM>",
"I'm in room 8. am exploring this room.", "I see flower vase(0) on the floor. I will remember the object and its location.".
"<GO TO NEW ROOM>".
"I'm in room 7. I exploring this room.", "I do not any of the items am looking for.". "<GO TO NEW ROOM>",
"I'm in room 9. I am exploring this room.", ", do not see any of the items am looking for.",
"<GO TO NEW ROOM>".
"Now am in room
am exploring this room.", ", "I do not see any of the items am looking for.", "<GO TO NEW ROOM>".
"Now I am in room 2. am exploring this room.". ", "I see box(0) on the armchair(1). I will remember these objects and their locations.",
"<GO TO NEW ROOM>".
"I'm in room 1. I do not see any items that are part of the rearrangement"", "<GO TO NEW ROOM>",
"Now I'm m in room 12. I will start exploring this room.", ", "I think I've the vase, printer, and box in other rooms, so I'll head back to room to pick up the vase first.", "<GO TO ROOM(8)>".
"I need to pick up flower vO)e(O) in this room first. I pick up flower vase(O).", "<PICK UP flower vase(0) from room(8) in room(8)>".
"I'm carrying flower vase(O) to room 10 where saw printer and box.". "<GO TO ROOM(10)>"
"Now am in room 10. I will put flower vase(O) on the desk(O) for now so I can pick up box(O).".
"<PUT DOWN flower vase(0) from room(8) desk(() in room
"Now will pick up box(O) from cabinet(O)...
"<PICK UP box(O) from room( 10) in room 10)>".
"Now I will put box(O) on the desk(O) so I can pick up printer(O).",
"<PUT DOWN (0) from room (10) on desk(() in room(10)>", "Now I will pick up pOi from beside the desk(O).",
"<PICK UP (r) from room 10) in room(10)>",
need to compare the sizes of the vase, box and printer now to determine the rearranging order.", recall that printer(O) in this room had the largest bounding box dimensions based on my earlier observation. I need to put down printer(O)...
"<PUT DOWN printer(O) from room(10) on desk(O) in room(10)>
"Now I will pick up box(O).".
22

---

"<PICK UP (0 from room( 10) in (oom(10)>
"I remember that box(0) from cabinet(O) had medium bounding box dimensions, I will put on
the desk.".
"<PUT DOWN box(0) from room(10) on desk(0) in room(10)>".
"Now I will pick up flower vase(0).".
"<PICK UP flower vase(O) from r8)m(8) in room 10)>".
"I remember that flower vase(0) had the smallest bounding box dimensions, so I will put it on the desk.", ",
"<PUT DOWN flower vO)e(0) from room(8) desk(O) in room(10)>",
"I will now arrange the objects from small to large: flower vase(0), box(0) and printer(O) on the desk(0).". "Now I pick up the medium object, box(0).",
"<PICK UP box(O) from room( 10) in room(10)>?)
"I will put the medium object the left of the largest object, printer(O).",
"<PUT DOWN box(O) from room 10) on desk(0) in room(10>>",
"Now I pick up the smallest object, flower vase(0).".
"<PICK UP flower vase(0) from room(8) in room(10)>"
will put the smallest object on the left of the medium object, box(0).".
"<PUT DOWN flower vase(O) from room(8) on desk(() in room(10)>",
"The objects are now arranged from small to large: flower vase(0), box(0) and printer(O) on the desk(0). Task Complete."
Here is an example of in-domain task:
Task: Preparing for Cooking Stir-fried Tomato and Scrambled Eggs. You need to prepare the ingredients, seasonings and cooking tools.
Trajectory: "I'm room 4. It seems to be living room, need to prepare ingredients and seasonings and cooking tools and setup them in the kitchen", "There is seasonings on the table.", "<PICK UP seasonings(() from room(4) in room(4)>", "No other things found for my task", "<GO TO NEW ROOM>".
"I'm at room 5. It seems to be dining room. I saw an apron on dining table.", "I need the apron but I have seasonings(O) on my hand. Lets find kitchen to put the seasonings first and go back for apron", "<GO TO NEW ROOM>".
"I'm at room 8. It seems to be kitchen with two cabinets. I found tomatoes and eggs on the cabinet(1) that is far away from me and cooking pan in the sink.", "There are two countertops in this room. countertop( () is closer to stove(O). should setup on countertop(I).",
"<PUT DOWN seasonings(() from room(4) on countertop(1) 1) in room(8)>",
"<PICK UP tomatoes(O) from room(8) room(8)>",
"<PUT DOWN tomatoes(O) from room(8) countertop(1) l) in room(8)>",
"<PICK UP eggs(O) from room(8) (8 in room(8)>".
"<PUT DOWN c0g from room(8)
countertop(I) (1) in room(8)>", "<PICK UP cooking pO) from room(8) in room(8)>",
"<PUT DOWN cooking pOn from room(8) s0)ve(O) in room(8)>",
remember saw apron room 5. need to find back.",
"<GO TO ROOM(5)>",
"<PICK UP apron(0) from room(5) in room(5)>",
"<GO TO ROOM(8)>", "<PUT DOWN apron(0) from room(5) on countertop(1) in room(8)>", "The setup in kitchen has been prepared. Task Complete."
D Implementation Details
We implement our model based on LLaVA-3D (Zhu et al., 2024), modifying it to be compatible with Google TPUs with Py PyToch/XLL frameworks (Paszke et al., 2019; team, 2017-2025). We first expand the model's 's context window to 8192 tokens to accommodate long-term memory inputs. We then fine-tune our proposed memory module along with the LLM decoder using our training split, initializing from LLaVA-3D's pretrained weights. Training is conducted on 8 Google Cloud TPU v5p cores with batch size of 256 for 1000 steps, which takes about day to complete. We use Adam optimizer with learning rate of 2e-5 with no weight decay. Additionally, apply a linear warmup
23

---

of the learning rate during the initial 3% steps, increasing from 10-$ to 10-6, followed by cosine decay scheduler.
E Prompts for Gemini
As mentioned in $ 2.2, we prompt Gemini to generate the long-term trajectories as illustrated in Table 4, generate the creestion-answeriie tasks as shown in Table 5, and generate caption tasks as shown in Table 6. For open-ended QA evaluation, we followed standard I.LM-as-judge protocol by prompting Gemini as illustrated Table 7.
F
Data Validation
E1 Trajectory Validation
We implement a trajectory simulation pipeline driven by the commands listed in Table 4. For each command, the simulator records the agent's 's current room and the full set of objects it is holding, then updates the set of objects in each room to reflect pick-up and put-down actions. A pick-up removes the specified object (along with any nested items) from the room the agent occupies and adds it the agent's hand; a put-down removes the object from the agent's hand and places into the designated room. The pipeline validates each command based on these criteria: (1) the agent's 's location; (2) the referenced object and (3) the correctness pick-up and puw-down actions. For location validation, a command is marked as invalid if the agent attempts pick up object from room that does match its current room, or tries to drop an object into room other than the one it currently occupies. Additionally, if the agent tries visit new room when all rooms have already been explored, the trajectory is also considered invalid. For object validation, a pick-up command is invalid the target object does not exist in the current room, and a put-down command is invalid if the agent is not currently holding the specified object. For pick-up and put-down validation, the agent is allowed to hold only one object at a time. A command is considered invalid if the agent attempts to pick up an object while already holding one, tries to put down an object when its hand is empty. Finally, after all commands have been executed, if the trajectory ends with the agent still holding an object that was never put down, the entire trajectory is marked as invalid.
that does not exist in the scene, or attempts enter
F.2 Human Validation
As mentioned in $2.3 After automatic trajectory validation, we further conduct human validation, in which four student experts in the field manually inspect each benchmark example. We render multi- view images of the entire scene using the simulator and verify whether the benchmark annotations accurately correspond to the simulated environment as illustrated in Figure 5.
G Evaluation Setup Details
3D-LLM Similar to the 3D-LLM work (Hong et al., 2023b), we use their direct reconstruction method extract the 3D features from each scene in our training data. To process our long terg- memory data, which requires multi-scene input across each task, we feed each room in the task through the 3D-LLm Q-Former head independently to get separate 32-token dense representation of each room with per-room 3d positional embeddings injected into the features. Then we concatenate the representations before feeding the input into the frozen (5-flanxl (Chung et al., 2022) backbone like the original work.
The 3D-LLM model also included learned location tokens used deseribe certain locations within each room in the scene. To fit 3D-LLM to our task data, we substitute the location tokens with our specific interaction tokens (eg. <GO TO ROOM> used by all models in our experiments) and train the model to learn the new tokens to stay consistent with our higher level interaction used across our training data. Analysis of the 3D-LLM model evaluation output, indicated the primary struggle for the model was retaining long term memory of semantic observations in the scene, so we prioritized aligning 3D-L.LM with the high level long-term memory representation our data over low level spatial understanding of the scene.
24

---

System message
You are an assistant and task generator for 3D embodied agent operating a multi-room environment. The environment provides detailed object instance information, including bounding boxes and DDs. Your goal is to generate a complex task that requires the agent explore multiple rooms, navigate, and crucially use long-term memory to recall details observed carlier.
Prompt
Environment and Object Information
Object Representation: Each object is given with a bounding box in the format: "cob- ject "ot_nam>>num)": [x_min, y_min, z_min], [x_max, y_max, x_max] Here, (num) indicates the ID, with (0) being the closest the origin [0,0,0]. IDs reset for each room (c.g.. sofa(0) in room 2 and sofa(0) in room 4 if each room has sofa).
Actions Available: <GO ROOM(id)>: Navigate a room that has already been visited. <GO TO NEW ROOM>: Navigate to a new, unexplored room (and unlock its objects). Do not use this for rooms that have been visited before. <PICK UP object name(id) from room(id) in room(id)>: Pick up object that originally belongs specific
while in that room. <PUT DOWN object name(id) from room(id) on object name(id) in room(id)>: Place an object (that originally belongs to a room) onto another object (such as table or floor) in room.
New Objects: You can add extra objects to diversify the task. Important: Use only object names from the provided new objects name list. If a room already has object with the same name, new object should have a new ID (e.g.. if lamp(0) exists, the added one should be lamp(1)). These extra objects are only for task design; the agent's 's trajectory should not mention adding them.
Task Design Requirements
Multi-Room Exploration: Design a task that spans several rooms. The room order (given a Room Order list) should be chosen so that necessary items are distrihuted across rooms. The agent should explore every room in the specified order.
Long-Term Memory and Implicit Cues: Do simply list all items checklist the start. Instead: Provide vague overall goal (e.g., "prepare meal"). Later in trajectory. the agent recall these earlier observations when the need arises. Ensure the agent must remember something seen long ago rather than simply following an explicit list.
Update Memory and make new decision based your current observations: The agent originall planned use one object completing task, but couldn't find after exploration rooms. has to change a another similar object complete its task.
Inventory and Action Constraints: The agent can only hold one item at a time. Never perform consecutive PICK UP PUT DOWN actions. If the agent holds item, it must put down hefore picking up another. When temporarily storing an object (e.g., table), include "thought" explaining why the object is being set down and later recalled.
3. S.Ressoning and Object Comparisons: If your task requires choosing specific object instance (e.g., selecting table(1) because it is higger than table(0)), compare their bounding boxes and explain your choice in the trajectory. For clarity, consider these examples: (In-context examples)
Here is the scene information: [Input scene information]
Table 4: Prompt template for generating task trajectories. (In-context examples) are in-context examples. (Input scene information) } are scene, and object semantics along with their bounding boxes.
Our longer task data input also required truncation to fit within the 512 token context length of 3D- LLM's fl-flanxl backbone. We retain the task description and move the question the beginning of the prompt for the QA data to ensure the model still receives the information necessary understand its tasks. The longer trajectory of past events is then the only information that gets truncated before fed into the t5 encoder.
25

---

Prompt
You are an assistant / task generator in room. All object instances this scene are given, along with their bounding boxes and ids." Each object's bounding boxes are represented by 3D coordinate sobj_name><numm [x min, min, z min],[x max, max, z max]" with units of meters, and each represents left-botton corner and the right-top comer coordinate.
You will also receive a trajectory composed of the following tokens and reasoning chains. <GO TO ROOM(id)>: which navigates back to specific room (id). This can only be done if the agent already go to this room. <PICK UP object_ name((dd) from room(id) in room(id)>: Pick up an object that originally belongs to a specific room while in that same room. <PUT Place an object
name(id) DOWN nime((d) from room(id) on (bjec)_name(id) in room(id)>: (that originally belongs to room) onto another object (such as table or floor) in room. <GO TO NEW ROOM>: which navigates to new room you haven't It explored and unlocks objects there.
This trajectory is what the agent have executed over the past. You need to propose several questions and answers that focused on the reasoning abilities of the long-term memory of the agent. These reasoning questions should focus what have changed temporally or spatially this agent's memory. It's important that this change challenged the agent's memory. For example the questions should contain object counting, spatial relation, comparison between objects across rooms, long-term multi-room room layout, long-term multi-room object navigation. Remember spatial memory is important, you should design questions that asked about the 3D object spatial relation and layout in the that need the agent to perform hard reasoning for the final answer.
For clarity, consider these examples:
[In-context examples)
Here is the scene information: (Input scene information) ) Here is the agent's trajectory: [Input agent's trajectory)
Table Prompt template for generate QA data. [In-context examples] are in-context examples. (Input scene information] are scene, room and object semantics along with their bounding boxes. (Input agent's trajectory) is the 3D agent's explored trajectories and action chains.
the hyperparameters provided by 3D-LLM and finetune until model loss stops decreasing. Due to compute limitations, we trained on captioning task for 15 epochs, question-answering task for 20 epochs, and allocated most of the compute time the embodied task, which we trained on for 75 epochs.
For finetuning data,
3D-Mem We benchmark 3D-Mem (Yang et al., 2025b) the question-answeringg and caption- ing splits of 3DMEM -ENCH. 3D-Mem is a snapshot- baase 3D memory architecture originally developed for embodied exploration and reasoning; it keeps two complementary stores-memory snapshots, a compact set of multi-view RGB-D frames with per-object bounding boxes summarizing the areas the agent has inspected, and frontier snapshots, boundary views that suggest where useful new information may be found next. In its native setting the agent navigates an unfamiliar scene by selecting the frontier view most likely advance its task and then answers visual questions using the most relevant memory snapshots. Because our evaluation focuses on post-exploration reasoning rather than active exploration, we disable the frontier component and retain only the memory snapshots. For these two tasks, the system will capture memory snapshots in each room from the room center, and finish the QA and captioning base on the memory snapshots of all the explored rooms.
H Qualitative Examples
We provide qualitative examples as shown in Figure 6. It demonstrates that 3DLLM-MEM can maintain a long-term memory and perform complex tasks in the embodied environments. More examples can be found in the supplementary materials.
26

---

Prompt
You are provided with scene description containing multiple rooms. Each room includes a list of objects along with their positions in the room, represented by bounding boxes. Each object's s bounding box is defined by a 3D coordinate in the format: <object__aane _ame>>num)): [x min, min, min]./x max, max, max] with units in meters (defining the left- bot-bottom and right-top corners). Your task generate an object caption for each
the form coherent, descriptive paragraph that conveys the 3D spatial arrangement relative positions of all objects within that room.
Then, you will receive the object descriptions and caption for the current 3D room you in. You will also be provided with the previous rooms' captions well. Your task is to generate new captions covering the summarization of common features across all rooms based on your current room and important difference based on your current room. The reasons of generating the new caption is to help the agent to remind of what are in previous rooms memories can help the agent in this current room. The past objects and observations should be related current room by examining the summarization of common things and differences. For clarity, consider these examples: (In-context examples)
Here is the scene information: [Input scene information]
Here is current room you are in and previous rooms you went: [Input agent's location)
Prompt template for generate QA data. (In-context examples) are in-context examples. (Input scene information) are scene, room and object semantics along with their bounding boxes. (Input agent's location) is the location for current room in the scene and the past explored rooms.
Table 6:
System message
Please act as impartial judge and evaluate the quality of the response provided by AT assistant the user question. Your evaluation should consider correctness and helpfulness. You be given reference answer focus on the assistant's 's answer to the second question. Begin your evaluation by comparing the assistant's 's answer with the reference answer. Identify and correct any mistakes. Be as objective possible. After providing your explanation, you must rate the response scale of to by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".
assistant's 's answer. You evaluation should
Prompt
<lThe Start Reference Answert>
### User:
question_1
###l Reference answer:
ref Lnaswr__
User:
question_2
#Hill Reference answer:
ref answer 2
<iThe End of Reference Answerbz
<IThe Start of Assistant A's Conversation with Userl>
### User:
question_1
##B Assistant A:
answer
### User:
question_2
#iilt Assistant A:
answer
<Thhe End Assistant A's Conversation with Userb>
Table 7: Prompt template for open-ended QA evaluation following standard LLM-as-judge protocol.
27

---

Room 8
Room
Object Captioning
Room 8 contains distinct sleeping, working, and lounging zones. Bed(O) extends along one side, with pillow(O) near head and cabinet(1) beside which supports decoration(O) and is flanked by book(O) and book(1). Near the foot of bed(O), yoga mat((), bag(O), and box(O) form small activity area. At the far end, wardrobe(O) stands near wall(4), next beanbag chair(0) and exereise ball(O), with clothes(O) draped nearby, creating relaxed comer. The office positioned across from bed(O), includes desk()), office chair(O), and cabinet(O), with trashcan(0) nearby and ceiling lamp(0) overhead. Window(O) the parapet(O) brings in natural light. The arrangement ensures functional use of space while maintaining clear separation between different activities.
Question Answering
Object Counting
Q: How many chairs are there Room 8?
A: There are chairs total: one office chair(0) one beanbag chair(O).
Spatial Relation
Q: you are sitting the office chair(0) working the desk(0) in Room 8, where the window(0)
relative you?
A: The window(0) is behind you the left.
Comparative Q4
Q: Which room has more books, Room 8 or Room 9?
A: Room 8 has more books. It contains two books, book(() and book(1), both placed on top of the cabinet(0) near the desk(O).
Object Navigation
Q: you are sitting on the beanbag chair(O) Room 8, how you get to books?
You need to stand up, turn left, and walk across the room toward the cabinet(0) near the desk(O),
where the books are placed.
Room Layout
Q: In Room 8, what is the relative layout of the bed(O), desk(O), and office chair(0)? Which one is closest to the window(O)?
A: 'Tee bed(0) is placed along the side wall near the window(O), the desk(0) is located in the far corner of the room along wall(5), and the office chair(0) is positioned directly in front of the desk(O). Among these, the bed(0) closest to the window(O), which is embedded in the wall beside it.
Figure 5: Example of human annotators manually check the data quality on QA and captioning tasks through multiple rendered multi-view images from each room.
28

---

(2)
(4)
(6)
Figure 6: Qualitative example of 3DLLM- MEM.. The task instruction is: Prepare a cozy reading nook in the living room with two books and teacup. In images (1) and (2), the agent explores the environment randomly, forming an initial memory of the scene. After receiving the task instruction, it recalls its memory and navigates to the bedroom to pick up book from the cabinet, shown in images (3) and (4). The agent then returns to the living room and places the book on the table in front of the sofa (image 5). Unable to recall any additional books, the agent resumes exploration and finds a second book on the bed, which picks up (image 6) and stacks on top of the first book (image 7). Finally, the agent recalls seeing teacup in the kitchen, navigates to retrieve (image 8), and places on the table in the living room (image 9). The task is successfully completed.
29


Reviews:
[{"review_id": "UjpedYc6Nq", "reviewer": "Reviewer_syxz", "summary": "This paper introduces a novel benchmark 3DMEM-BENCH for reasoning, planning and acting with long-term spatial-temporal memory in embodied environments. In the meantime ,this paper introduces  a 3D embodied LLM with dynamic memory management capabilities designed specifically for embodied spatial-temporal reasoning, planning and acting, and evaluates popular 3D-LLMs and memory mechanisms on 3DMEM-BENCH.", "questions": "Section 2.2 notes a low 24% validation rate for Gemini-generated trajectories. What were the most common failure modes that caused these trajectories to be invalid (e.g., illegal navigation, violations of object interaction rules (such as the \"hold only one object at a time\" constraint mentioned in Appendix F.1)? Could you provide several failure case analysis?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n1. This paper collected a fine-grained benchmark with multiple difficulty levels and varying numbers of multi room scene settings, which can assist the model in evaluation.\n2. This paper proposes a new mechanism for modeling long-term spatio-temporal memory, which interacts with attention and historical memory to effectively improve the performance of the model in complex scenarios.\n3. The approach proposed in the paper when switching environments involves storing working memory in the episode memory bank, which greatly reduces costs through this mode of thinking.\n\nWeaknesses\n1. The paper describes how memories are added to and updated in the episodic memory bank, but not how they might be pruned or consolidated. In extremely long-horizon scenarios, this could lead to the information redundancy and the memory bank growing indefinitely, which might reintroduce the very problems of computational inefficiency.\n2. While a significant effort was made to validate the data , tasks generated by an LLM (Gemini)  and executed within a simulator may not fully capture the physical world. This creates a risk that the model may be learning to exploit artifacts of the simulation rather than developing skills that are robustly transferable to real-world problems.", "was_revised": true, "final_justification": "The author's response addressed most of my concerns. I will maintain a positive score.", "rebuttal": "We sincerely thank you for your constructive feedback and for describing 3DMem-Bench as a **\"novel and fine-grained\"** benchmark and 3DLLM-Mem \"**effectively improves the performance of the model in complex scenarios**\". We greatly appreciate your suggestions which are crucial in improving the quality of our paper.\n\n>W1: How to handle scenarios when memory bank growing indefinitely? \n\nThank you for raising this valuable concern. In the current benchmark configuration (Sec.2, L.142), an embodied task involves at most 10 rooms. Yet we have constructed a challenging benchmark as all models' performance is still low. Therefore, in the context of 3DMem-Bench,  the memory bank **remains small enough to avoid computational inefficiency**.\n\nNevertheless, we agree that scalability is an essential consideration for broader real-world applications. We are happy to discuss the following promising approaches to manage memory growth:\n\n1.  Sliding-window memory (FIFO). Maintain only the most recent N memories and discard older ones. This is similar to the “Most-Recent Memory” baseline in our paper but now applies to the memory bank. This method keeps both latency and memory O(N).\n2.  Memory consolidation / compression. The cross attention complexity of the query ($N$ x $d$) and past memories key and values which are ($T$ x $N$ x $d$) is $O(TN^2d)$. \nWe can: 1) down-sample token length $N$. 2) project to a lower dimension $d$ (e.g., LoRA-style adapters), or 3) cluster similar memories and retain only their representative centroids. While promising, a careful study of the resulting accuracy–efficiency trade-off would be an excellent topic for future work.\n3. Retrieval-augmented subsets.\nIndex memories with a light-weight key–value store (FAISS, ScaNN) and then apply memory fusion only to the top-K retrieved memories. This reduces the effective $T$ and is orthogonal to the above techniques.\n\nWe believe all these approaches can be promising future works but **beyond the scope of the current paper**.\n\n>W2: The tasks in a simulator may not fully capture the physical world.\n\nThank you for this feedback. We share this concern and directly investigated the generalization and transferability of our model within the 3DMem-Bench by splitting tasks into \"in-domain\" and \"in-the-wild\" sets. Our results show that 3DLLM-Mem demonstrates **strong generalization** capabilities on the unseen \"in-the-wild\" tasks.\n\nMoreover, we believe our approach has **high potential for real-world transferability**. As illustrated in paper's Figure 1 and 3, the model outputs high-level, symbolic actions ``(e.g., Navigate(room=kitchen), PickUp(bottle))``. The low-level motion primitives are executed by modules that are already deployed and tested on real robots. Consequently, the learned policy primarily needs to decide *what to do*, not *how to do it*, which substantially narrows the reality gap often encountered in methods that learn low-level continuous control policies.\n\n>Q1: Failure Cases. \n\nThank you for this question. We'll clarify the failure cases for both the data generation process and our trained model.\n\nRegarding the **task generation process**, as you noted, we discuss these limitations in Appendix F.1. The primary failure modes involve Gemini occasionally generating trajectories that violate constraints related to: 1) the agent’s final location, 2) the specific object referenced, and 3) the correctness of pick-up and put-down actions. The single most common failure is violating the \"hold only one object at a time\" constraint, as this requires strong reasoning and a consistent memory of the agent's current inventory.\n\nFor the failure cases of our trained **3DLLM-Mem model**, we have identified three common categories:\n\n1. **Perception Misalignment**: The alignment of the 3D perception encoder with the Large Language Model (LLM) remains a bottleneck in this domain, a challenge we also observed. When a room contains noisy depth data, incomplete 3D meshes, or small/occluded objects, our model can fail to perceive the target objects correctly, leading to incorrect subsequent actions.\n2. **Premature Task Completion**: We append a ``Task Complete`` token to the end of every training sequence to signal task termination. For complex tasks requiring multiple steps (e.g., collecting four instances of the same object from different rooms), the agent can sometimes lose track of its progress and incorrectly output the ``Task Complete`` token before the goal is fully achieved. We believe employing an explicit \"counting\" module could help the model track this information, which is an interesting direction for future work.\n3. **Cascading Errors**: We were pleasantly surprised to find that our model can sometimes recover from an incorrect action mid-task and still successfully complete its goal. However, a more common scenario is that an initial mistake (e.g., picking up the wrong object) leads to a cascade of subsequent incorrect actions. We believe that incorporating reinforcement learning with sparse rewards to penalize incorrect actions could improve model robustness and is another promising area for future research.\n\n*We sincerely appreciate your thoughtful comments and hope our responses have addressed your concerns. Please let us know if you have any further questions.*\n\nBest,\nAuthors"}, {"review_id": "W8vM8yczv1", "reviewer": "Reviewer_9Co4", "summary": "This paper focuses on the challenge of 3D spatial understanding and memory retrieval in 3D-LLMs. To that end, the authors propose 3DLLM-Mem, a 3D LLM that features a novel memory fusion mechanism inspired by human memory and cognitive maps. In particular, the model uses working memory and episodic memory tokens to represent current and past observations and retrieve relevant memories.\n\nTo evaluate this method and support other research in this direction, the authors also introduce 3DMem-Bench, a benchmark with trajectories and tasks spanning embodied navigation, question answering, and captioning. They benchmark their method, along with recent 3D-LLMs and memory mechanisms, and show that their method significantly outperforms prior work on all tasks.", "questions": "1. What are the core differences between in-domain and in-the-wild tasks?\n2. How are the following terms defined: instance, environment, unseen memory context? \n3. What is the relationship between the working and episodic memory, and how are observations transferred from one to the other? Does the working memory contain observations from more than a single timestep, and if so, how is a new environment (as mentioned in lines 204-205) determined?\n4. Do you have insight into why 3DLLM-Mem demonstrates strong generalization capabilities? Is this simply due to better overall performance? Or are there specific components of the method that enable it to maintain performance compared to other approaches?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Proposed memory fusion mechanism is novel and well-justified\n- The proposed 3DMem-Bench benchmark is the first benchmark focusing on long-term memory and fine-grained complexity, and good details about the benchmark design principles and dataset collection process are provided\n- Strong results on both navigation and embodied QA tasks, demonstrating significant improvement over prior works\n- Demonstrates strong generalization to unseen environments\n\n**Weaknesses**\n\nThat said, I have some concerns about the clarity of the method and generalization contribution.\n1. Throughout the paper, there are some terms that are not properly defined and missing details, which make parts of the method unclear. In particular:\n    - It’s unclear what the difference is between the in-domain vs in-the-wild tasks. A better understanding of this would be helpful for understanding the generalization capabilities of 3DLLM-Mem.\n         - The details in Section 2.3 about how the in-domain and in-the-wild datasets are collected are highly specific, which makes it harder to understand what the core difference is between in-the-wild and in-domain tasks. \n         - Also, this section references “instances” and “unseen memory context”, which are not defined earlier and also make the description unclear\n    - The definitions of the environment, working memory, and episodic memory are unclear\n         - I’m confused about what observations are part of the working versus episodic memory. In line 182, it says that the current observation at each time step is stored in working memory, implying that the working memory stores a single timestep. In lines 204-205 though, it seems like the working memory is longer since it explains that “When the agent moves to a new environment, the previous working memory is transferred to the episodic memory bank.” \n         - It would help to have a concrete formulation of these terms, as they’re core to the method\n    - Details about training are missing.\n\n2. The paper emphasizes that 3DLLM-Mem demonstrates strong generalization to unseen environments. While 3DLLM-Mem has much higher absolute performance on in-the-wild tasks, the relative drop in performance between in-domain and in-the-wild is only slightly lower than other methods. More insight into this would strengthen this contribution, such as explaining which components of the method lead to this improvement.", "was_revised": true, "final_justification": "The authors have addressed my concerns about the clarity of the problem setup and generalization contribution, showing that they've made significant improvements compared to prior works by highlighting the relative performance drop as a metric and providing high-level insight into how their method is capable of generalizing. I've chosen to raise my score to a 5.", "rebuttal": "Thank you for your careful reading and detailed feedback. We're glad you found our proposed memory fusion mechanism to be \"**novel and well-justified**\", and we appreciate you highlighting our model's \"**strong results and generalization to unseen environments**\".\n\nWe provide detailed explanations for the points of clarity you raised below and will update our paper accordingly to ensure these concepts are easier to understand. We're also happy to engage further during the discussion period to clarify any remaining questions.\n\n> W1 & Q1: What the difference is between the in-domain vs in-the-wild tasks? What are “instances” and “unseen memory context”? \n\nThank you for this question. Let's first clarify the terms to facilitate the explanation.\n\n* Instance: An \"instance\" is simply a single data point or evaluation episode.\n* Environment: An \"environment\" refers to a specific 3D scene, such as a particular house layout with its rooms and objects. As illustrated in Figure 1, the agent navigated from the bedrooms to the living room, and then to the kitchen—all of which constitute the environment.\n* Unseen Memory Context: This means the agent is operating in an environment that it has never encountered during training.\n\nWith these definitions, the distinction between our evaluation splits (please also refer to paper's Lines 144-148) is as follows:\n\n1. **From an Environment/Memory Perspective**:\n* In-domain tasks take place in environments that may have been seen during training. Therefore, the agent's episodic memory bank might contain memories from a familiar scene. However, the agent's current observation (its working memory) is always from a novel, unseen state within that environment.\n* In-the-wild tasks, as noted in Line 147, feature **\"entirely unseen memory context.\"** This means the agent operates in completely new environments that were not part of the training data.\n\n2. **From a Task Difference Perspective**:\n* In-domain tasks follow the same distribution and types of tasks as those in the training set. For example, as illustrated in Figure 2, one such task is \"Find an ideal container to store all the cookie dough.\"\n* In-the-wild tasks, as noted in Line 148, introduce **novel challenges** that are more complex and qualitatively different from those seen during training. Examples of these challenging tasks are provided in Figure 2; corresponding to the in-domain example above, for instance, is \"Collect all cookie dough and then **rearrange** it on the living room table in descending order of size.\"\n\n*A brief note on design*: We designed the in-domain set this way to ensure the in-the-wild set remained a reasonably sized and robust evaluation pool containing our most valuable, entirely novel instances. The in-domain tasks are still challenging, as shown by the experimental results, because the agent must still act based on its immediate, unseen working memory.\n\n\n> W2: The definitions of the environment, working memory, and episodic memory are unclear. \n\nWe apologize for the confusion regarding the terms \"working memory\" and \"episodic memory,\" which are inspired by concepts in neuroscience. Your understanding that **\"working memory stores a single timestep\"** is exactly right! It represents the agent's perception of its current 3D environment at a specific moment.\n\nWe believe the confusion may have stemmed from the sentence: \"When the agent moves to a new environment, the previous working memory is transferred to the episodic memory bank.\" To be more precise: at each timestep $t$, the agent's current observation is its working memory. When the agent proceeds to timestep $t$+1, the observation from the previous step $t$ becomes a part of its past experience. This past observation is then incorporated into the episodic memory bank. This process is visualized in Figure 3(b), where the working memory tokens $f_t^Q$ are fused with the past memory bank features.\n\nTo further emphasize this distinction, **our paper defines** their structures with mathematical notation:\n\n* Working Memory (Line 182) has a shape of $R^{(N \\times d)}$, representing features from one time step.\n* Episodic Memory (Line 184) has a shape of $R^{(T \\times N \\times d)}$, representing features from $T$ past time steps.\n\nWe hope this resolves the confusion, and we will revise the text in the paper to make this relationship clearer\n\n> W3: Details of training are missing. \n\nWe appreciate you raising this, and we would like to clarify where these details are located in the paper. We describe our training setup in **Section 4.1**, under the paragraph **Implementation Details** (Lines 218-224). This section covers the training computation, architecture, and loss function.\n\nFor more granular details, we mention in Line 224 that a full list of hyperparameters is available in **Appendix D** (Lines 691-699). There, we specify the learning rate, optimizer, weight decay, and learning rate scheduler used for our experiments. We will also open-source our code after the review period. Thank you for your understanding.\n\n> W4.1: While 3DLLM-Mem has much higher absolute performance on in-the-wild tasks, the relative drop in performance between in-domain and in-the-wild is only slightly lower than other method. \n\nThank you for this insightful observation. To better address your point, we've created a new table using the scores from Table 2, adding columns for absolute and relative performance drops.\n\n||In-domain SR|In-the-wild SR|Drop (Absolute)|Drop (Relative)|\n-|-|-|-|-|\nMost Recent Memory|21.1|13.7|-7.4|-35.1%|\nRetrieval-Augmented Memory|22.3|15.6|-6.7|-30.0%|\n3DLLM-Mem (Ours)|37.6|32.1|-5.5|-14.6%|\n\n\nAs illustrated above, our method demonstrate a **-5.5** absolute and a **-14.6%** relative performance drop which is significantly smaller than the >30% relative drop seen with other methods. We believe this demonstrates a substantial improvement in generalization, not just a \"slight\" one.\n\nFurthermore, we'd argue that relative drop is a more equitable metric here. It's inherently more challenging to **maintain a high level of performance** (like our 37.6% -> 32.1%) on difficult, unseen tasks than it is to **maintain a low level of performance** (like 21.1% -> 13.7%). Our model's ability to retain most of its high performance highlights its robustness.\n\n> W4.2: More insight into this would strengthen this contribution, such as explaining which components of the method lead to this improvement.\n\nThank you for recognizing our method's effectiveness. We attribute this strong generalization capability directly to our **novel memory fusion module** (Section 3.2), which excels for two key reasons:\n\n1. **Relevant Initialization**: The fusion process begins with query tokens initialized from the agent's current working memory. This is crucial because it primes the attention mechanism to selectively retrieve and focus on past memories that are most relevant to the agent's immediate surroundings and task. As shown in our ablation study (Table 3), initializing from other sources leads to inferior performance.\n2. **Holistic Fusion Process**: Unlike simple retrieval methods that might only fetch the single \"best\" memory, our fusion module **attends to all past memories simultaneously**. It learns to weigh and synthesize them into a single, compact, and context-rich representation. This allows the model to capture complex, long-term dependencies and make more informed decisions, which is especially critical for the novel and challenging tasks in the in-the-wild set.\n\n> Q2: How are the following terms defined: instance, environment, unseen memory context? \n\nPlease see Weakness 1 and 2. \n\n> Q3: What is the relationship between the working and episodic memory? \n\nPlease see Weakness 2. In short, working memory is the agent's current observation, which becomes part of the episodic (past) memory in the next timestep.\n\n> Q4: Do you have insight into why 3DLLM-Mem demonstrates strong generalization capabilities? Is this simply due to better overall performance? Or are there specific components of the method that enable it to maintain performance compared to other approaches?\n\nPlease see Weakness 4.2. The performance improvement is from our deisgned fusion memory module as introduced in paper's section 3.2. \n\n*We sincerely appreciate your thoughtful comments and hope our responses have addressed your concerns. Please let us know if you have any further questions.*\n\nBest,\nAuthors"}, {"review_id": "OMns3cUVeG", "reviewer": "Reviewer_VsZD", "summary": "This paper proposes a 3D Vision-Language-Action (3D-VLA) model equipped with long-term memory capabilities. In the context of this work, a 3D-VLA model begins with random exploration, during which it samples scenes randomly and collects RGBD + pose pairs for selected actions (exploration phase). This is followed by task-trajectory execution, where the agent (i) performs actions, (ii) captures corresponding visuals, and iterates until the task is completed or terminated.\n\nThe novel contribution of this paper, compared to prior work, is its argument that working memory, commonly used in such agents, is insufficient for tasks requiring reasoning over extended horizons. It further posits that existing benchmarks are too simplistic to evaluate such capabilities effectively.\n\nTo address this, the paper introduces a new benchmark based on a synthetic Habitat dataset (covering 182 spaces and 2.6k rooms). Tasks are generated using Gemini, leveraging pre-known scene/object geometry and semantics, and are filtered/verified via a trajectory simulator. A key strength of the dataset is its categorization of trajectories into three difficulty levels (based on the number of rooms required to complete a task) and its division into in-domain and in-the-wild settings (the latter includes areas not seen during model training).\n\nOverall, this benchmark offers a robust testbed for evaluating long-term action planning in indoor environments.\n\nThe second contribution is a baseline model that extends LLAVA-3D with long-term memory capabilities. The base network projects patches (tokens) from image/RGBD pairs into the LLM’s embedding space, enabling them to be processed alongside instruction tokens by the base LLM. For spatial reasoning, these embeddings are augmented with 3D positional encodings. To incorporate long-term memory, the model maintains a queue of past observations (termed episodic memory) in addition to the standard LLM context window, enabling retrieval. A memory fusion module allows tokens from active memory to attend to episodic memory, producing a fused representation that is then fed to the base LLM. This enables the LLM to operate not only on active memory but also on context retrieved from long-term storage.", "questions": "On the memory update, the paper states:\n\n\"The working memory is dynamic and updated online. As the agent interacts with the environment, changes in the environment are immediately reflected in the working memory through updated 3D representations. When the agent moves to a new environment, the previous working memory is transferred to the episodic memory bank. If the corresponding environment already exists in the memory bank and has been modified by the agent, the memory entry is updated accordingly\". \n\nCould you please clarify how precisely this is implemented/executed? This appears rather important, I imagine that 'carrying' around full past is in practice not feasible.  \n\nAlso, please see **Why is the proposed method superior to the Retrieval-Augmented Memory baseline?** above.", "limitations": "A single sentence in the conclusion: \"One limitation of our model is that currently 3DLLM-MEM does not involve low-level navigation and control policy, but utilizes high-level pre-defined policies in simulator for carrying out the actions\".\n\nI think this discussion should have been expanded. The success rate of the proposed method is still low (not surprising, the task is hard), but readers would greatly benefit from understanding where the proposed method still fails. Also, at the moment, this model is entirely constrained to synthetic data. There is no evidence for generalization outside of Habitat sandbox. I understand that evaluation outside of that may be difficult, but the bottom line is: no, limitations are not adequately discussed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "###  Strengths \n\n* The paper tackles a significant and underexplored challenge in the 3D-VLM domain: long-term exploration.\n* The proposed benchmark is a great contribution and has the potential to advance the field. The benchmark construction process is innovative, particularly with the inclusion of objects from Objaverse (to address the lack of objects in Habitat), alongside LLM-based trajectory generation and simulation-based verification.\n* The proposed methodology is a sound adaptation of prior work (LLAVA-3D) and long-term memory mechanisms established in the literature.\n* The experimental evaluation is convincing, and I appreciate the inclusion of well-constructed baselines.\n\n### Weaknesses\n**Prior work & contextalizing this paper w.r.t prior art.** \n\nThe paper effectively discusses related work in the 3D-VLM domain. However, its treatment of long-term memory techniques is incomplete and poorly contextualized. The proposed mechanism appears heavily inspired by existing work in the LLM domain, which extends the \"working memory\" window by attending to a subset of previously processed data (stored as a KQV queue) retrieved via kNN. For instance, Wu et al.’s “Memorizing Transformers” (ICLR ’22) employs a similar approach (note that the paper states in Section 3.2: “To manage episodic memory, we propose the use of a memory feature bank,” whereas it should acknowledge following established practice).\n\nWhile extending this concept to the 3D-VLM domain is a notable contribution, the omission of this discussion is concerning.\n\n\n**Presentation:**\n\n1. Certain sections of the paper are well-written, particularly those drawing parallels between memory modules and neuroscience, which I appreciate. However, the paper falters in addressing fundamental questions: What is the task at hand? What are the model’s inputs and outputs? What is the problem statement or task definition? The paper is so focused on the “how” that it fails to clearly communicate the “what.” While this information can be pieced together from the paper or related works, its absence in the introductory section places an unnecessary burden on the reader. I only fully understood the task definition in the “Embodied Data Collection” paragraph of Section 2.2 (Data Collection).\n\n2. The paper builds upon LLAVA-3D and augments it with long-term memory capabilities. To be self-contained, it should include a preliminary section briefly describing the LLAVA-3D architecture. Additionally, the paper would benefit from a clear problem setting and task definition section before delving into data collection, generation, and modeling, with these elements foreshadowed in the introduction.\n\n**Why is the proposed method superior to the Retrieval-Augmented Memory baseline?**\nThe aforementioned baseline seems very similar to the proposed method, yet experimentally, the proposed model performs significantly better. It is unclear from the experimental evaluation where this difference comes from, as well as what precisely is the key difference at the methodology level responsible for the performance gains. From the paper description, it appears to me that the key difference is in how the memory fusion is performed?  \n\n**Nit:**\n* Abstract: “reason over long-term memory in 3D environments” → consider rephrasing to “reason over extended temporal spans using long-term memory” for clarity.\n* Table 1 analyzes different datasets and includes a column labeled “Long-Term Memory.” Benchmarks do not possess memory, so this label is misleading.\n\n\nI am genuinely excited about this work, but its current presentation issues are significant. I am open to revising my rating upward after the rebuttal if my concerns are adequately addressed.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns very well, specifically, it clarified the difference between retrieval baseline (a la memorizing transformers) and the proposed methodology (memory fusion), and highlighted conceptually why this approach performs better. Please revise the paper as suggested, especially with regard. crisp problem definition/description.", "rebuttal": "We'd like to express our sincere gratitude for your thorough review of our paper. We're thrilled that you found our work **\"tackles a significant and underexplored challenge,\"** that our **\"benchmark is a great contribution,\"** and that our **\"experimental evaluation is convincing.\"** We greatly appreciate your suggestions, which are crucial for improving the quality of our paper.\n\n> W1: Contextalizing our work w.r.t prior work. \n\nThank you for pointing out “Memorizing Transformers” (ICLR ’22). While our initial literature survey focused on embodied AI, we agree this is a relevant work and will cite and discuss it in our revised manuscript. Its approach highlights **key differences that underscore the novelty of our method**:\n\n1. **Memery Fusion vs. Retrieval**: “Memorizing Transformers” utilizes a k-NN lookup to retrieve top (key, value) pairs from memory. This is a form of **retrieval**, similar in principle to our \"Retrieval-Augmented Memory\" baseline. Our proposed 3DLLM-Mem method is **fundamentally different**; it uses **fusion**. While retrieval selects a subset of the \"best\" memories based on a fixed similarity score, our fusion module **attends to all memories simultaneously**. It uses a parameterized cross-attention mechanism (Line 196) to learn a new, task-relevant representation by selectively aggregating information from the entire memory bank. This allows the model to learn what past information is most important for the current step, a capability our experiments show is highly effective.\n2. **Dynamic vs. Static Memory**: “Memorizing Transformers” stores a long-term context (like a text document) as a static KV cache. In our embodied setting, the memory is dynamic. As the agent interacts with the environment, the world state changes (Line 202, Memory Update). Our model is designed to handle this dynamism by continuously updating its memory bank and fusing these updated memories at each step to inform its actions, as shown in Figure 3.\n3. **Different Problem Settings**: As you acknowledged, the settings are distinct. “Memorizing Transformers” focuses on extending the effective context window for language models on text-based tasks. Our work, as stated in our abstract, introduces \"a novel dynamic memory management and fusion model for embodied spatial-temporal reasoning and actions.\"\n\nMoreover, these **significant differences in application settings fundamentally shape model design choices**—for example, the approach to extending a language transformer’s context window via KV cache versus enabling spatial-temporal memory for dynamic 3D embodied tasks.\n\n> W2: Presentation: What is the task definition and what are the model’s inputs and outputs? The 'what' is missed from paper's Introductory section. \n\nThank you for highlighting this. We apologize if the introduction was not perfectly clear. We will definitely revise the introduction to be more explicit.\n\nTo clarify here:\n\n* The task is for an embodied agent, given a high-level language instruction (e.g., \"prepare the most suitable gift box for the teddy bear.\"), to navigate a 3D environment and perform a sequence of actions to complete the goal. We provide a visual \"teaser\" of this in Figure 1.\n* The model's input at each step is the current 3D visual observation (working memory) and a bank of past 3D observations (episodic memory). Please also refer to paper's Sec 1, Line 71.\n* The model's output is a symbolic action, such as PickUp(teddy bear).\n\nWe will incorporate this clear problem definition into the introductory section of our revised paper.\n\n\n> W3: Presentation: A preliminary section briefly describing the LLAVA-3D and clear problem setting and task definition section before Section 2. \n\nThank you for this excellent structural suggestion. We'd like to clarify that we do introduce LLaVA-3D as a preliminary in **Section 3.1 (Lines 165-176)**.\n\nHowever, we agree with your point about the overall structure. Our original intention was to present our benchmark (which includes the task settings) before the model. Based on your feedback, we see that introducing the **high-level problem setting and task definition _before_ Section 2** would improve the paper's flow. We will make this structural change in our final version.\n\n> W4 & Q2: Why is the proposed method superior to the Retrieval-Augmented Memory baseline? \n\nYou are exactly right that \"the key difference is in how the memory fusion is performed.\" As we elaborated in our response to W1, the core distinction is **Fusion vs. Retrieval**.\n\nTo summarize: The Retrieval-Augmented baseline selects the top-K \"best\" memories based on a static similarity metric. This has two potential drawbacks: the similarity metric may not be optimal, and the top-K memories might not contain all the necessary context.\n\nIn contrast, our fusion-based method **attends over all memories** and learns a **dynamic, task-dependent** way to combine their information. It isn't limited to a few retrieved memories; it can synthesize information from the entire history. This allows it to capture more complex, long-range dependencies and makes it more robust, as it's not dependent on a single, potentially flawed retrieval step.\n\n\n> Reponse to Nit: \n\nThank you so much for your careful reading! We will happily make these changes for clarity in the revised version:\n\n* We'll change “reason over long-term memory in 3D environments” to “reason over extended temporal spans using long-term memory.”\n* We'll change the column name \"Long-term Memory\" in Table 1 to something less ambiguous, such as \"Evaluates Memory,\" to avoid confusion.\n\n> Q1: Clarify how precisely memery update is implemented/executed? \n\nWe are happy to further explain the memory update process (Lines 202-209).\n\nFirst, as noted in Line 142, the number of distinct locations (e.g., rooms) in any task is finite, so the memory bank doesn't grow indefinitely. The memory bank itself is a structured collection of feature vectors, where each vector corresponds to a specific, previously visited location.\n\nWhen the agent revisits a location and performs an action that alters the scene (e.g., picking up an object), we re-compute the 3D feature representation for that location. We then **update the corresponding entry in the memory bank** with this new feature vector. Implementation wise, it's as simple as Bank(entry)=new_vector. This is what's described in Lines 205-207: \"If the corresponding environment already exists in the memory bank and has been modified by the agent, the memory entry is updated accordingly.\" This process ensures the agent's memory of the world state remains current. \n\n\n> Limitation: \n\n\nThank you for this valuable suggestion. We did have to shorten our limitation section due to space constraints, but we will add a more comprehensive discussion to the Appendix, including real-world evaluation and a detailed breakdown of failure cases.\n\nAs a preview, here are the three common failure categories we've identified:\n\n1. **Perception Misalignment**: The alignment of the 3D perception encoder with the Large Language Model (LLM) remains a bottleneck in this domain, a challenge we also observed. When a room contains noisy depth data, incomplete 3D meshes, or small/occluded objects, our model can fail to perceive the target objects correctly, leading to incorrect subsequent actions.\n2. **Premature Task Completion**: We append a ``Task Complete`` token to the end of every training sequence to signal task termination. For complex tasks requiring multiple steps (e.g., collecting four instances of the same object from different rooms), the agent can sometimes lose track of its progress and incorrectly output the ``Task Complete`` token before the goal is fully achieved. We believe employing an explicit \"counting\" module could help the model track this information, which is an interesting direction for future work.\n3. **Cascading Errors**: We were pleasantly surprised to find that our model can sometimes recover from an incorrect action mid-task and still successfully complete its goal. However, a more common scenario is that an initial mistake (e.g., picking up the wrong object) leads to a cascade of subsequent incorrect actions. We believe that incorporating reinforcement learning with sparse rewards to penalize incorrect actions could improve model robustness and is another promising area for future research.\n\n*We sincerely appreciate your thoughtful comments and hope our responses have addressed your concerns. Please let us know if you have any further questions.*\n\nBest,\nAuthors"}, {"review_id": "H7vQFM326a", "reviewer": "Reviewer_kteJ", "summary": "This paper first proposes a 3D benchmark that is called 3DMEM-Bench, covering 26000 trajectories and 2892 embodied tasks. Then this paper proposes the formulation of 3DLLM-EME, which is a model with dyanmic memory management and fusion for embodied spatial-temporal reasoning and actions. In details, this model us working memory tokens to select from episodic memory. Experiments on 3DLLM-MEM shows the effect of the proposed method.", "questions": "1. What is the efficiency of the proposed model compared with other methods?\n\n2. How to adopt this model into scenes with more dynamics and interaction? \n\n3. What is the contribution of each component in your framework?", "limitations": "No limitation is claimed in this paper. I think this paper should discuss more about their limitations in dealing with various scenes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nThis paper builds a new 3D benchmark, and the idea of working memory and episodic memory is interesting.\n\nWeakness:\n\n1. I think the idea of querying tokens from the episodic memory is rational. However, it may cause the huge computation for reasoning. The authors should list the efficiency comparison. Moreover, if the thinking process is too long, the environment will be changed a lot, and the past observations and interactions might be useless. \n\n2. Especially, I think this model can be only adopted to the static environment, while cannot handle the dynamic environment. Especially, the interactions with others are also not described.\n\n3. This model is built on the base model of LLaVA-3D. Thus, I wonder whether the advantage of 3DLLM-MEM is derived from the base model or the memory setting or the training data? This is not analysized sufficiently.\n\n4. The authors could discuss some failing cases.", "was_revised": true, "final_justification": "After reading the paper, rebuttal from authors, and the comments from other reviewers, I think this paper has some significant contributions, such as building a new 3D benchmark and proposing the working and episodic memory. However, I think some concepts are not convincing, such as I still do not think this work can solve the dynamic scenes. Thus, I give the borderline score, but prefer to accept.", "rebuttal": "We appreciate your valuable comments, which are crucial for improving our paper! Thank you for finding our \"3D benchmark is **novel**\" and acknowledging that \"the idea of working memory and episodic memory is **interesting**.\" \n\n> W1.1 & Q1: Efficiency comparison with other methods. \n\nThank you for this insightful question. We've benchmarked the efficiency on the medium set of our embodied tasks and calculated the average inference time per question, and the results show that our method achieves a **massive improvement** in success rate with only a **marginal increase** in inference time.\n\n**In-domain Tasks**\n| Method | Success Rate ↑ | Time(s)↓ | TFLOPs |\n| :--- | :---: | :---: | :---: |\n| Most Recent Memory | 20.1% | 12.3 | 0.31 |\n| Retrieval-Augmented | 21.8% | 12.1 | 0.36 |\n| 3DLLM-Mem (Ours) | 36.8% | 13.6 | 0.33 |\n\n**In-the-wild Tasks**\n| Method | Success Rate ↑ | Time(s)↓ | TFLOPs |\n| :--- | :---: | :---: | :---: |\n| Most Recent Memory | 12.4% | 13.6 | 0.38 |\n| Retrieval-Augmented | 13.7% | 14.0 | 0.42 |\n| 3DLLM-Mem (Ours) | 31.6% | 14.1 | 0.39 |\n\nThe main computational cost for all methods comes from the **7B LLM backbone**. The slight difference in overhead arises from the memory processing step:\n\n* Most Recent Memory: No processing is needed; the memory is fed directly to the LLM.\n* Retrieval-Augmented: Overhead comes from calculating similarity scores against the memory bank.\n* Our Method: Overhead comes from a single cross-attention layer.\n\nThe complexity of our fusion step of the query ($N$ x $d$) and past memories key and values which are ($T$ x $N$ x $d$) is $O(TN^2d)$. Since the memory bank size $T$ is bounded by our task definition (Line 142), the $N$ and $d$ shares the same across all methods, this operation is highly efficient. Furthermore, its performance could be easily accelerated using standard optimizations like FlashAttention. \n\n> W1.2: Moreover, if the thinking process is too long, the environment will be changed a lot, and the past observations and interactions might be useless.\n\nThank you for raising this point. We respectfully suggest that for long-horizon reasoning tasks, **past observations often remain highly relevant**, even after significant time has passed.\n\nFor example, in the task shown in Figure 1, the agent must remember the size of the first teddy bear it found to correctly select a matching gift box much later in the episode. If it were to forget this initial information, the task would become unsolvable. The \"thinking process\" our model performs is precisely what allows it to connect these temporally distant but causally linked events. We are, of course, open to further discussion on scenarios where this might not hold.\n\n> W2: Limited to static environment, while cannot handle the dynamic environment.\n\nThank you for this concern. We'd like to clarify that our framework is **indeed designed for and evaluated in dynamic environments**. This is a core aspect of our work, mentioned in Lines 3, 7, and 64.\n\nThe agent is constantly interacting with and changing the environment by executing actions (as seen in Figures 1 and 3). Our **Memory Update** mechanism (Section 3, Lines 202-209) is specifically designed to handle this. When the agent alters a part of the scene, the memory entry for that location is updated with a new observation, ensuring the model's knowledge of the world stays current.\n\n> W3: Ablation of 3DLLM-Mem effectiveness with base model, the memory setting, the training data. \n\nOur short answer is that the effectiveness comes from our **novel dynamic memory management and fusion module**.\n\nTo isolate this contribution, our experimental setup uses the same base model (LLaVA-3D) and the **same training dataset** for all methods. We then compare several memory management techniques: \"Everything in Context,\" \"Most Recent Memory,\" \"Retrieval-Augmented Memory,\" and our proposed 3DLLM-Mem.\n\nAs demonstrated in Table 2, our method significantly outperforms all other baselines under these identical conditions. This shows that the performance gain is attributable to our memory fusion design, not the base model or the data. We provide further targeted ablations on the components of our fusion module in Table 3 to reinforce this conclusion.\n\n\n> W4: Failure Cases \n\nFor the failure cases, we have identified three common categories:\n\n1. **Perception Misalignment**: The alignment of the 3D perception encoder with the Large Language Model (LLM) remains a bottleneck in this domain, a challenge we also observed. When a room contains noisy depth data, incomplete 3D meshes, or small/occluded objects, our model can fail to perceive the target objects correctly, leading to incorrect subsequent actions.\n2. **Premature Task Completion**: We append a ``Task Complete`` token to the end of every training sequence to signal task termination. For complex tasks requiring multiple steps (e.g., collecting four instances of the same object from different rooms), the agent can sometimes lose track of its progress and incorrectly output the ``Task Complete`` token before the goal is fully achieved. We believe employing an explicit \"counting\" module could help the model track this information, which is an interesting direction for future work.\n3. **Cascading Errors**: We were pleasantly surprised to find that our model can sometimes recover from an incorrect action mid-task and still successfully complete its goal. However, a more common scenario is that an initial mistake (e.g., picking up the wrong object) leads to a cascade of subsequent incorrect actions. We believe that incorporating reinforcement learning with sparse rewards to penalize incorrect actions could improve model robustness and is another promising area for future research.\n\n> Q2: How to adopt this model into scenes with more dynamics and interaction? \n\nPlease see Weakness 2. Our setting is **already dynamic**, and the model is designed to handle interactions that change the environment.\n\n> Q3: What is the contribution of each component in your framework?\n\nPlease see Weakness 3. The primary contribution comes from our memory fusion module. We also provide a detailed ablation study in Table 3 that breaks down the effectiveness of its specific design choices. \n\n> Limitation. \n\nWe mentioned limitations in our conclusion (Lines 327-330), but we agree this section can be expanded. We will add a more detailed discussion to the appendix in our final version, including the failure cases listed in our response to W4. Thank you for the suggestion.\n\n*We sincerely appreciate your thoughtful comments and hope our responses have addressed your concerns. Please let us know if you have any further questions.*\n\nBest,\nAuthors"}]

Author rebuttal:

Meta-review: This paper proposes 3DMem-Bench designed to evaluate an agent's ability to reason over long-term memory in 3D environments, and 3DLLM-Mem, a novel dynamic memory management and fusion model for embodied spatial-temporal reasoning and actions in LLMs. Reviewers acknowledge the strengths of the paper, including the importance of the problem, the novelty and significance of the approach, and the convincing experiments. Reviewers raised concerns about generalizability to real-world scenes, clarity of evaluation setting, computation cost, comparison with other approaches, etc. Most questions are effectively addressed by authors during the rebuttal, except for one reviewer still having concerns on dynamic scenes. Despite this, all reviewers agreed on positive scores after rebuttal and recognize the merits of the work. After carefully considering the paper, reviewer comments, and rebuttal, the area chair recommends accepting this paper.
Decision: Accept (poster)